Opened 5 years ago

Closed 5 years ago

#12869 closed task (fixed)

protocol api separation

Reported by: iwakeh Owned by: iwakeh
Priority: Medium Milestone:
Component: Metrics/Onionoo Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

provide the protocol docs classes in a separate jar.

see comment 10 and 11 in #12732

Child Tickets

Change History (15)

comment:1 Changed 5 years ago by karsten

Or how about we add an Ant target for producing a .jar file with all classes, including all referenced jars ("fat jar")? Would that help?

comment:2 Changed 5 years ago by iwakeh

Owner: set to iwakeh
Status: newassigned

I'll look into it and post suggestions here.

comment:3 Changed 5 years ago by iwakeh

It would be nice to eliminate the util package dependency entirely.
Ideas for removing it:

  • I just became aware that util.Time is just there for testing.

Why don't you use Mocking for testing?
Here is an older reference
http://www.lshift.net/blog/2009/12/06/testing-times-in-java

The time testing could use this this (near the end of the page):

@MockClass(realClass = System.class)
public class MockSystem {
    private Calendar now = null;

    public MockSystem(Calendar now) {
        this.now = now;
    }

    @Mock
    public long currentTimeMillis() {
        return now.getTimeInMillis();
    }
}

This would free the onionoo runtime from test dependencies.

  • util.Logger is a mixture of formatting and logging.

In docs only DocumentStore.getStatsString uses the number formatting
functionality of Logger. This could be changed? E.g. replaced,
i.e. use String.format("%,d", decimalNumber) directly, and
by moving something like the byte formatting elsewere inside docs?

And, a general question, why doesn't onionoo use a logging framework?

---
Here is the ant task based on docs being independent of util:

  <target name="protocol-api-jar" depends="test">
    <jar destfile="protocol-api.jar"
	 basedir="${classes}"
	 includes="**/docs/" 
	 excludes="**/*Test.class"
	 duplicate="fail">
      <manifest>
	<attribute name="Built-By" value="onionoo project"/>
	<!-- maybe other attributes like the license etc. -->
      </manifest>
    </jar>
  </target>
Last edited 5 years ago by iwakeh (previous) (diff)

comment:4 in reply to:  3 Changed 5 years ago by karsten

Replying to iwakeh:

It would be nice to eliminate the util package dependency entirely.

Agreed, let's try that.

Ideas for removing it:

  • I just became aware that util.Time is just there for testing.

Why don't you use Mocking for testing?
Here is an older reference
http://www.lshift.net/blog/2009/12/06/testing-times-in-java

The time testing could use this this (near the end of the page):

@MockClass(realClass = System.class)
public class MockSystem {
    private Calendar now = null;

    public MockSystem(Calendar now) {
        this.now = now;
    }

    @Mock
    public long currentTimeMillis() {
        return now.getTimeInMillis();
    }
}

This would free the onionoo runtime from test dependencies.

Sure, happy to talk about using a mocking framework. But we'll have to find one that is shipped with Debian stable. See development guidelines here: https://trac.torproject.org/projects/tor/wiki/org/operations/Guidelines

Do you have any suggestions? And do you want to create a new ticket for that?

  • util.Logger is a mixture of formatting and logging.

In docs only DocumentStore.getStatsString uses the number formatting
functionality of Logger. This could be changed? E.g. replaced,
i.e. use String.format("%,d", decimalNumber) directly, and
by moving something like the byte formatting elsewere inside docs?

And, a general question, why doesn't onionoo use a logging framework?

It doesn't use a logging framework yet, because I didn't look into which framework would work well with our two execution modes: started once per hour by the system cron and running inside Tomcat. If you have suggestions or want to help with patches, awesome!

Also, new ticket? :)

---
Here is the ant task based on docs being independent of util:

  <target name="protocol-api-jar" depends="test">
    <jar destfile="protocol-api.jar"
	 basedir="${classes}"
	 includes="**/docs/" 
	 excludes="**/*Test.class"
	 duplicate="fail">
      <manifest>
	<attribute name="Built-By" value="onionoo project"/>
	<!-- maybe other attributes like the license etc. -->
      </manifest>
    </jar>
  </target>

Or should we include all packages in that jar, not just the docs package? We can still continue working on making docs independent of util, but it seems that a more general Ant target might also be useful for future use cases we didn't think of yet. And the jar shouldn't be that much bigger than with just docs classes.

Also, what do you think about including referenced jars? I don't feel strongly here, so whatever works better for you is fine with me.

comment:5 Changed 5 years ago by karsten

Another thought: if you plan to re-use the docs classes in your tool, we should probably extract interfaces from them. What functionality would you need in those? a) parser from JSON string and b) getters for all fields? (Of course, we'd have to solve the instantiation problem somehow, either by adding a factory, or dependency injection, or ...)

comment:6 Changed 5 years ago by iwakeh

I created two new tickets:

I'll think more about the protocol-api while attempting to integrate it.
Maybe I'll find time to investigate what the other java clients did in order to process onionoo output.

For reproducibility it might be good to add the external jars for the onionoo build to the git repo?

comment:7 in reply to:  6 ; Changed 5 years ago by karsten

Replying to iwakeh:

I created two new tickets:

Thank you!

I'll think more about the protocol-api while attempting to integrate it.
Maybe I'll find time to investigate what the other java clients did in order to process onionoo output.

Sounds good.

For reproducibility it might be good to add the external jars for the onionoo build to the git repo?

Actually, I'd rather not want to add binaries to the Git repo.

As an alternative for reproducibility, do you know Vagrant? I'm using a local Vagrantfile to automatically set up a virtual Debian machine to build and run Onionoo. If you want to give that a try, I'll clean up the Vagrantfile and Puppet files and commit them to Git. Let me know!

comment:8 in reply to:  7 Changed 5 years ago by iwakeh

Replying to karsten:

... As an alternative for reproducibility, do you know Vagrant? I'm using a local Vagrantfile to automatically set up a virtual Debian machine to build and run Onionoo. If you want to give that a try, I'll clean up the Vagrantfile and Puppet files and commit them to Git. Let me know!

I'd like to try Vagrant and it would be great, if you could provide the Onionoo Vagrant file!
Thanks!

comment:9 Changed 5 years ago by karsten

Just committed a Vagrantfile to master. When you try that and have suggestions or ideas to make it better, please open a new ticket for that.

comment:10 in reply to:  9 Changed 5 years ago by iwakeh

Replying to karsten:

Just committed a Vagrantfile to master. When you try that and have suggestions or ideas to make it better, please open a new ticket for that.

Thanks!
I will reserve some time during the weekend to try Vagrant.

comment:11 Changed 5 years ago by iwakeh

About the java api situation/libs in git (might be a little of topic):
Is there a discussion thread about the development guidelines?
I assume, I'm not the first to ask some of these questions.

What are your concerns about binaries in git? I know older vcs couldn't handle
binaries correctly, but a modern git is fine.

Of course, the guidelines for developing tor-services discourage providing
your own libs. But debian java libraries are - in some cases - really ancient.
This might hinder good design which also provides security.

A third party library in git is easier to verify than drawing it from some third party repo.
The involved tor java project would have a clear responsability
and tight control.

Providing all dependencies via git would remove the necessity for debian packages
other than plain java, hence make base system set-up easier.
A java (standalone) service could be provided in one jar which also
simplifies deployment.

In order to only use verified java libs I think it needs some other
mechanism than debian package management.

comment:12 Changed 5 years ago by iwakeh

This is a really long issue discussion, but it might come to a close soon:

little survey of other java/javascript clients

Atlas and Globe use javascript and access the JSON protocol directly;
Globe defines its own javascript objects for the protocol.

The code of Nos Oignons I couldn't find.

AnOnionooid the android java client accesses the protocol
directly using org.json api.

Well, for koninoo it's also direct access and no special objects
for the protocol using javax.json. (And, I think I'll stick with
that in order to keep it simple. In addition java 9 promises a lightweight
json api (in java.util).)

In total, the 'market' for an onionoo-protocol-api is not that big yet.

suggestion

Onionoo could provide an interface api for the onionoo protocol, as you suggested in your comment above.

This would be the more technical java description of the onionoo protocol.
With sufficient Javadoc inside it could replace the contents of protocol.html.

This also provides some more modularization and abstraction, which
could be helpful when switching to a db.
If the need arises, a protocol library could be designed either in the Onionoo project
or be provided by someone else using the interface api.

So, what about creating an interface api for the onionoo protocol
(for example in org.toproject.onionoo.protocol) and providing it as a separate jar?

comment:13 Changed 5 years ago by karsten

Regarding development guidelines: these guidelines were the result of a discussion between sysadmins and developers about two months ago. What you see on that wiki page is the compromise we came up with. I'm aware that Debian packages are sometimes old, but they should at least see all security updates. This is not perfect for software development, but it's what keeps sysadmins happy. And who doesn't want happy sysadmins?

Regarding an interface API: I agree that there's not a huge market yet for a Java library for Onionoo, but that could be a chicken-and-egg problem. Let's provide one. But interfaces alone are not sufficient, we'll also need implementing classes, and we may want to provide some download and caching logic. How about we create a package org.torproject.onionoo.client with the API that a Java Onionoo client would need? That package would depend on the docs package and maybe others. We could provide an Ant target to produce a jar with just the interfaces and classes that a Java Onionoo client needs. Does that make sense? Want to design such an API package?

comment:14 Changed 5 years ago by iwakeh

Well, that might be fun. I'll think about it.

In any case a seperate jar for running the standalone verion should be created.
It's just cleaner to just do java -jar onionoo.jar wherever the jar is.

PS: You're perfectly right about sysadmins :-)

new ticket

#12944 for the client api.

now, this issue could be closed.

Last edited 5 years ago by iwakeh (previous) (diff)

comment:15 Changed 5 years ago by karsten

Resolution: fixed
Status: assignedclosed

Agreed. Closing this ticket, now that the API discussion has moved to #12944. Thanks!

Note: See TracTickets for help on using tickets.