#24036 closed enhancement (fixed)

Create a unified package naming scheme for all metrics code bases

Reported by: karsten Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics Version:
Severity: Normal Keywords:
Cc: irl Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

The discussion behind #24028 made me think whether we could find a unified package naming scheme for all metrics code bases.

I reserve the right to come up with an even smarter plan after today, but sharing my thoughts here might allow us to come up with an even smarterer plan together.

Current code base Current package and classes, starting with org.torproject. Suggested package, starting with org.torproject.metrics. Suggested code base, only if different
CollecTor collector (1) collector
CollecTor collector.bridgedescs (3) collector.bridgedescs
CollecTor collector.conf (5) collector.conf
CollecTor collector.cron (3) collector.cron
CollecTor collector.exitlists (1) collector.exitlists
CollecTor collector.index (1) index metrics-lib (*1)
CollecTor collector.onionperf (2) collector.onionperf
CollecTor collector.persist (12) collector.persist
CollecTor collector.relaydescs (6) collector.relaydescs
CollecTor collector.sync (4) collector.sync
ExoneraTor exonerator.ExoneraTorDatabaseImporter exonerator
ExoneraTor exonerator.ExoneraTorServlet web metrics-web (*2)
ExoneraTor exonerator.QueryServlet exonerator
ExoneraTor exonerator.QueryResponse exonerator metrics-lib (*3)
metrics-lib descriptor (28) descriptor
metrics-lib descriptor.impl (31) descriptor.impl
metrics-lib descriptor.index (4) index (*4)
metrics-lib descriptor.internal (1) util (*5)
metrics-web metrics.advbwdist (1) stats.advbwdist
metrics-web metrics.clients (1) stats.clients
metrics-web metrics.collectdescs (1) stats.collectdescs
metrics-web metrics.connbidirect (1) stats.connbidirect
metrics-web metrics.hidserv (11) stats.hidserv
metrics-web ernie.cron (6) stats.servers (*6)
metrics-web metrics.onionperf (1) stats.onionperf
metrics-web metrics.webstats (1) stats.webstats
metrics-web metrics.web (27) web
metrics-web metrics.web.graphs (7) web.graphs
metrics-web metrics.web.research (1) web (*7)
Onionoo onionoo.cron (1) onionoo.cron
Onionoo onionoo.docs (21) onionoo.docs metrics-lib (*8)
Onionoo onionoo.server (15) onionoo.server
Onionoo onionoo.updater (18) onionoo.updater
Onionoo onionoo.util (1) util metrics-lib (*9)
Onionoo onionoo.writer (8) onionoo.writer

(*1) Right now, only CollecTor produces an index.json file, but with #23285 we might extend that to metrics-web producing an index.json file of available stats files.

(*2) This is part of moving all presentation things to metrics-web, including ExoneraTor (#23549), and once we do that we'll have to write and read the QueryResponse from two different code bases.

(*3) See (*2).

(*4) These classes don't depend on the descriptor-parsing part of metrics-lib but are generally useful for other metrics code bases.

(*5) See (*4), though we might rename "internal" to "util" to focus what this package is for (utility classes) rather than who it's not for (external folks).

(*6) The old package name is so legacy, let's take the opportunity to pick a more useful name now. It's all about servers and bandwidth (provided by servers).

(*7) This servlet class is the last survivor of the "research" package and could as well join the other servlets.

(*8) Moving the docs to metrics-lib would allow metrics-bot and other Java Onionoo clients to use Onionoo documents without writing their own parser.

(*9) We could gather all utility-type classes that are truly independent of their respective code bases in metrics-lib.

Child Tickets

Change History (14)

comment:1 Changed 13 months ago by iwakeh

org.torproject.collector.index.CreateIndexJson currently uses org.torproject.descrptor.index in order to regularly provide the new index files. Thus, this is truly CollecTor functionality.

comment:2 Changed 13 months ago by irl

metrics-bot looks like:

Current code base Current package and classes, starting with org.torproject. Suggested package, starting with org.torproject.metrics. Suggested code base, only if different
metricsbot Core metrics-bot application bot
metricsbot.factoids Factoids for interactive components bot.factoids
metricsbot.irc IRC Component for metrics-bot bot.irc
metricsbot.mastodon Mastodon Component for metrics-bot bot.mastodon
metricsbot.microblog Generic microblogging interfaces and utilities bot.microblog
metricsbot.tor Tor network objects (relays, bridges, flags, etc.) onionoo.docs metrics-lib
metricsbot.twitter Twitter Component for metrics-bot bot.twitter

Currently metricsbot.tor does contain some metrics-bot specific code relating to generating microblog statuses, but this code should be moved into bot.microblog instead and then we can use Onionoo documents from metrics-lib.

It might also be desirable to move MicroblogUtils to metrics-lib.

comment:3 in reply to:  2 Changed 13 months ago by iwakeh

Replying to irl:

metrics-bot looks like:

Current code base Current package and classes, starting with org.torproject. Suggested package, starting with org.torproject.metrics. Suggested code base, only if different
metricsbot Core metrics-bot application bot

[snip]

metricsbot.tor Tor network objects (relays, bridges, flags, etc.) onionoo.docs metrics-lib

[snip]

Currently metricsbot.tor does contain some metrics-bot specific code relating to generating microblog statuses, but this code should be moved into bot.microblog instead and then we can use Onionoo documents from metrics-lib.

metrics-lib won't know about Onionoo, because Onionoo is a client far up the ladder using metrics-lib. Thus, metrics-bot will only internally be able to use part of Onionoo's API. Actually, long ago I proposed to provide an Onionoo Java API for reading Onionoo's JSon documents. That would now make more sense with metrics-bot and prevent updating in two or more places.

It might also be desirable to move MicroblogUtils to metrics-lib.

These are only two very small methods so far. Maybe, they could find a home elsewhere or are offered somewhere anyway?

While taking a look at metrics-bot git repo I noticed that metrics-bot doesn't comply yet to Metrics' project guidelines and java style rules (like using metrics-base etc.). Metrics-web and ExoneraTor still need to be migrated, but new Metrics products should comply initially. New ticket?

comment:4 Changed 13 months ago by iwakeh

As I mentioned this in comment:3 above where it came to mind again, but it really is a topic unrelated to metrics-bot except that metrics-bot would be a first client: we should consider supplying an Onionoo JSON API.
As a first step only internally, i.e., for metrics-bot, and later maybe public.

Last edited 13 months ago by iwakeh (previous) (diff)

comment:5 Changed 13 months ago by irl

Cc: irl added

comment:6 Changed 13 months ago by karsten

This seems like a fine topic for a pad discussion. Just some quick thoughts here:

  • iwakeh, regarding your first comment, I agree that, right now, CollecTor is the only application that creates index files. My suggesion was to move that functionality to metrics-lib in order to use it in other applications, too, in this case metrics-web. See my (*1) above.
  • irl and iwakeh, I think you're talking about the same thing, which is an Onionoo JSON API. That's what I meant in my (*8) above, and I'm aware that we discussed this long ago and dropped it on the floor. It might make more sense now. We just need to be very clear that providing another API in metrics-lib means we cannot make changes as quickly as we can right now. Major releases of metrics-lib and 1-month delays required for backward-incompatible changes, and all that. Still worth discussing if it helps avoid writing code twice.

Feel free to add (many) more thoughts, but let's also think about switching to a more synchronous way to discuss this topic.

comment:7 in reply to:  6 ; Changed 13 months ago by iwakeh

Replying to karsten:

This seems like a fine topic for a pad discussion. Just some quick thoughts here:

Yes, that would make sense. We should collect and sort a little more on the ticket here before using the pad, though. That gives more time to think through suggestions more thoroughly, as you also suggested.

  • iwakeh, regarding your first comment, I agree that, right now, CollecTor is the only application that creates index files. My suggesion was to move that functionality to metrics-lib in order to use it in other applications, too, in this case metrics-web. See my (*1) above.

Yes, I understand that. What I tried to say above is that Collector's index package only contains code regularly producing index files, i.e., only code that facilitates configuration and scheduling. The index-file generation is based on metrics-lib already. Thus, no code needs to be moved from CollecTor. metrics-lib contains all index-json producing code and would be ready to be used elsewhere as is.

  • irl and iwakeh, I think you're talking about the same thing, which is an Onionoo JSON API. That's what I meant in my (*8) above, and I'm aware that we discussed this long ago and dropped it on the floor. It might make more sense now. We just need to be very clear that providing another API in metrics-lib means we cannot make changes as quickly as we can right now. Major releases of metrics-lib and 1-month delays required for backward-incompatible changes, and all that. Still worth discussing if it helps avoid writing code twice.

Yes, but the question would also be where to add it. Onionoo or metrics-lib? (for the pad discussion)

comment:8 in reply to:  7 Changed 13 months ago by karsten

Replying to iwakeh:

Yes, I understand that. What I tried to say above is that Collector's index package only contains code regularly producing index files, i.e., only code that facilitates configuration and scheduling. The index-file generation is based on metrics-lib already. Thus, no code needs to be moved from CollecTor. metrics-lib contains all index-json producing code and would be ready to be used elsewhere as is.

Ah, okay. Makes sense.

Yes, but the question would also be where to add it. Onionoo or metrics-lib? (for the pad discussion)

Agreed, fine question! My idea was to stuff anything into metrics-lib that will be shared, to have a single dependency. But I also thought about keeping things in Onionoo and depending on it. However, that latter approach might not scale, with metrics-web then depending on metrics-lib, Onionoo, ExoneraTor, and so on. But: pad discussion!

comment:9 Changed 12 months ago by iwakeh

Also refer to the second part of this mail, see "A Unified Java Package Naming Scheme" for further clarification and decisions.

comment:10 Changed 12 months ago by karsten

Status: newneeds_review

Looks like we mostly agree about the changes above. Time to summarize and outline some next steps to move forward. Each of the following could become its own ticket:

  1. Rename CollecTor packages
    • Rename root package org.torproject.collector to org.torproject.metrics.collector to make it part of the Tor Metrics name space.
    • Further rename index package to indexer to reflect that this is the scheduled task for creating index files, not the classes for generating or parsing those index files.
  2. Rename ExoneraTor packages
    • Rename root package org.torproject.exonerator to org.torproject.metrics.exonerator to make it part of the Tor Metrics name space.
    • Reconsider moving some classes to metrics-lib for re-use from metrics-web, but not before the Jetty switch and possibly related changes are over.
  3. Rename metrics-lib packages
    • Rename root package org.torproject.descriptor to org.torproject.metrics.descriptor to make it part of the Tor Metrics name space.
    • org.torproject.descriptor.index becomes org.torproject.metrics.index (without descriptor), because it contains functionality that is supposed to be internally shared with other code bases that is unrelated to descriptor parsing.
  4. Create new metrics-lib package with generally useful classes and functions
    • At some point in the future we may add a new package org.torproject.metrics.util with generally useful classes and functions, including fingerprint calculations and commonly used regular expressions.
  5. Rename metrics-web packages
    • Rename root package org.torproject.metrics to org.torproject.metrics.stats to make it part of the Tor Metrics name space.
    • ernie.cron becomes org.torproject.metrics.stats.servers to get rid of the legacy name and to reflect that it's all about servers and bandwidth (provided by servers).
    • org.torproject.metrics.web and subpackages remain unchanged.
    • org.torproject.metrics.web.research, containing just a single class, is merged into org.torproject.metrics.web.
  6. Rename Onionoo packages
    • Rename root package org.torproject.onionoo to org.torproject.metrics.onionoo to make it part of the Tor Metrics name space.
  7. Make Onionoo's document classes available as part of metrics-lib
    • At some point in the future we may want to move parts of org.torproject.onionoo.docs to metrics-lib, so that metrics-bot and other clients can re-use that code. More specifically, we should only move the externally provided JSON objects, not the internally used status objects.
  8. Rename metrics-bot packages
    • Rename root package org.torproject.metricsbot to org.torproject.metrics.bot (every dot counts!) to make it part of the Tor Metrics name space.

What did I miss? Should I move forward with creating these tickets and preparing patches?

comment:11 Changed 12 months ago by irl

Looks good to me. I don't think anything is missed here.

comment:12 Changed 12 months ago by iwakeh

Looks fine. Some suggestions (that might have been intended, but are not listed explicitly):

1) The tickets for renaming packages should be tickets on their own.
2) The update of all depending products to the latest metrics-lib are the next round of tickets;
3) next badge are all tickets moving code between products or creating new functionality, e.g., creating a util package or Onionoo's JSON API etc. Here an internal order needs to be defined , too.

Numbers indicate a suggested order for tackling the tasks&tickets.

Last edited 12 months ago by iwakeh (previous) (diff)

comment:13 in reply to:  12 Changed 12 months ago by karsten

Replying to iwakeh:

Looks fine. Some suggestions (that might have been intended, but are not listed explicitly):

Good points!

1) The tickets for renaming packages should be tickets on their own.

Yes, that's what I meant: 8 new tickets. Otherwise this ticket will become crazy!

2) The update of all depending products to the latest metrics-lib are the next round of tickets;

Right. This is also related to #24028, because we wanted to take this opportunity to redesign some of the descriptor interfaces. I guess that means only 7 new tickets and adding a comment to #24028.

3) next badge are all tickets moving code between products or creating new functionality, e.g., creating a util package or Onionoo's JSON API etc. Here an internal order needs to be defined , too.

Yep. Shouldn't be that difficult, though. We'll build something new in metrics-lib and then switch over to using that in dependent code bases.

Numbers indicate a suggested order for tackling the tasks&tickets.

Makes sense.

I'll create 7 new tickets now and also add a comment to #24028.

comment:14 Changed 12 months ago by karsten

Resolution: fixed
Status: needs_reviewclosed

There, I just created #24291, #24292, #24293, #24294, #24295, #24296, and #24297, and I updated #24028. I think that concludes this ticket, as all actual work parts are now moved to other tickets. Closing. Thanks for the discussion here and on the pad!

Note: See TracTickets for help on using tickets.