Opened 3 years ago

Last modified 4 weeks ago

#24041 assigned enhancement

Unify Metrics' products operational configuration

Reported by: karsten Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics Version:
Severity: Normal Keywords: metrics-team-roadmap-2020
Cc: Actual Points:
Parent ID: Points: 2
Reviewer: Sponsor:

Description (last modified by iwakeh)

The method/process of deployment configuration of Metrics' products should be unified.
Afterwards, we look for the 'tool' or lib helping to achieve a simplification.


The thought that triggered this ticket:
Looks like metrics-bot recently switched to using Apache Commons Configuration as configuration library (#23933). We should consider doing the same for other metrics parts, maybe after waiting a bit and hearing how that works out.

Child Tickets

TicketTypeStatusOwnerSummary
#24328enhancementnewmetrics-teamMake db credentials and url as well as paths (webstats, legacy, onionperf etc.) configurable

Attachments (1)

metrics-configuration-overview-2020-04-12.pdf (66.8 KB) - added by karsten 7 weeks ago.

Download all attachments as: .zip

Change History (13)

comment:1 Changed 3 years ago by iwakeh

Maybe rather summarize this ticket as: "Unify Metrics' products operational configuration"

The method/process of deployment configuration of Metrics' products should be unified.
Afterwards, we look for the 'tool' or lib helping to achieve a simplification.


Some thoughts:

It seems that from all options (hard-coding, command-line parameters, and config files) the config file solution is best. Also, because a configuration file can be checked into a local git on the deployment machine, i.e., less guessing what parameters are needed/ were used.

A configuration file should follow a format/standard. Java properties should be first choice because they are easy to understand and Java provides the appropriate functionality for accessing them. Second choice to make is between simple key-value properties and XML properties. IMO, the simple format ought to be sufficient and is better for 'human consumption'.

CollecTor features the most complex configuration settings of all Metrics' products and is still based on simple key-value Java properties.
All other products (cf. list below) have less configuration demands that could all be met by Java properties.

Tools like commons-configuration provide 99% functionality that is beyond the needs of Metrics' products. Using the functionality from there does not really simplify the code compared to using jdk-provided methods.

Suggestion:

1) Use key value properties files for configuration.
2) Use Java provided functionality for accessing the properties.
3) If there is the need for more sophisticated configuration methods throughout several products, look to provide them in a standardized way (i.e., in a metrics-lib's util package).


Overview of the current state as reference:

  • Onionoo: command line settings and hard coded settings
  • CollecTor: configuration file and Metrics' code for configuration checks and convenience methods incl. provision of a basic config file. Using Java properties format.
  • ExoneraTor: configuration file read as a file (back end) and system properties (web part).
  • Web: mixture off all.
  • Bot: Apache Commons Configuration as configuration library (#23933).

comment:2 Changed 3 years ago by karsten

Agreed, this is what I meant. No need to add another dependency if we can solve the problem with the JDK alone.

irl, what was the reason that you switched to Apache Commons Configuration? What parts do you use that are harder or impossible to do with Java properties?

comment:3 Changed 3 years ago by iwakeh

Description: modified (diff)
Summary: Add Apache Commons Configuration as configuration libraryUnify Metrics' products operational configuration

Changing the ticket's summary and description a little to reflect the actual topic.

comment:4 Changed 2 years ago by iwakeh

Owner: changed from metrics-team to iwakeh
Status: newaccepted

comment:5 Changed 2 years ago by irl

Java properties didn't really provide the hierarchical structure to the configuration that is needed for the bot. There are types of accounts, and then there can be 0 or more of each type, and then those have properties within them.

An example XML file is now included in the documentation:

https://people.torproject.org/~irl/metrics-bot/org/torproject/metrics/bot/Configuration.html

If all you need are strings and lists then properties is fine, but once you get into lists of lists it gets a lot more difficult and you have to start working out your own syntax for separating items, etc.

Commons Configuration is in Debian as libcommons-configuration-java (1.10 in both testing and stable).

comment:6 Changed 2 years ago by irl

Cc: metrics-team added

metrics-team was not added to cc

comment:7 Changed 2 years ago by karsten

Sounds like a good topic for the next team meeting.

comment:8 Changed 2 years ago by iwakeh

Cc: metrics-team removed
Owner: changed from iwakeh to metrics-team
Status: acceptedassigned

Most Metrics' products, like Onionoo, Exonerator, and Web don't even have the single file as configuration option (cf. comment:1 overview).

It seems we all agree on the goal to have one file that configures each product's operational options.

Both Bot and CollecTor are at that stage. Onionoo, Exonerator, and Web need to be 'pushed' to that level of configuration, which will be more work in making the various settings accessible in one configuration file. AfaIk, simple Java properties suffice, which could come in either properties layout or simple XML; and could be either implemented in plain Java or using commons configuration.

The focus is the operational advantage, which will consist in having the 'single point of configuration' and we should just make sure to choose the same kind of implementation approach for Onionoo, Exonerator, and Web; Bot and CollecTor work as they are. Implementing the configuration choice for these two is a secondary (more long-term) issue, because both work fine as they are.

Another question is what all should be configurable? I.e., this ticket should be also used for checking what values, e.g., working directories, base URLs etc., ought to be configurable and are maybe are not yet.

comment:9 Changed 2 months ago by gaba

Actual Points: 6
Keywords: metrics-team-roadmap-2020April added
Points: 8

comment:10 Changed 2 months ago by gaba

Actual Points: 6
Points: 82

Changed 7 weeks ago by karsten

comment:11 Changed 7 weeks ago by karsten

I made some progress here by creating an overview of configuration options in our various code bases. This isn't really ready for review yet, but I'm planning to do something else next week, so I'm putting it here for picking it up later. It's the attached PDF.

I also made a couple comments while putting together this overview:

  • It would be nice to have a single naming scheme for options. Right now we have ALLCAPITAL, CamelCase, dotted.notation, and --two-dashes.
  • It would be nice to have a single way of specifying options, rather than having CollecTor's collector.properties file, ExoneraTor‘s config file, Onionoo‘s command line arguments, and some -Djava.properties.
  • The metrics-web updater is lacking options to disable or select single modules. Right now this requires a temporary code change.
  • The metrics-web updater is also lacking an option to disable just the module that fetches descriptors from CollecTor. Right now we're removing a line in the source code when running integration tests.
  • The Onionoo updater is lacking an option to disable rDNS lookups, which is also something where we're removing some code when running integration tests.

comment:12 Changed 4 weeks ago by gaba

Keywords: metrics-team-roadmap-2020 added; metrics-team-roadmap-2020April removed

We need to review all this tickets for metrics roadmap.

Note: See TracTickets for help on using tickets.