Opened 2 years ago

Last modified 19 months ago

#24041 assigned enhancement

Unify Metrics' products operational configuration

Reported by: karsten Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by iwakeh)

The method/process of deployment configuration of Metrics' products should be unified.
Afterwards, we look for the 'tool' or lib helping to achieve a simplification.


The thought that triggered this ticket:
Looks like metrics-bot recently switched to using Apache Commons Configuration as configuration library (#23933). We should consider doing the same for other metrics parts, maybe after waiting a bit and hearing how that works out.

Child Tickets

TicketTypeStatusOwnerSummary
#24328enhancementnewmetrics-teamMake db credentials and url as well as paths (webstats, legacy, onionperf etc.) configurable

Change History (8)

comment:1 Changed 2 years ago by iwakeh

Maybe rather summarize this ticket as: "Unify Metrics' products operational configuration"

The method/process of deployment configuration of Metrics' products should be unified.
Afterwards, we look for the 'tool' or lib helping to achieve a simplification.


Some thoughts:

It seems that from all options (hard-coding, command-line parameters, and config files) the config file solution is best. Also, because a configuration file can be checked into a local git on the deployment machine, i.e., less guessing what parameters are needed/ were used.

A configuration file should follow a format/standard. Java properties should be first choice because they are easy to understand and Java provides the appropriate functionality for accessing them. Second choice to make is between simple key-value properties and XML properties. IMO, the simple format ought to be sufficient and is better for 'human consumption'.

CollecTor features the most complex configuration settings of all Metrics' products and is still based on simple key-value Java properties.
All other products (cf. list below) have less configuration demands that could all be met by Java properties.

Tools like commons-configuration provide 99% functionality that is beyond the needs of Metrics' products. Using the functionality from there does not really simplify the code compared to using jdk-provided methods.

Suggestion:

1) Use key value properties files for configuration.
2) Use Java provided functionality for accessing the properties.
3) If there is the need for more sophisticated configuration methods throughout several products, look to provide them in a standardized way (i.e., in a metrics-lib's util package).


Overview of the current state as reference:

  • Onionoo: command line settings and hard coded settings
  • CollecTor: configuration file and Metrics' code for configuration checks and convenience methods incl. provision of a basic config file. Using Java properties format.
  • ExoneraTor: configuration file read as a file (back end) and system properties (web part).
  • Web: mixture off all.
  • Bot: Apache Commons Configuration as configuration library (#23933).

comment:2 Changed 2 years ago by karsten

Agreed, this is what I meant. No need to add another dependency if we can solve the problem with the JDK alone.

irl, what was the reason that you switched to Apache Commons Configuration? What parts do you use that are harder or impossible to do with Java properties?

comment:3 Changed 2 years ago by iwakeh

Description: modified (diff)
Summary: Add Apache Commons Configuration as configuration libraryUnify Metrics' products operational configuration

Changing the ticket's summary and description a little to reflect the actual topic.

comment:4 Changed 19 months ago by iwakeh

Owner: changed from metrics-team to iwakeh
Status: newaccepted

comment:5 Changed 19 months ago by irl

Java properties didn't really provide the hierarchical structure to the configuration that is needed for the bot. There are types of accounts, and then there can be 0 or more of each type, and then those have properties within them.

An example XML file is now included in the documentation:

https://people.torproject.org/~irl/metrics-bot/org/torproject/metrics/bot/Configuration.html

If all you need are strings and lists then properties is fine, but once you get into lists of lists it gets a lot more difficult and you have to start working out your own syntax for separating items, etc.

Commons Configuration is in Debian as libcommons-configuration-java (1.10 in both testing and stable).

comment:6 Changed 19 months ago by irl

Cc: metrics-team added

metrics-team was not added to cc

comment:7 Changed 19 months ago by karsten

Sounds like a good topic for the next team meeting.

comment:8 Changed 19 months ago by iwakeh

Cc: metrics-team removed
Owner: changed from iwakeh to metrics-team
Status: acceptedassigned

Most Metrics' products, like Onionoo, Exonerator, and Web don't even have the single file as configuration option (cf. comment:1 overview).

It seems we all agree on the goal to have one file that configures each product's operational options.

Both Bot and CollecTor are at that stage. Onionoo, Exonerator, and Web need to be 'pushed' to that level of configuration, which will be more work in making the various settings accessible in one configuration file. AfaIk, simple Java properties suffice, which could come in either properties layout or simple XML; and could be either implemented in plain Java or using commons configuration.

The focus is the operational advantage, which will consist in having the 'single point of configuration' and we should just make sure to choose the same kind of implementation approach for Onionoo, Exonerator, and Web; Bot and CollecTor work as they are. Implementing the configuration choice for these two is a secondary (more long-term) issue, because both work fine as they are.

Another question is what all should be configurable? I.e., this ticket should be also used for checking what values, e.g., working directories, base URLs etc., ought to be configurable and are maybe are not yet.

Note: See TracTickets for help on using tickets.