Opened 4 years ago

Last modified 3 years ago

#20518 assigned enhancement

Make various architecture improvements and modernizations

Reported by: iwakeh Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics/CollecTor Version:
Severity: Normal Keywords: metrics-2018
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:


This is the parent ticket for further architecture improvements and modernization of the CollecTor code-base.

Child Tickets

#8799enhancementclosediwakehcollector's downloads: avoid httpurl-connection
#20350enhancementassignedmetrics-teamReplace shell script with Java module
#20351enhancementclosedmetrics-teamTurn the updateindex module into a function that runs after each module run
#20519enhancementassignedmetrics-teamUntangle and modernize relaydescs module
#20542enhancementassignedmetrics-teamStructure and modernize bridgedescs module
#20543enhancementassignedmetrics-teamRestructure exitlists module
#20546enhancementassignedmetrics-teamImplement CleanUtils class for common file system operations
#21219enhancementassignedmetrics-teamRemove old descriptor files from out/ after archiving

Change History (7)

comment:1 Changed 4 years ago by iwakeh

#8799 already describes the download modernization.

comment:2 Changed 4 years ago by iwakeh

Further goals are

1) to replace the tarball-script with a Java module of CollecTor and
2) to make index.json as accurate as possible.

For 2) it might be useful to remove the separately configurable create-index module. The index should just be created after each (major) change, e.g. any download or sync module run.

The tarball creation could run as a continuous background process that is stops during sync and download runs, i.e. basically 'stays out of the way' when other modules run. It also relates to the index creation as it changes the 'archive' path. Maybe, trigger an index-run after a substantial change?

Other thoughts and general goals?

comment:3 Changed 4 years ago by iwakeh

Addendum to comment:1:

  • descriptor writing should only be performed by the 'o.t.c.persist' package
  • One class of each module should extend CollecTorMain. It should be named <Module>Main, e.g. RelayMain or BridgeMain, and contain the code for structuring the modules work, i.e. no detailed functionality.
  • The other classes of each module should provide the actual module functionality.

comment:4 Changed 4 years ago by iwakeh

Addendum comment:1:
Several modules perform some sort of cleaning, i.e. erase files based on their age. This should be offered by a utility class/package.

Added child ticket for CleanUtils.

This affects relay, bridge, exitlists, and torperf.

Last edited 4 years ago by iwakeh (previous) (diff)

comment:5 Changed 3 years ago by karsten

Summary: CollecTor: architecture improvements and modernizationMake various architecture improvements and modernizations

Tweak summary.

comment:6 Changed 3 years ago by karsten

Keywords: metrics-2018 added

comment:7 Changed 3 years ago by karsten

Owner: set to metrics-team
Status: newassigned
Note: See TracTickets for help on using tickets.