Opened 10 years ago

Closed 10 years ago

#2627 closed enhancement (implemented)

Come up with a better interface between metrics-db and metrics-web

Reported by: karsten Owned by: karsten
Priority: Medium Milestone:
Component: Metrics/CollecTor Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:


We should rethink the interface between metrics-db and metrics-web to make it easier for other people to create a better metrics or TorStatus-like website.

The current distinction is mostly that metrics-db contains all parts that need to be run periodically, triggered by the operating system, whereas metrics-web has the code to display the website, triggered by web users. The main interface between the two is the database schema.

It seems that the current distinction is not ideal, because changes to the website affect both metrics-db (for the database schema) and metrics-web (for the display parts). We should move the database to metrics-web and not make it the interface between metrics-db and metrics-web. A better interface would be the various descriptor formats that we use for metrics. Having the files as the interface also makes more sense, because we don't provide a database to the world anyway, but tarballs. If someone wants to set up a metrics-website, they need to set up metrics-db to import descriptors into a database and metrics-web for the website part. It would be better to give them the tarballs and just metrics-web. If they don't like their metrics website, they can easily change or replace metrics-web and do their own thing.

Here's the proposed distinction: metrics-db shall do all the data processing and sanitizing and metrics-web shall import descriptors into a database and run the website. In particular, they would perform the following tasks:

  • metrics-db
    • read relay descriptors from a local Tor data directory and/or weasel's directory archive script, and download missing relay descriptors from the directory authorities
    • read and sanitize bridge descriptors and bridge pool assignments
    • download and archive exit lists
    • download and archive GetTor files
    • download and archive Torperf files
    • write all output files to the local file system
  • metrics-web
    • read input files from the local file system
    • import relay descriptors into a database
    • provide relay search capability
    • provide ExoneraTor capability
    • process relay and bridge descriptors for daily user statistics
    • process Torperf files for Torperf statistics
    • process GetTor files for GetTor statistics
    • display customizable graphs
    • provide CSV files
    • make tarballs for download
    • analyze current consensus and votes for possible voting process problems
    • provide list of currently running relay

If we want to implement this ticket, a fair amount of code will move from metrics-db to metrics-web. I might start on this unless someone tells me it's a really stupid idea.

Child Tickets

Change History (6)

comment:1 Changed 10 years ago by karsten

Status: newassigned

Here's a list of substeps that are necessary to implement this ticket:

  1. Copy the Java parts from metrics-db to metrics-web that are required to run stuff periodically. These parts include the classes Main, Configuration, LockFile, and LoggingConfiguration. Remove most parts of Main and Configuration in metrics-web, because there's nothing to do for a periodically running metrics-web yet. Add parts when needed.
  2. Copy ArchiveReader, RelayDescriptorParser, and BridgeDescriptorParser and move RelayDescriptorDatabaseImporter, SanitizedBridgesReader, BridgeStatsFileHandler, and ConsensusStatsFileHandler from metrics-db to metrics-web. These classes are required to process relay and bridge descriptors and prepare them for being displayed on the metrics website.
  3. Move ConsensusHealthChecker from metrics-db to metrics-web.
  4. Split GetTorProcessor into the part that downloads GetTor's statistics file and the part that imports GetTor statistics into a database. Move the database import part to metrics-web.
  5. Move TorperfProcessor from metrics-db to metrics-web. metrics-db doesn't process Torperf files at all right now. The only processing that takes place is calculating daily medians and quartiles, which is only necessary for displaying graphs.

It should be safe to implement substeps 2 to 5 in any order once 1 is implemented.

comment:2 in reply to:  1 Changed 10 years ago by karsten

Replying to karsten:

Here's a list of substeps that are necessary to implement this ticket:

Steps 1 and 3 from this list are implemented.

comment:3 Changed 10 years ago by karsten

The first half of 2 (importing relay descriptors) is implemented, too. What's missing from 2 is the part that generates bridge statistics.

comment:4 Changed 10 years ago by karsten

The second half of 2 is implemented. 4 and 5 remain.

comment:5 Changed 10 years ago by karsten

5 is done. 4 remains.

comment:6 Changed 10 years ago by karsten

Resolution: implemented
Status: assignedclosed

4 is implemented. Both repositories may require some cleanup, but the bulk of the work is done. Closing.

Note: See TracTickets for help on using tickets.