Opened 6 years ago

Closed 3 years ago

#11041 closed enhancement (wontfix)

Add new document type with monthly relay and bridge statistics to Onionoo

Reported by: karsten Owned by:
Priority: Medium Milestone:
Component: Metrics/Onionoo Version:
Severity: Normal Keywords:
Cc: gamambel Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Last week, I talked to Torservers people about generating reports on total traffic relayed by a relay or average number of users of a bridge (related to #10331). I suggested using the data contained in Onionoo's bandwidth documents and yet-to-be-implemented clients documents and aggregating them by month.

While this approach is certainly doable, it has the downside that Onionoo's graph data becomes less detailed over time. Which is okay for graphs, but not nice if one wants a report for 12 or 24 months ago. Also, it's not exactly trivial to transform data from the current format to monthly intervals.

Here's another approach: Onionoo could serve another document which aggregates statistics by month, regardless of how long observations lie in the past. Each data point would start on the first day of a month at 00:00 UTC and end on the last day of the month at 23:59 UTC, except for the first and last data point.

Possible data for this new document type:

  • Bandwidth: how many bytes did the relay or bridge write and read in a month? The document would say when the relay showed up first (which was probably not exactly on the first day of a month at 00:00 UTC), and it would say until when in the current month bandwidth data has been included. Would be available for relays and bridges, though it's probably more important as a success metric for relays than for bridges.
  • Uptime: what fraction of known network statuses contained the relay or bridge with the Running flag? Would be available for relays and bridges.
  • Clients: how many users did the bridge have on average when it reported client statistics? Also, what countries did users come from (only listing countries with at least 1% of users of this bridge), what transports did they use (only included if reported), and what IP versions did they use (only included if reported)? Would only be available for bridges, because directory mirrors and directory guards make it impossible to get these data for single relay.

(We'll want to create three or more child tickets if we agree that this new document type would be useful.)

Would this new document type be useful for Torservers people?

Child Tickets

Change History (4)

comment:1 Changed 6 years ago by karsten

Here's a slightly different plan that I came up with while working on #10331: instead of creating a new document type, we could add new graph lines to existing document types.

Current graph lines contain the following fields:

  • first: datetime of the first data point
  • last: datetime of the last data point
  • interval: interval length in seconds
  • count: number of data points

Example with 5 data points:

        interval
        +-----+           count = 5
        |     |
--------+-----+-----+-----+-----+-------
        0     1     2     3     4
      first                    last

(To be extra precise, the datetime of a data point is the interval midpoint, not the start. A data point represents data between that datetime - interval/2 and datetime + interval/2. This doesn't matter for making graphs, but the new monthly data below would be different.)

The new monthly data line could contain the following fields:

  • start: datetime of the first data point
  • end: datetime of the last data point
  • breaks: time unit when a data point ends and the next begins
  • count: number of data points

Example with 5 data points:

          breaks = "1 month"    count = 5

--------+---+-------+---------+-------+---+------------
        | 0     1        2        3     4 |
      start                              end

In that example, data points 1, 2, and 3 cover full months (note how 2 is a longer month than 1 and 3), and 0 and 4 are only partial months. This shouldn't matter for making tables, but it becomes relevant when people start transforming data, say, to convert total bytes into bytes per time unit.

This is mostly a note for myself and for people interested in seeing progress on this ticket.

comment:2 Changed 6 years ago by karsten

Uptime data is now available in Onionoo (https://onionoo.torproject.org/uptime?search=turtles):

{"relays_published":"2014-03-11 07:00:00",
"relays":[
{"fingerprint":"F397038ADC51336135E7B80BD99CA3844360292B",
"uptime":{
"1_week":{"first":"2014-03-04 07:30:00","last":"2014-03-11 07:30:00","interval":3600,"factor":0.001001001,"count":169,"values":[999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999]},
"1_month":{"first":"2014-02-08 06:00:00","last":"2014-03-11 06:00:00","interval":14400,"factor":0.001001001,"count":187,"values":[999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999]},
"3_months":{"first":"2013-12-09 06:00:00","last":"2014-03-11 06:00:00","interval":43200,"factor":0.001001001,"count":185,"values":[999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,832,582,666,249,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999]},
"1_year":{"first":"2013-03-11 00:00:00","last":"2014-03-10 00:00:00","interval":172800,"factor":0.001001001,"count":183,"values":[999,999,999,999,999,999,978,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,707,457,999,999,999,999,999,999,999,999,999,999,999,999,999,999,978,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,333,478,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,582,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999]},
"5_years":{"first":"2009-03-12 00:00:00","last":"2014-03-06 00:00:00","interval":864000,"factor":0.001001001,"count":183,"values":[994,999,999,999,999,723,999,999,999,999,999,999,999,994,536,472,999,890,994,999,999,999,999,999,986,999,999,999,999,999,999,928,847,932,690,924,969,944,853,974,961,924,857,999,953,999,952,978,965,982,944,778,944,999,718,840,999,999,999,999,999,999,999,999,999,944,999,999,969,999,999,999,985,999,990,832,940,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,957,447,844,951,994,990,999,809,999,999,999,539,471,476,503,632,820,999,999,948,999,749,994,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,749,999,911,999,999,994,999,994,999,999,986,999,999,999,999,994,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,999,832,999,999,994,999,999,999,999,761,999,999,999,999,999,915,999,999,999]}}
}
],
"bridges_published":"2014-03-11 06:37:04",
"bridges":[
]}

Adding a "monthly" data line is the next step.

comment:3 Changed 6 years ago by karsten

Status: newneeds_information

Small note from working on this: I suggested above to add new graph lines to existing document types instead of creating a new document type. On second thought, that might be too confusing for people writing Onionoo clients that display graphs. A separate document type might be less confusing. I started cleaning up the Onionoo codebase to produce different documents from the same data, so that we can easily have a new document type for this.

So, we should brainstorm what data should go into the new monthly statistics document. What are people most interested in?

comment:4 Changed 3 years ago by karsten

Resolution: wontfix
Severity: Normal
Status: needs_informationclosed

It seems nobody really wants this, so it won't happen. Closing.

Note: See TracTickets for help on using tickets.