Opened 8 months ago

Last modified 2 months ago

#29005 assigned enhancement

PrivCount proof of concept: implement consumed bandwidth counters

Reported by: teor Owned by:
Priority: Medium Milestone: Tor: 0.4.2.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: privcount, 040-unreached-20190109, 041-accepted-20190115, network-team-unreached-2019-Q1Q2
Cc: irl Actual Points: 0.4
Parent ID: #27908 Points: 2
Reviewer: Sponsor:

Description

We want to add a copy of the current consumed bandwidth statistic to PrivCount:

  • consumed bandwidth by relay flags

https://metrics.torproject.org/bandwidth-flags.html

We can defer these statistics, because they are not as sensitive to manipulation by hostile clients or services:

  • Bandwidth spent on answering directory requests
  • Fraction of connections used uni-/bidirectionally

We will need counters for:

  • read-history
  • write-history

Split each counter into 4 (G, E, G and E, not G and not E) using these flags:

  • Guard
  • Exit and not BadExit

We can do the G/E split on the Data Collector (DC).
We can also do a smaller check aggregation for each group on the Tally Reporter (TR).

Check that relays are Running.
We can do the Running check on the DC.
We should also check Running on the TR, and exclude DCs that weren't Running at all during the period.

How often should we update the bandwidth counters?

If we update counters at the end of the day, we can match the current statistics exactly:
"we only include bandwidth histories for a given day if a relay was listed as running in a consensus at least once on that day. We attribute bandwidth to guards and/or exits if a relay was a guard and/or exit at least in one consensus on a day."
But we would be storing some sensitive data in memory for the whole day.

Instead, we could update the counters whenever we queue data, based on the flags in the current consensus we have. (The difference will likely be minimal.)

If updating the counters multiple times per second is too CPU-intensive, we can update them every few seconds. If that's too often, we can update them just before we delete an old consensus.

Sources:
https://metrics.torproject.org/reproducible-metrics.html#traffic
https://trac.torproject.org/projects/tor/wiki/org/meetings/2018MexicoCity/Notes/PrivCount

Child Tickets

TicketTypeStatusOwnerSummary
#24250defectnewIn a private network some relays advertise zero bandwidth-observed
#29019enhancementneeds_revisionMake relays report bandwidth usage more often in test networks

Change History (17)

comment:1 Changed 8 months ago by teor

Type: defectenhancement

These should all be enhancements

comment:2 Changed 8 months ago by teor

Points: 2

If we hard-code a lot of things, this should not take more than a few days, depending on how many changes we need to make to the existing tor bandwidth reporting code.

comment:3 Changed 8 months ago by teor

Actual Points: 0.4

I have a proof-of-concept for PrivCount in extra-info descriptors in https://github.com/torproject/tor/pull/640

Let's review it in #29004.

comment:4 Changed 8 months ago by teor

Keywords: 040-unreached-20190109 041-proposed added
Milestone: Tor: 0.4.0.x-finalTor: unspecified

These tasks are on the 0.4.1 roadmap for PrivCount and Sponsor V.

comment:5 Changed 8 months ago by teor

Keywords: 041-proposed-on-roadmap added; 041-proposed removed
Milestone: Tor: unspecifiedTor: 0.4.1.x-final

Let's review these tickets at the next meeting using our 041-proposed process.

They're on the roadmap, so the review should focus on ticket size and team capacity (and sponsor expectations).

comment:6 Changed 7 months ago by teor

Keywords: 041-accepted-20190115 added; 041-proposed-on-roadmap removed

These PrivCount tickets are on the 041 roadmap, we accepted their points estimates in 041 without discussion.

comment:7 Changed 7 months ago by nickm

Sponsor: SponsorV

comment:8 Changed 6 months ago by teor

Keywords: network-team-roadmap-2019-Q1Q2 added

This is part of an existing roadmap task.

comment:9 Changed 5 months ago by teor

Status: assignedneeds_revision

I have part of a patch for this ticket, but it needs to be revised after #29018 and #29019, and with #29005.

Last edited 5 months ago by teor (previous) (diff)

comment:10 Changed 5 months ago by teor

Keywords: backlog added

comment:11 Changed 4 months ago by teor

I have incomplete branches for #29004, #29005, and #29019 here:
https://github.com/teor2345/tor/tree/ticket29004-wip
https://github.com/teor2345/tor/tree/ticket29005
https://github.com/teor2345/tor/tree/ticket29019

I think all the necessary code is present in these branches.

But it needs some cleanup:

  • rebase on to the current master,
  • put the commits on the right branches
  • make sure it does what these tickets say it should do

I'm happy to do that after I come back from leave. I am also happy if Nick wants to clean up this code.

comment:12 Changed 3 months ago by nickm

Milestone: Tor: 0.4.1.x-finalTor: 0.4.2.x-final

Move privcount tickets to 0.4.2

comment:13 Changed 3 months ago by teor

Keywords: network-team-roadmap-unreached-2019-Q1Q2 added; backlog network-team-roadmap-2019-Q1Q2 removed

comment:14 Changed 3 months ago by teor

Keywords: network-team-unreached-2019-Q1Q2 added; network-team-roadmap-unreached-2019-Q1Q2 removed

comment:15 Changed 3 months ago by teor

Owner: teor deleted
Status: needs_revisionassigned

comment:16 Changed 3 months ago by gaba

Sponsor: SponsorV

comment:17 Changed 2 months ago by irl

Cc: irl added
Note: See TracTickets for help on using tickets.