Opened 2 years ago

Last modified 18 months ago

#24674 new task

Bandwidth authorities should use geographically distributed bandwidth servers

Reported by: teor Owned by: tom
Priority: Medium Milestone:
Component: Core Tor/Torflow Version:
Severity: Normal Keywords: tor-bwauth
Cc: ln5, juga Actual Points:
Parent ID: #24499 Points: 3
Reviewer: Sponsor:

Description

When a bandwidth authority is configured with a nearby bandwidth server, it measures nearby relays really high.

Its measurements for those relays don't help "to balance load across the network such that a user can expect to have the same average stream capacity regardless of path".
https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/README.spec.txt#n353

Instead, bandwidth authorities should use a geographically distributed set of bandwidth servers. We can do this by:

  • each operator setting up their own set of geographically distributed bandwidth servers
  • using one or more CDNs (#24506)
  • sharing the current bandwidth servers between bandwidth authorities

Ideally, every bandwidth authority would use the same set of bandwidth servers. That would eliminate one source of bias.

Child Tickets

TicketStatusOwnerSummaryComponent
#24506needs_informationtomMove some bandwidth authority servers to a CDNCore Tor/Torflow
#24730newtomClarify the bandwidth authority spec to include client and server/service pathsCore Tor/Torflow
#24834newmetrics-teamMap consensus weight vs bandwidth for each bandwidth authority's votesMetrics/Relay Search

Change History (7)

comment:1 Changed 2 years ago by ln5

Cc: ln5 added

comment:2 Changed 2 years ago by mikeperry

We discussed this on tor-internal two years ago. The conclusion we came to then was that the simplest (and best) plan is to make the list of URLs in https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/bwauthority_child.py#n51 contain a handful of fast, geographically distributed mirrors.

This way, all bw authorities will use the same file servers, and the results that they produce for each relay should be an approximate arithmetic mean of its performance with each bandwidth file server (this is approximate because we do some measurement filtering, and because the choice of url is random, not strictly round robin: https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/bwauthority_child.py#n131).

Then, for more diversity, we can move individual bw auths to geographically diverse locations. The result of this is that the directory authorities will take the median of their votes for each relay.

The key advantage between this plan and just having each bw auth pick its own file server location is that we avoid having degenerate pairing effects of fast location to fast location, and we also avoid the need to do the full pairing combinations of (bw auth, file server) to achieve similar diversity with fixed pairings.

I believe that controlling things in this way is also superior to surprise emergent effects of using a CDN.

comment:3 in reply to:  2 Changed 2 years ago by teor

It's unclear whether the "average stream capacity regardless of path" includes the path from the client to the entry, and the exit to the internet server. Pragmatically, in the current design, it has to include client and internet server. (Or, in the case of onion services, client and service.)

I don't know if this affects our design at all, but it should be clarified in the spec. I opened #24730 for this.

Replying to mikeperry:

We discussed this on tor-internal two years ago. The conclusion we came to then was that the simplest (and best) plan is to make the list of URLs in https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/bwauthority_child.py#n51 contain a handful of fast, geographically distributed mirrors.

This way, all bw authorities will use the same file servers, and the results that they produce for each relay should be an approximate arithmetic mean of its performance with each bandwidth file server (this is approximate because we do some measurement filtering, and because the choice of url is random, not strictly round robin: https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/bwauthority_child.py#n131).

Then, for more diversity, we can move individual bw auths to geographically diverse locations. The result of this is that the directory authorities will take the median of their votes for each relay.

The key advantage between this plan and just having each bw auth pick its own file server location is that we avoid having degenerate pairing effects of fast location to fast location, and we also avoid the need to do the full pairing combinations of (bw auth, file server) to achieve similar diversity with fixed pairings.

I believe that controlling things in this way is also superior to surprise emergent effects of using a CDN.

I am fine with this plan, as long as we are willing to set up servers in the relevant locations within a reasonable amount of time.

I am also fine with using a CDN, because it's a fast way to get access to a large number of locations. And CDNs are already used for significant amounts of actual Tor client traffic, so excluding them might create less accurate results.

Unless there's a show-stopper here, let's document the pros and cons, and leave the choice up to the (bandwidth) authority operators?

comment:4 Changed 2 years ago by teor

From irl's quick analysis in #24834 at https://people.torproject.org/~irl/volatile/rsvotes/ it looks like gabelmoo is the most significant source of bias towards DE and FR.

comment:5 Changed 2 years ago by teor

We have confirmed that using a CDN works and creates a small amount (15%) of additional variation between two bandwidth authorities run from the same location. This is about what we expected.

Our next step here is start sharing bandwidth servers. (The bandwidth authority operators can decide whether to use a CDN in that set.)

Do we need a proposal for this?

I believe that controlling things in this way is also superior to surprise emergent effects of using a CDN.

mikeperry, can you be more specific about the surprises we should be looking for?

I've looked at tom's analysis of using Fastly, and I don't see anything surprising:

https://trac.torproject.org/projects/tor/ticket/24506#comment:9

comment:6 Changed 19 months ago by juga

Cc: juga added

comment:7 Changed 18 months ago by teor

Cc: teor@… removed

Remove useless CC

Note: See TracTickets for help on using tickets.