Opened 18 months ago

Last modified 11 months ago

#24506 needs_information task

Move some bandwidth authority servers to a CDN

Reported by: teor Owned by: tom
Priority: High Milestone:
Component: Core Tor/Torflow Version:
Severity: Major Keywords: tor-bwauth
Cc: starlight@…, juga Actual Points:
Parent ID: #24674 Points:
Reviewer: Sponsor:

Description

We've experimented with using fastly as a bandwidth server.
Once the bandwidth authority code is stable, we should test this in parallel for a week or two, then make the switch.

Child Tickets

Attachments (1)

fastly-01.png (196.1 KB) - added by tom 17 months ago.
Very early comparison of fastly as the server for a bwauth

Download all attachments as: .zip

Change History (16)

comment:1 Changed 18 months ago by starlight

Cc: starlight@… added

I'd like to point out a problem with sourcing bandwidth measurement files on a CDN.

CDN's function by placing cached copies of content on servers located diversely around the world. The problem is that some of the CDN servers reside in the same physical datacenters as many of the fastest relays in the network. In some cases relays will have 10Gbit local-network connectivity to a CDN server hosting the bandwidth files, with sub-millisecond round-trip ping latency possibly reaching below 100 microseconds.

Therefore serving bandwidth files via CDN will result in unfairly high rating biases for co-located relays and unfairly low ratings for relays with good connectivity but without close proximity to a CDN cache-server. This unfairness will likely be impossible to correct out of bandwidth measurement results.

comment:2 Changed 18 months ago by teor

This is a fundamental issue with the design of the bandwidth authority system: it measures both bandwidth and latency. (See #24506 and BandwidthAuthorityMeasurements.)

It's already an issue with the existing locations of bandwidth scanners and bandwidth servers: nearby relays are rated higher than other relays. This is a big issue when there are only 5 scanners and 6 servers. Using a CDN actually makes this better, because it makes sure that more relays are close to one of the CDN bandwidth servers. (Fastly would give us about 20 CDN locations.)

And I don't think this issue will be as bad as you think:

  • bandwidth scanners build a two-hop path to the bandwidth server like this: scanner - entry - exit - server. So most paths will involve an entry that isn't in a CDN data center, and
  • even if some paths are all in the same data center, the bandwidth aggregator doesn't take the lowest value, it takes a combination of all the measured values.

To mitigate this issue, we should:

  • make sure that the scanner isn't in one of the CDN data centers,
  • use different CDNs for different scanners,
  • use CDNs in combination with other bandwidth servers that aren't in one of the CDN data centers.

comment:3 Changed 18 months ago by teor

Priority: MediumHigh
Severity: NormalMajor
Status: newneeds_revision

Hi Tom,

Can we please test this next on the parallel bandwidth scanner?
Exit weighting issues continue to come up on tor-dev and tor-relays.

I think using a CDN would make a huge difference here, if we had enough bandwidth scanners adopt it.

Do you need me to write a patch?
Or forward the email I sent to Micah about it?

comment:4 Changed 18 months ago by tom

If you can send me the email (I just need the url) I can def set it up this week.

comment:5 Changed 17 months ago by teor

Status: needs_revisionneeds_information

This is in progress on the parallel test bandwidth authority and one other bandwidth authority.

comment:6 Changed 17 months ago by mikeperry

FWIW, I think a better approach is to ensure we are getting the diversity we need by specifying a list of mirrors in specific geographic locations. If we can somehow specify which location that the CDN always uses for us, rather than having it picking the closest one to whichever exit, that would be better IMO. See also #24674.

Changed 17 months ago by tom

Attachment: fastly-01.png added

Very early comparison of fastly as the server for a bwauth

comment:7 in reply to:  6 ; Changed 17 months ago by teor

Thanks Tom!
A 15% increase in the differences between bandwidth authorities seems reasonable for such a major change,
Are you only using fastly? Or are you using fastly alongside your existing server?

Replying to mikeperry:

FWIW, I think a better approach is to ensure we are getting the diversity we need by specifying a list of mirrors in specific geographic locations.

We can use one or more CDNs to cover some geographic locations, and specifically set up servers in other locations that aren't covered by the CDNs.

If we can somehow specify which location that the CDN always uses for us, rather than having it picking the closest one to whichever exit, that would be better IMO. See also #24674.

Fastly does not have this feature: it only allows us to select between US-EU POPs, and all POPs.

Can you explain why choosing the nearest server is a bad thing?
If our goal is to provide each client with approximately the same bandwidth regardless of path, then we should include paths that clients typically use. This includes downloads from CDNs. And it probably does include a bias towards the US and EU, because many websites are hosted there, or are hosted on CDNs that have a bias towards there. (But maybe not a bias that is as extreme as the one we have right now.)

comment:8 in reply to:  7 Changed 17 months ago by tom

Replying to teor:

Thanks Tom!
A 15% increase in the differences between bandwidth authorities seems reasonable for such a major change,
Are you only using fastly? Or are you using fastly alongside your existing server?

Only fastly.

comment:10 Changed 16 months ago by teor

Parent ID: #24499#24674

Thanks!
This outcome looks good.

Our next steps are:

  • encourage individual bandwidth authority operators to use a CDN
  • create a proposal for sharing bandwidth servers between bandwidth authorities, see #24674

It will be up to each bandwidth authority operator to decide which servers they use.

comment:11 Changed 16 months ago by teor

Also, can we get a map of the Fastly bandwidth authority like the one in #24834?
https://people.torproject.org/~irl/volatile/rsvotes/

That would involve giving the bandwidth files to irl.

The details are here:
https://trac.torproject.org/projects/tor/ticket/24834#comment:10

I think it might help detect the kinds of issues that Mike is concerned about.

comment:12 Changed 16 months ago by teor

Also, bastet added Fastly in mid-December.
That made its measurements much more like the other bwauths.
So maybe using a CDN is helpful for consistency?

https://tomrittervg.github.io/bwauth-tools/#updated-01

comment:13 Changed 12 months ago by juga

Cc: juga added

comment:14 Changed 12 months ago by teor

If we want to run bandwidth authorities on a CDN, we should use a CDN config that isn't run by the tor project. We were running a few bandwidth authorities off a tor project server, and then the server went down. So the bandwidth authorities stopped working. That's not a great outcome.

If bandwidth authority operators run their own servers (whether on a CDN or not), then we won't have these kinds of issues. Operators can still share servers if they want.

comment:15 Changed 11 months ago by teor

Cc: teor@… removed

Remove useless CC

Note: See TracTickets for help on using tickets.