wiki:doc/UpdatingFallbackDirectoryMirrors

Updating Tor's Hard-Coded Fallback Directory Mirror List

This page has instructions for updating Tor's list of fallback directory mirrors.

Read Fallback Directory Mirrors for information about how fallbacks work.

A Typical Release

So you'd like to update Tor's hard-coded list of fallback directory mirrors for the next release?

  1. Ask relay operators to opt-in their stable relays to the fallback list.
  1. Run the script, and look at the warnings to see which addresses have changed.
  1. Update the fallback whitelist with new and changed relay details
    • File: tor/scripts/maint/fallback.whitelist
  1. Run updateFallbackDirs.py with the new whitelist. You'll need a decent network connection for this. If you have IPv6, please set PERFORM_IPV6_DIRPORT_CHECKS. Please attach the script logs to the trac ticket.
    • Command: tor/scripts/maint/updateFallbackDirs.py > src/or/fallback_dirs.inc 2> fallback_dirs.log
  1. Write a changes file
    • See below for an example, and commands that count fallback changes
  1. Announce the new list
  1. Tell downstream maintainers that the list has changed

The list should be rebuilt once ~25% of fallbacks go down. Tor clients are designed to work if 30% of the list is reachable, or 30%of the authorities are reachable, with only minor delays.

How to Contact Relay Operators

When you email relay operators individually, use the email address (or other contact) in their relay's ContactInfo.

Group Emails

Send the email to tor-relays. This allows operators without ContactInfo to opt-in (and it's good for transparency).

Please also CC:

  • teor (who did the first few fallback lists).

1. Asking Potential Fallbacks to Opt-In

The fallback.whitelist contains relays whose operators opted-in to become a fallback directory mirror at the last release. But new, stable relays could have been started since then, so we need to ask their operators if they want to opt-in. We:

  1. ask relay operators to opt-in,
  2. add opt-ins to the whitelist (and opt-outs to the blacklist).

a. Asking Relay Operators

Contact operators on tor-relays using the process in 'Group Emails' section above.

b. Whitelisting Fallbacks

As responses come in, add them to the whitelist.

If any operators confirm their relays will never be suitable, add them to the blacklist. (This makes it easier to switch to opt-out in future.)

2. Creating a New List of Fallbacks

a. How Many Fallbacks? (optional)

Every extra fallback:

  • spreads the load more evenly,
  • provides diverse IP addresses and ports for clients to contact,
  • provides another relay that can take up the load if some fallbacks fail.

But it also:

  • adds to the size of Tor,
  • adds a relay that might go down or change details,
  • adds to a list that might be blocked on some networks.

The original proposal was to select 20% of the guards (~200 relays) as fallback directory mirrors. Having 100 by the end of the 2.5 years is less than we might have wanted, but since the extra load is so small, it's unlikely to be an issue.

b. Generating the List

We want to find the full list of fallback directory mirrors.

  1. Make sure the whitelist and blacklist and script are the current ones from git master (not the modified ones from other instructions)
  2. If you have a working IPv6 connection (or can set up an IPv6 tunnel), perform IPv6 checks:
    - PERFORM_IPV6_DIRPORT_CHECKS = False if OUTPUT_CANDIDATES else False
    + PERFORM_IPV6_DIRPORT_CHECKS = True
    
  3. Run the fallback selection script, saving the list:
    scripts/maint/updateFallbackDirs.py > src/or/fallback_dirs.inc 2> fallback_dirs.log
    

This can take a long time, as it downloads ~150MB of OnionOO data, parses it, then tries each fallback's DirPort.

Handling Consensus Download Failures

Sometimes fallbacks are temporarily slow to serve consensuses, or temporarily down. Check again, at a different time, and from a different network connection, if you can.

If relays are down a lot, the uptime checks will eliminate them in the next release. But it might be worth removing them from the current release.

c. Confirming Fallback Changes with Operators

Handling New IPv6 Addresses

Adding an IPv6 address is a special case: clients never knew about the new IPv6 address, so they will continue to bootstrap fine using the IPv4 address. There's no need to remove the relay from the existing hard-coded list.

But the relay's whitelist entry must include the new IPv6 address, so that it can be selected for the next release. (If the IPv6 address isn't stable, then that address can't be added to the whitelist. But since the descriptor and whitelist don't match, the relay can never be selected. We'll fix this in #20175.)

d. Updating the List in Tor's Development Branch

Do:

git add src/or/fallback_dirs.inc

Formatting Changes Files

When we replace the entire list of fallbacks, this is what a changes file should look like:

  o Minor feature (fallback directory mirrors):
    - Replace the 80 remaining fallbacks of the 100 originally introduced
      in Tor 0.2.8.1-alpha in January 2016, with a list of 100 fallbacks
      (40 new, 60 existing, 20 removed) generated in March 2016.
      Closes ticket 17158; patch by teor.

You can use the following commands to count fallbacks:

Remaining/Replacement:

grep -c id= src/or/fallback_dirs.inc

New:

git diff src/or/fallback_dirs.inc | grep id= | grep + | wc -l

Removed (by deletion):

git diff src/or/fallback_dirs.inc | grep id= | grep - | grep -v \* | wc -l

Removed (by comments):

git diff src/or/fallback_dirs.inc | grep id= | grep + | grep \* | wc -l

The existing count is the replacement count, minus the new count. (It can't be calculated directly from the git diff, because diff skips unchanged lines.)

Remember to run make check to check for parse errors and changes file errors.

e. Backporting the List to Other Tor Versions

We backport the fallback list to any tor versions >= 0.2.8 that are still supported. This makes sure that those versions still bootstrap without delay, and avoids them overloading the authorities.

Last modified 4 weeks ago Last modified on Jun 25, 2018, 6:19:14 PM