Skip to content
Snippets Groups Projects

UpdatingFallbackDirectoryMirrors

Last edited by Alexander Færøy

Updating Tor's Hard-Coded Fallback Directory Mirror List

This page has instructions for updating Tor's list of fallback directory mirrors.

Read [FallbackDirectoryMirrors Fallback Directory Mirrors for information about how fallbacks work.]

A Typical Release

So you'd like to update Tor's hard-coded list of fallback directory mirrors for the next release?

The 2019-2020 rebuild master ticket is #30971 (moved). It has child tickets that cover these steps.

  1. Ask relay operators to opt-in their stable relays to the fallback list.

  2. Update the fallback whitelist with new and changed relay details

    • File: fallback-scripts/fallback.whitelist

  3. Run updateFallbackDirs.py with the new whitelist. You'll need a decent network connection for this. If you have IPv6, please set PERFORM_IPV6_DIRPORT_CHECKS. Please attach the script logs to the trac ticket.

    • Command: fallback-scripts/updateFallbackDirs.py, read the script for usage instructions
    • We usually get 2 people to run the script from different locations, and merge the results.

  4. Write a changes file

  • See below for an example, and commands that count fallback changes
  1. Announce the new list
  1. Tell downstream maintainers that the list has changed

The list should be rebuilt once ~25% of fallbacks go down. Tor clients are designed to work if 30% of the list is reachable, or 30% of the authorities are reachable, with only minor delays.

How to Contact Relay Operators

When you email relay operators individually, use the email address (or other contact) in their relay's ContactInfo.

Group Emails

Send the email to tor-relays. This allows operators without ContactInfo to opt-in (and it's good for transparency).

Please also CC:

  • teor (who did the first few fallback lists).
  • gus (the relay operator advocate).

1. Asking Potential Fallbacks to Opt-In

The fallback.whitelist contains relays whose operators opted-in to become a fallback directory mirror at the last release. But new, stable relays could have been started since then, so we need to ask their operators if they want to opt-in. We: a. ask relay operators to opt-in, b. add opt-ins to the whitelist.

a. Asking Relay Operators

Contact operators on tor-relays using the process in 'Group Emails' section above.

b. Whitelisting Fallbacks

As responses come in, add them to the whitelist.

2. Creating a New List of Fallbacks

a. How Many Fallbacks? (optional)

Every extra fallback:

  • spreads the load more evenly,
  • provides diverse IP addresses and ports for clients to contact,
  • provides another relay that can take up the load if some fallbacks fail.

But it also:

  • adds to the size of Tor,
  • adds a relay that might go down or change details,
  • adds to a list that might be blocked on some networks.

The original proposal was to select 20% of the guards (~200 relays) as fallback directory mirrors. Having 100 by the end of the 2.5 years is less than we might have wanted, but since the extra load is so small, it's unlikely to be an issue.

b. Generating the List

We want to find the full list of fallback directory mirrors.

i. Make sure the whitelist and script are the latest versions (from git master, or a modified branch) ii. If you have a working IPv6 connection (or can set up an IPv6 tunnel), perform IPv6 checks:

- PERFORM_IPV6_DIRPORT_CHECKS = False if OUTPUT_CANDIDATES else False
+ PERFORM_IPV6_DIRPORT_CHECKS = True

iii. Run the fallback selection script, saving the list. See the top of the script for usage info. This can take a long time, as it downloads ~150MB of OnionOO data, parses it, then tries each fallback's DirPort.

Handling Consensus Download Failures

Sometimes fallbacks are temporarily slow to serve consensuses, or temporarily down. Check again, at a different time, and from a different network connection, if you can.

If relays are down a lot, the uptime checks will eliminate them in the next release. But it might be worth removing them from the current release.

c. Asking Operators if their Fallback Details have Changed

The fallback script fuzzy matches on relay fingerprint, or IPv4 address, or IPv6 address. If all of these details have changed on a relay, the fallback will be removed when the list is rebuilt.

To get back on the list, the operator should opt-in the fallback's new details.

d. Updating the List in Tor's Development Branch

For 0.2.9, do:

git add src/or/fallback_dirs.inc

And then merge forward.

In 0.3.5 and later, the script is at:

git add src/app/config/fallback_dirs.inc

The merge forward should handle this file move automatically.

Formatting Changes Files

When we replace the entire list of fallbacks, this is what a changes file should look like:

  o Minor feature (fallback directory mirrors):
    - Replace the 80 remaining fallbacks of the 100 originally introduced
      in Tor 0.2.8.1-alpha in January 2016, with a list of 100 fallbacks
      (40 new, 60 existing, 20 removed) generated in March 2016.
      Closes ticket 17158.

You can use the following commands to count fallbacks:

(In 2020, when 0.3.5 is the earliest supported version, use src/app/config/fallback_dirs.inc.)

Remaining/Replacement:

grep -c id= src/or/fallback_dirs.inc

New:

git diff src/or/fallback_dirs.inc | grep id= | grep + | wc -l

Removed (by deletion):

git diff src/or/fallback_dirs.inc | grep id= | grep - | grep -v \* | wc -l

Removed (by comments, only in early versions of the fallback list before 2017):

git diff src/or/fallback_dirs.inc | grep id= | grep + | grep \* | wc -l

The existing count is the replacement count, minus the new count. (It can't be calculated directly from the git diff, because diff skips unchanged lines.)

Remember to run make check to check for parse errors and changes file errors.

e. Backporting the List to Other Tor Versions

We backport the fallback list to any tor versions that are still supported. This makes sure that those versions still bootstrap without delay, and avoids them overloading the authorities.

Comments

    Please register or sign in to add a comment.