Regenerate fallback list for 0.3.2 or 0.3.3

This involves:

• checking for new relays
• contacting potential fallback operators
• rebuilding the list
• contacting all fallback operators

See detailed instructions here: https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors

changed milestone to %Tor: 0.3.3.x-final

added actualpoints::2 component::core tor/fallback scripts fallback milestone::Tor: 0.3.3.x-final owner::teor points::3 priority::medium resolution::fixed severity::normal status::closed type::task labels

I think we might want to aim for ~200 fallbacks this time, but I don't have the time to run an opt-in. I'll see if someone else wants to do it.

I've emailed tor-project about this.

Edit: clarify wording, note that I've emailed tor-project

I'm not going to get time to do these in 0.3.2. Moving them to 0.3.3.

Trac:
Milestone: Tor: 0.3.2.x-final to Tor: 0.3.3.x-final

Cc'ing the stem and relay search maintainers, because they want to know when the list changes. (For the record, probably not this year, unless I suddenly come up with half a day of free time, or it becomes urgent.)

Trac:
Cc: N/A to atagar, irl

Hi Tim. As mentioned in #21619 (moved) no need to notify me of fallback changes.

Tag with currently supported releases >= 0.2.8 according to https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases#Listofreleases

Trac:
Keywords: N/A deleted, 031-backport, 029-backport, 030-backport, 028-backport, 032-backport added

Adding nicknames (#24600 (moved)) and extra-info flags (#22759 (moved)) is optional, but they will help some people out.

0.2.8 is EOL, so we won't be backporting to it.

Trac:
Keywords: 028-backport deleted, N/A added

Please merge my branch fallback-code-2018-01 into master, and leave this ticket open for the remaining children.

#24742 (moved) is in https://github.com/teor2345/torspec.git All other branches are in https://github.com/teor2345/tor.git

This branch contains:

Fallback whitelist and blacklist changes:

• #22321 (moved) by pastly and teor, reviewed by teor and pastly, branch teor/fallback_whiteblack_changes_squashed

Fallback script changes:

• #20942 (moved) by minik, reviewed by teor, branch teor/bug20942
  • rejects any fallback that returns an expired directory document
• #24706 (moved) by teor and atagar, reviewed by atagar, teor, and pastly, branch teor/ticket24706
  • adds a separate script that takes fingerprints and creates fallback whitelist entries
• #24708 (moved) by teor, reviewed by pastly, branch teor/bug24708
  • makes the fallback script support python outside of /usr/bin
• #24711 (moved) by teor, no review (trivial comment change), branch teor/ticket24711
  • revises and simplifies the dependency comment at the top of the fallback script
• #24785 (moved) by teor, reviewed by pastly, branch teor/bug24785
  • modifies the fallback stability and bandwidth requirements to focus on keeping the same addresses and ports
• #24791 (moved) by teor, reviewed by pastly, branch teor/bug24791
  • removes two logging statements from the fallback script

Fallback script changes and changes to the format of fallback_dirs.inc (but not the fallbacks it contains, so there's no need to backport it):

• #22759 (moved) by teor, reviewed by pastly, branch teor/fallback-format-2-v4
  • adds a per-fallback comment containing an extrainfo flag
• #24600 (moved) by teor, reviewed by pastly, branch teor/fallback-format-2-v4
  • adds a per-fallback comment containing the nickname
• #24679 (moved) by teor, reviewed by pastly, branch teor/fallback-format-2-v4
  • removes per-fallback weights
  • we keep the same behaviour by changing Tor's default weight in #24681 (moved)
• #24725 (moved) by teor, reviewed by pastly, branch teor/fallback-format-2-v4
  • adds a comment header to the fallback file including type, version, and timestamp
• #24726 (moved) by teor, reviewed by pastly, branch teor/fallback-format-2-v4
  • filters some characters out of onionoo data, as a precaution

The fallback list format changes were also reviewed by atagar and irl in: https://lists.torproject.org/pipermail/tor-dev/2017-December/012741.html

These tickets are not in this branch:

• #24681 (moved) by teor, reviewed by pastly, branch teor/ticket24681_028
  • changes the default fallback weight in tor, because we remove per-fallback weights in #24679 (moved)
  • I want nickm's review on this for backporting
• #24742 (moved) by teor, revewed by atagar and pastly, torspec branch teor/fallback-format-2-v2
  • a specification for the new fallback file format, including structured comments
• #24801 (moved) is incomplete
  • we still need to generate and backport the new list

Edit: make sure each ticket has a description

Hi. I am fine with the changes in fallback-code-2018-01. I think #24681 (moved) is good to backport once we have a new fallback_dirs.inc, but see comment and question there?

Merged this to master. Please close appropriate subtickets?

The new list in the new format has been merged: https://gitweb.torproject.org/tor.git/tree/src/or/fallback_dirs.inc

Thanks to pastly, atagar, irl, and nickm.

There's one more step here: announce the new list on tor-relays. We should also let people know that the flags on atlas (metrics) and consensus-health (stem?) may take a while to update.

I'll do this when I'm on a device that can sign emails.

Trac:
Actualpoints: N/A to 2
Type: enhancement to task
Keywords: 030-backport, 029-backport, 032-backport, 031-backport deleted, N/A added

This is done. Thanks all!

https://lists.torproject.org/pipermail/tor-relays/2018-January/014080.html

#24786 (moved) is the master task for the next rebuild. Relay opt-ins and updates go to #24805 (moved).

Trac:
Status: new to closed
Resolution: N/A to fixed

closed

changed time estimate to 24h

added 16h of time spent

mentioned in issue #22321 (moved)

mentioned in issue #22527 (moved)

mentioned in issue #22759 (moved)