Opened 14 months ago

Closed 6 months ago

#22271 closed task (fixed)

Regenerate fallback list for 0.3.2 or 0.3.3

Reported by: teor Owned by: teor
Priority: Medium Milestone: Tor: 0.3.3.x-final
Component: Core Tor/Fallback Scripts Version:
Severity: Normal Keywords: fallback
Cc: atagar, irl Actual Points: 2
Parent ID: Points: 3
Reviewer: Sponsor:

Description

This involves:

  • checking for new relays
  • contacting potential fallback operators
  • rebuilding the list
  • contacting all fallback operators

See detailed instructions here:
https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors

Child Tickets

TicketStatusOwnerSummaryComponent
#20942closedteorMake consensus expiry tolerance for fallbacks lower when the stale consensus bug is fixedCore Tor/Fallback Scripts
#22321closedteorUpdate fallback directory whitelist based on operator relay changesCore Tor/Fallback Scripts
#22759closedteorMark the fallbacks that have extra-info descriptors, so stem can use themCore Tor/Fallback Scripts
#24600closedteorAdd fallback nicknames to the file, so stem can use themCore Tor/Fallback Scripts
#24679closedteorRemove custom weights from each fallback in the fallback listCore Tor/Fallback Scripts
#24681closedteorMake the default fallback weight in Tor 10.0Core Tor/Tor
#24682closedteorMake fallback script usage instructions create a log fileCore Tor/Fallback Scripts
#24706closedteorAdd a script that creates fallback directory whitelist entriesCore Tor/Fallback Scripts
#24708closedteorMake the fallback script search harder for pythonCore Tor/Fallback Scripts
#24711closedteorUpdate dependency list for the fallback scriptCore Tor/Fallback Scripts
#24725closedteorAdd a format version number to the fallback fileCore Tor/Fallback Scripts
#24726closedteorMake sure a comma never appears anywhere in a fallback entryCore Tor/Fallback Scripts
#24742closedteorAdd fallback list spec to torspecCore Tor/Fallback Scripts
#24785closedteorReduce the fallback stability and flag requirements due to extra network loadCore Tor/Fallback Scripts
#24791closedteorRemove excessive address info from the fallback script logCore Tor/Fallback Scripts
#24801closedteorGenerate a new fallback list and backport itCore Tor/Fallback Scripts

Change History (12)

comment:1 Changed 11 months ago by teor

I think we might want to aim for ~200 fallbacks this time, but I don't have the time to run an opt-in. I'll see if someone else wants to do it.

I've emailed tor-project about this.

Edit: clarify wording, note that I've emailed tor-project

Last edited 11 months ago by teor (previous) (diff)

comment:2 Changed 10 months ago by teor

Milestone: Tor: 0.3.2.x-finalTor: 0.3.3.x-final

I'm not going to get time to do these in 0.3.2.
Moving them to 0.3.3.

comment:3 Changed 8 months ago by teor

Cc: atagar irl added

Cc'ing the stem and relay search maintainers, because they want to know when the list changes.
(For the record, probably not this year, unless I suddenly come up with half a day of free time, or it becomes urgent.)

comment:4 Changed 8 months ago by atagar

Hi Tim. As mentioned in #21619 no need to notify me of fallback changes.

comment:5 Changed 8 months ago by teor

Keywords: 028-backport 029-backport 030-backport 031-backport 032-backport added

comment:6 Changed 7 months ago by teor

Adding nicknames (#24600) and extra-info flags (#22759) is optional, but they will help some people out.

comment:7 Changed 7 months ago by teor

Keywords: 028-backport removed

0.2.8 is EOL, so we won't be backporting to it.

comment:8 Changed 7 months ago by teor

Please merge my branch fallback-code-2018-01 into master, and leave this ticket open for the remaining children.

#24742 is in https://github.com/teor2345/torspec.git
All other branches are in https://github.com/teor2345/tor.git

This branch contains:

Fallback whitelist and blacklist changes:

  • #22321 by pastly and teor, reviewed by teor and pastly, branch teor/fallback_whiteblack_changes_squashed

Fallback script changes:

  • #20942 by minik, reviewed by teor, branch teor/bug20942
    • rejects any fallback that returns an expired directory document
  • #24706 by teor and atagar, reviewed by atagar, teor, and pastly, branch teor/ticket24706
    • adds a separate script that takes fingerprints and creates fallback whitelist entries
  • #24708 by teor, reviewed by pastly, branch teor/bug24708
    • makes the fallback script support python outside of /usr/bin
  • #24711 by teor, no review (trivial comment change), branch teor/ticket24711
    • revises and simplifies the dependency comment at the top of the fallback script
  • #24785 by teor, reviewed by pastly, branch teor/bug24785
    • modifies the fallback stability and bandwidth requirements to focus on keeping the same addresses and ports
  • #24791 by teor, reviewed by pastly, branch teor/bug24791
    • removes two logging statements from the fallback script

Fallback script changes and changes to the format of fallback_dirs.inc (but not the fallbacks it contains, so there's no need to backport it):

  • #22759 by teor, reviewed by pastly, branch teor/fallback-format-2-v4
    • adds a per-fallback comment containing an extrainfo flag
  • #24600 by teor, reviewed by pastly, branch teor/fallback-format-2-v4
    • adds a per-fallback comment containing the nickname
  • #24679 by teor, reviewed by pastly, branch teor/fallback-format-2-v4
    • removes per-fallback weights
    • we keep the same behaviour by changing Tor's default weight in #24681
  • #24725 by teor, reviewed by pastly, branch teor/fallback-format-2-v4
    • adds a comment header to the fallback file including type, version, and timestamp
  • #24726 by teor, reviewed by pastly, branch teor/fallback-format-2-v4
    • filters some characters out of onionoo data, as a precaution

The fallback list format changes were also reviewed by atagar and irl in:
https://lists.torproject.org/pipermail/tor-dev/2017-December/012741.html

These tickets are *not* in this branch:

  • #24681 by teor, reviewed by pastly, branch teor/ticket24681_028
    • changes the default fallback weight in tor, because we remove per-fallback weights in #24679
    • I want nickm's review on this for backporting
  • #24742 by teor, revewed by atagar and pastly, torspec branch teor/fallback-format-2-v2
    • a specification for the new fallback file format, including structured comments
  • #24801 is incomplete
    • we still need to generate and backport the new list

Edit: make sure each ticket has a description

Last edited 7 months ago by teor (previous) (diff)

comment:9 Changed 7 months ago by nickm

Hi. I am fine with the changes in fallback-code-2018-01. I think #24681 is good to backport once we have a new fallback_dirs.inc, but see comment and question there?

comment:10 Changed 7 months ago by nickm

Merged this to master. Please close appropriate subtickets?

comment:11 Changed 6 months ago by teor

Actual Points: 2
Keywords: 029-backport 030-backport 031-backport 032-backport removed
Type: enhancementtask

The new list in the new format has been merged:
https://gitweb.torproject.org/tor.git/tree/src/or/fallback_dirs.inc

Thanks to pastly, atagar, irl, and nickm.

There's one more step here: announce the new list on tor-relays.
We should also let people know that the flags on atlas (metrics) and consensus-health (stem?) may take a while to update.

I'll do this when I'm on a device that can sign emails.

comment:12 Changed 6 months ago by teor

Resolution: fixed
Status: newclosed

This is done.
Thanks all!

https://lists.torproject.org/pipermail/tor-relays/2018-January/014080.html

#24786 is the master task for the next rebuild. Relay opt-ins and updates go to #24805.

Note: See TracTickets for help on using tickets.