Opened 12 months ago

Last modified 8 days ago

#24786 new task

Rebuild the fallback list in 2018

Reported by: teor Owned by:
Priority: Medium Milestone: Tor: 0.4.0.x-final
Component: Core Tor/Fallback Scripts Version:
Severity: Normal Keywords: s8-bootstrap, fallback, 035-roadmap-master, 029-backport, 033-backport, 034-backport, 034-roadmap-master, 034-triage-20180328, 034-included-20180328, 031-unreached-backport-maybe, 035-triaged-in-20180711, 035-can, 032-unreached-backport-maybe
Cc: Actual Points:
Parent ID: Points: 3
Reviewer: Sponsor: Sponsor8-can

Description

We need to rebuild the list of fallbacks in late 2018 or early 2019.
We usually do this when 25% or more go down.
(This is tracked in #tor-bots on IRC.)

Here are the instructions for running a rebuild:
https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors

Child Tickets

TicketStatusOwnerSummaryComponent
#24803closedphoulGenerate a new fallback list in 2018 and backport it to all supported versionsCore Tor/Tor
#24804closedphoulRun an opt-in process for relay operators to become fallbacks in 2018Core Tor/Fallback Scripts
#24805closedphoulUpdate fallback whitelist in late 2018Core Tor/Tor
#24838needs_reviewteorFuzzy match the fallback whitelistCore Tor/Tor
#26502closedStop using the fallback blacklist, and delete itCore Tor/Fallback Scripts

Change History (22)

comment:1 Changed 9 months ago by nickm

Keywords: 034-roadmap-master added

The *minimal* version of this ticket -- whatever that entails! -- is on the 034 roadmap.

comment:2 Changed 9 months ago by nickm

Keywords: 034-triage-20180328 added

comment:3 Changed 9 months ago by nickm

Keywords: 034-included-20180328 added

comment:4 Changed 8 months ago by teor

I am very happy with the current set of fallbacks, which we generated in January 2018 using a minimum 90 day stability period.
(The previous stability period was 30 days, which was too low - it led to a rebuild after ~6 months,)

We have only lost 8 falllbacks after ~3 months:
https://consensus-health.torproject.org/graphs.html

At this rate, it will take us ~14 months to lose 25% of our 150 original fallbacks (or ~38 fallbacks).
But we also want to rebuild the list every 12 months, so let's rebuild in January 2019 at the latest, or 0.3.6:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases#Current

If we're still doing well with fallbacks in 3-6 months' time, let's send out an email to thank all the fallback operators.

comment:5 Changed 8 months ago by teor

Milestone: Tor: 0.3.4.x-finalTor: 0.3.5.x-final

8/150 = 5% of fallbacks are currently down.
https://consensus-health.torproject.org/graphs.html
At this rate, it will be 2019 before we reach our 25% failure threshold.
So we don't need to do a fallback rebuild in 0.3.4.

comment:6 Changed 8 months ago by teor

Summary: Rebuild the fallback list for 0.3.4 or 0.3.5Rebuild the fallback list in 2018

comment:7 Changed 6 months ago by teor

Keywords: 031-unreached-backport added

0.3.1 is end of life, there are no more backports.
Tagging with 031-unreached-backport instead.

comment:8 Changed 6 months ago by teor

Keywords: 031-unreached-backport-maybe added; 031-backport-maybe removed

0.3.1 is end of life, there are no more 0.3.1 maybe backports.
Tagging with 031-unreached-backport-maybe.

comment:9 Changed 6 months ago by teor

Keywords: 031-unreached-backport removed

Keywords does substrings in search, but not replace.

comment:10 Changed 6 months ago by nickm

Keywords: 035-roadmap-master added; 029-backport 032-backport-maybe 033-backport 034-backport 034-roadmap-master 034-triage-20180328 034-included-20180328 031-unreached-backport-maybe removed

comment:11 Changed 6 months ago by teor

Keywords: 029-backport 032-backport-maybe 033-backport 034-backport 034-roadmap-master 034-triage-20180328 034-included-20180328 031-unreached-backport-maybe added

Restore keywords

comment:12 Changed 5 months ago by teor

Cc: teor@… removed

Remove useless CC

comment:13 Changed 5 months ago by nickm

Keywords: 035-triaged-in-20180711 added

comment:14 Changed 4 months ago by nickm

Milestone: Tor: 0.3.5.x-finalTor: 0.3.6.x-final

comment:15 Changed 4 months ago by teor

Milestone: Tor: 0.3.6.x-finalTor: 0.3.5.x-final

Hi Colin,

Our monitoring system checked the fallbacks this morning, and found that 25% of them are down. So we should generate a new list of fallbacks in the next few weeks.

Here's what needs to be done:

  • you can do a final call for opt-ins, if you'd like (#24803)
  • Do a final update to the whitelist (#24805)
  • I'll make the fallback script ignore addresses in the whitelist (#24838)
  • I can merge the changes from #24805, #24838, and #26502 into a branch (or master)
  • We can both run the fallback script from that branch to generate a fallback_dirs.inc file (#24803)
    • Last time, pastly and I ran the script, and then we merged our results together
  • I can write a changes file (#24803)
  • I can backport the list to src/or/fallback_dirs.inc in maint-0.2.9 (#24803)

The changes file looks like:

  o Minor features (fallback directory list):
    - Replace the 177 fallbacks originally introduced in Tor 0.2.9.8 in
      December 2016 (of which ~126 were still functional), with a list of
      151 fallbacks (32 new, 119 existing, 58 removed) generated in
      May 2017.
      Resolves ticket 21564.

But I think I might need to tweak the wording to pass make check-changes.

comment:16 Changed 6 weeks ago by nickm

Keywords: 035-can added

comment:17 Changed 5 weeks ago by teor

Keywords: 032-unreached-backport added

0.3.2 is end of life, so 032-backport is now 032-unreached-backport.

comment:18 Changed 5 weeks ago by teor

Keywords: 032-unreached-backport-maybe added; 032-backport-maybe removed

Tag 032-backport-maybe with 032-unreached-backport-maybe

comment:19 Changed 5 weeks ago by teor

Keywords: 032-unreached-backport removed

Remove redundant 032-unreached-backport on 032-unreached-backport-maybe

comment:20 Changed 11 days ago by teor

Keywords: s8-bootstrap added
Milestone: Tor: 0.3.5.x-finalTor: 0.4.0.x-final
Sponsor: Sponsor8-can

This task and its subtasks improve Tor's bootstrapping speed, so they can be part of Sponsor 8.

comment:21 Changed 8 days ago by teor

New updates to the fallback list can go in #28793. In particular, whitelist changes go in #28794.

comment:22 Changed 8 days ago by teor

This ticket can close when all its child tickets have been merged.

Note: See TracTickets for help on using tickets.