Generate a new fallback list in 2019 and backport it to all supported Tor versions

We usually get two people to run:

scripts/maint/updateFallbackDirs.py > src/or/fallback_dirs.inc 2> fallback_dirs.log

from two different locations, then merge the lists.

Please attach the logs to this ticket.

changed milestone to %Tor: 0.2.9.x-final

added 029-backport 035-backport 040-backport 041-backport 041-should actualpoints::1.5 component::core tor/tor fallback milestone::Tor: 0.2.9.x-final owner::teor points::1 priority::medium resolution::implemented reviewer::nickm severity::normal status::closed type::task labels

Trac:
Description: N/A

to

We usually get two people to run:

scripts/maint/updateFallbackDirs.py > src/or/fallback_dirs.inc 2> fallback_dirs.log

from two different locations, then merge the lists.

Please attach the logs to this ticket.
Component: Core Tor/Fallback Scripts to Core Tor/Tor
Keywords: N/A deleted, 034-backport, 029-backport, 040-backport, 035-backport added

Trac:
Cc: teor to teor, starlight@binnacle.cx

Trac:
Type: defect to task

to view full explanations for reasons why relays are excluded this edit is needed

@@ -2214,4 +2219,4 @@                                                           
 def process_default():                                                         
-  logging.basicConfig(level=logging.WARNING)
-  logging.getLogger('stem').setLevel(logging.WARNING)
+  logging.basicConfig(level=logging.INFO)
+  logging.getLogger('stem').setLevel(logging.INFO)
   whitelist = {'data': read_from_file(WHITELIST_FILE_NAME, MAX_LIST_FILE_SIZE),
@@ -2235,3 +2240,3 @@
   if get_command() == 'check_existing':
-    logging.warning(msg, *args)
+    logging.info(msg, *args)
   else:

otherwise little is revealed

We don't use check_existing mode when generating the list, so the final part of the patch shouldn't be needed.

Maybe we should make warn the default, and add an argument for info. Or the other way around.

This is the ticket for list generation. I opened #28986 (moved) so we don't forget your change.

Replying to teor:

We don't use check_existing mode when generating the list, so the final part of the patch shouldn't be needed.

But it would be good to be consistent.

Maybe we should make warn the default, and add an argument for info. Or the other way around.

Let's make info-level the default, so we don't have to remember extra options.

Removing 034-backport from all open tickets: 034 has reached EOL.

Trac:
Keywords: 034-backport deleted, N/A added

Trac:
Keywords: N/A deleted, 041-backport added
Points: N/A to 1
Status: new to assigned
Owner: N/A to teor

Trac:
Status: assigned to needs_review

I attached a draft fallback list and log, based on this branch: https://github.com/teor2345/fallback-scripts/tree/tickets29100_28794

I still need to:

• activate info-level logging (#28986 (moved))
• work out why there are only 127 entries in the list (#30952 (moved))

Trac:
Milestone: Tor: unspecified to Tor: 0.4.1.x-final
Keywords: N/A deleted, 041-should added
Actualpoints: N/A to 0.5

Trac:
Status: needs_review to needs_revision

Trac:
fallback_dirs_2019-06-25-11-49-10_AU_a37adb956fbb5cd2.inc

fallback dir list 3

Trac:
fallback_dirs_2019-06-25-11-49-10_AU_a37adb956fbb5cd2.log

fallback dir list 3 log with info

List 3 is based on this branch, which merges #29100 (moved) and #28794 (moved): https://github.com/torproject/fallback-scripts/pull/4

I'm going to run the script from the US in the morning, and maybe Europe or another continent if I can find a trusted volunteer.

Trac:
fallback_dirs_2019-06-28-08-58-39_AU_f0437a39ddbc8459.inc

fallback dir list 4

Trac:
fallback_dirs_2019-06-28-08-58-39_AU_f0437a39ddbc8459.log

fallback dir list 4 log

Trac:
fallback_dirs_2019-06-28-09-00-11_DE_f0437a39ddbc8459.inc

fallback dir list 4 DE

Trac:
fallback_dirs_2019-06-28-09-00-11_DE_f0437a39ddbc8459.log

fallback dir list 4 DE log

Trac:
fallback_dirs_2019-06-28-10-26-11_US_246ed3b179f1b237.log

fallback dir list 4 US log

Trac:
fallback_dirs_2019-06-28-10-26-11_US_246ed3b179f1b237.inc

fallback dir list 4 US

See my pull request:

• 0.2.9: https://github.com/torproject/tor/pull/1146

And clean merges:

• 0.3.5: https://github.com/torproject/tor/pull/1147
• 0.4.0: https://github.com/torproject/tor/pull/1148
• 0.4.1: https://github.com/torproject/tor/pull/1149
• master: https://github.com/torproject/tor/pull/1150

This is a merge of the AU lists 3 & 4. I tried merging the DE and US lists, but they didn't add any more fallbacks. (It looks like timing makes the biggest difference to the results, not location. At least at my current location. So that's good to know.)

Trac:
Actualpoints: 0.5 to 1.5
Status: needs_revision to needs_review

Trac:
Reviewer: N/A to nickm

Merged to 0.2.9 and forward. Hooray!

Trac:
Resolution: N/A to implemented
Status: needs_review to closed

(I did the full merge because this is one that historically we have given an automatic backport, on the theory that it is a good idea to keep all the branches together, and there are unlikely to be bugs here.)

Replying to nickm:

(I did the full merge because this is one that historically we have given an automatic backport, on the theory that it is a good idea to keep all the branches together, and there are unlikely to be bugs here.)

Thanks!

I've documented this process on the wiki on our backports page: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Backports#ImmediateBackports

Some things worth noting:

• backports happen after CI passes
• GeoIP files, fallbacks, and authorities get backported immediately, feel free to add more things to the list

Trac:
Parent: #28793 (moved) to N/A
Milestone: Tor: 0.4.1.x-final to Tor: 0.2.9.x-final

closed

changed time estimate to 8h

added 12h of time spent

moved to tpo/core/tor#28795 (closed)