Opened 23 months ago

Closed 16 months ago

Last modified 15 months ago

#28795 closed task (implemented)

Generate a new fallback list in 2019 and backport it to all supported Tor versions

Reported by: teor Owned by: teor
Priority: Medium Milestone: Tor: 0.2.9.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: fallback, 029-backport, 035-backport, 040-backport, 041-backport, 041-should
Cc: teor, starlight@… Actual Points: 1.5
Parent ID: Points: 1
Reviewer: nickm Sponsor:

Description (last modified by teor)

We usually get two people to run:

scripts/maint/updateFallbackDirs.py > src/or/fallback_dirs.inc 2> fallback_dirs.log

from two different locations, then merge the lists.

Please attach the logs to this ticket.

Child Tickets

Attachments (8)

fallback_dirs_2019-06-25-11-49-10_AU_a37adb956fbb5cd2.inc (21.6 KB) - added by teor 16 months ago.
fallback dir list 3
fallback_dirs_2019-06-25-11-49-10_AU_a37adb956fbb5cd2.log (337.3 KB) - added by teor 16 months ago.
fallback dir list 3 log with info
fallback_dirs_2019-06-28-08-58-39_AU_f0437a39ddbc8459.inc (21.6 KB) - added by teor 16 months ago.
fallback dir list 4
fallback_dirs_2019-06-28-08-58-39_AU_f0437a39ddbc8459.log (337.2 KB) - added by teor 16 months ago.
fallback dir list 4 log
fallback_dirs_2019-06-28-09-00-11_DE_f0437a39ddbc8459.inc (21.6 KB) - added by teor 16 months ago.
fallback dir list 4 DE
fallback_dirs_2019-06-28-09-00-11_DE_f0437a39ddbc8459.log (337.7 KB) - added by teor 16 months ago.
fallback dir list 4 DE log
fallback_dirs_2019-06-28-10-26-11_US_246ed3b179f1b237.log (338.2 KB) - added by teor 16 months ago.
fallback dir list 4 US log
fallback_dirs_2019-06-28-10-26-11_US_246ed3b179f1b237.inc (21.5 KB) - added by teor 16 months ago.
fallback dir list 4 US

Download all attachments as: .zip

Change History (27)

comment:1 Changed 23 months ago by teor

Component: Core Tor/Fallback ScriptsCore Tor/Tor
Description: modified (diff)
Keywords: 029-backport 034-backport 035-backport 040-backport added

comment:2 Changed 23 months ago by starlight

Cc: starlight@… added

comment:3 Changed 23 months ago by teor

Type: defecttask

comment:4 Changed 22 months ago by starlight

to view full explanations for reasons why relays are excluded this edit is needed

@@ -2214,4 +2219,4 @@                                                           
 def process_default():                                                         
-  logging.basicConfig(level=logging.WARNING)
-  logging.getLogger('stem').setLevel(logging.WARNING)
+  logging.basicConfig(level=logging.INFO)
+  logging.getLogger('stem').setLevel(logging.INFO)
   whitelist = {'data': read_from_file(WHITELIST_FILE_NAME, MAX_LIST_FILE_SIZE),
@@ -2235,3 +2240,3 @@
   if get_command() == 'check_existing':
-    logging.warning(msg, *args)
+    logging.info(msg, *args)
   else:

otherwise little is revealed

comment:5 Changed 22 months ago by teor

We don't use check_existing mode when generating the list, so the final part of the patch shouldn't be needed.

Maybe we should make warn the default, and add an argument for info. Or the other way around.

comment:6 in reply to:  5 Changed 22 months ago by teor

This is the ticket for list generation. I opened #28986 so we don't forget your change.

Replying to teor:

We don't use check_existing mode when generating the list, so the final part of the patch shouldn't be needed.

But it would be good to be consistent.

Maybe we should make warn the default, and add an argument for info. Or the other way around.

Let's make info-level the default, so we don't have to remember extra options.

comment:7 Changed 17 months ago by nickm

Keywords: 034-backport removed

Removing 034-backport from all open tickets: 034 has reached EOL.

comment:8 Changed 16 months ago by teor

Keywords: 041-backport added
Owner: set to teor
Points: 1
Status: newassigned

comment:9 Changed 16 months ago by teor

Status: assignedneeds_review

comment:10 Changed 16 months ago by teor

I attached a draft fallback list and log, based on this branch:
https://github.com/teor2345/fallback-scripts/tree/tickets29100_28794

I still need to:

  • activate info-level logging (#28986)
  • work out why there are only 127 entries in the list (#30952)

comment:11 Changed 16 months ago by teor

Actual Points: 0.5
Keywords: 041-should added
Milestone: Tor: unspecifiedTor: 0.4.1.x-final

comment:12 Changed 16 months ago by teor

Status: needs_reviewneeds_revision

Changed 16 months ago by teor

fallback dir list 3

Changed 16 months ago by teor

fallback dir list 3 log with info

comment:13 Changed 16 months ago by teor

List 3 is based on this branch, which merges #29100 and #28794:
https://github.com/torproject/fallback-scripts/pull/4

I'm going to run the script from the US in the morning, and maybe Europe or another continent if I can find a trusted volunteer.

Changed 16 months ago by teor

fallback dir list 4

Changed 16 months ago by teor

fallback dir list 4 log

Changed 16 months ago by teor

fallback dir list 4 DE

Changed 16 months ago by teor

fallback dir list 4 DE log

Changed 16 months ago by teor

fallback dir list 4 US log

Changed 16 months ago by teor

fallback dir list 4 US

comment:14 Changed 16 months ago by teor

Actual Points: 0.51.5
Status: needs_revisionneeds_review

See my pull request:

And clean merges:

This is a merge of the AU lists 3 & 4. I tried merging the DE and US lists, but they didn't add any more fallbacks. (It looks like timing makes the biggest difference to the results, not location. At least at my current location. So that's good to know.)

comment:15 Changed 16 months ago by asn

Reviewer: nickm

comment:16 Changed 16 months ago by nickm

Resolution: implemented
Status: needs_reviewclosed

Merged to 0.2.9 and forward. Hooray!

comment:17 Changed 16 months ago by nickm

(I did the full merge because this is one that historically we have given an automatic backport, on the theory that it is a good idea to keep all the branches together, and there are unlikely to be bugs here.)

comment:18 in reply to:  17 Changed 16 months ago by teor

Replying to nickm:

(I did the full merge because this is one that historically we have given an automatic backport, on the theory that it is a good idea to keep all the branches together, and there are unlikely to be bugs here.)

Thanks!

I've documented this process on the wiki on our backports page:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Backports#ImmediateBackports

Some things worth noting:

  • backports happen after CI passes
  • GeoIP files, fallbacks, and authorities get backported immediately, feel free to add more things to the list

comment:19 Changed 15 months ago by teor

Milestone: Tor: 0.4.1.x-finalTor: 0.2.9.x-final
Parent ID: #28793
Note: See TracTickets for help on using tickets.