Opened 8 months ago

Closed 7 weeks ago

Last modified 13 days ago

#28795 closed task (implemented)

Generate a new fallback list in 2019 and backport it to all supported Tor versions

Reported by: teor Owned by: teor
Priority: Medium Milestone: Tor: 0.2.9.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: fallback, 029-backport, 035-backport, 040-backport, 041-backport, 041-should
Cc: teor, starlight@… Actual Points: 1.5
Parent ID: Points: 1
Reviewer: nickm Sponsor:

Description (last modified by teor)

We usually get two people to run:

scripts/maint/updateFallbackDirs.py > src/or/fallback_dirs.inc 2> fallback_dirs.log

from two different locations, then merge the lists.

Please attach the logs to this ticket.

Child Tickets

Attachments (8)

fallback_dirs_2019-06-25-11-49-10_AU_a37adb956fbb5cd2.inc (21.6 KB) - added by teor 8 weeks ago.
fallback dir list 3
fallback_dirs_2019-06-25-11-49-10_AU_a37adb956fbb5cd2.log (337.3 KB) - added by teor 8 weeks ago.
fallback dir list 3 log with info
fallback_dirs_2019-06-28-08-58-39_AU_f0437a39ddbc8459.inc (21.6 KB) - added by teor 8 weeks ago.
fallback dir list 4
fallback_dirs_2019-06-28-08-58-39_AU_f0437a39ddbc8459.log (337.2 KB) - added by teor 8 weeks ago.
fallback dir list 4 log
fallback_dirs_2019-06-28-09-00-11_DE_f0437a39ddbc8459.inc (21.6 KB) - added by teor 8 weeks ago.
fallback dir list 4 DE
fallback_dirs_2019-06-28-09-00-11_DE_f0437a39ddbc8459.log (337.7 KB) - added by teor 8 weeks ago.
fallback dir list 4 DE log
fallback_dirs_2019-06-28-10-26-11_US_246ed3b179f1b237.log (338.2 KB) - added by teor 8 weeks ago.
fallback dir list 4 US log
fallback_dirs_2019-06-28-10-26-11_US_246ed3b179f1b237.inc (21.5 KB) - added by teor 8 weeks ago.
fallback dir list 4 US

Download all attachments as: .zip

Change History (27)

comment:1 Changed 8 months ago by teor

Component: Core Tor/Fallback ScriptsCore Tor/Tor
Description: modified (diff)
Keywords: 029-backport 034-backport 035-backport 040-backport added

comment:2 Changed 8 months ago by starlight

Cc: starlight@… added

comment:3 Changed 8 months ago by teor

Type: defecttask

comment:4 Changed 8 months ago by starlight

to view full explanations for reasons why relays are excluded this edit is needed

@@ -2214,4 +2219,4 @@                                                           
 def process_default():                                                         
-  logging.basicConfig(level=logging.WARNING)
-  logging.getLogger('stem').setLevel(logging.WARNING)
+  logging.basicConfig(level=logging.INFO)
+  logging.getLogger('stem').setLevel(logging.INFO)
   whitelist = {'data': read_from_file(WHITELIST_FILE_NAME, MAX_LIST_FILE_SIZE),
@@ -2235,3 +2240,3 @@
   if get_command() == 'check_existing':
-    logging.warning(msg, *args)
+    logging.info(msg, *args)
   else:

otherwise little is revealed

comment:5 Changed 8 months ago by teor

We don't use check_existing mode when generating the list, so the final part of the patch shouldn't be needed.

Maybe we should make warn the default, and add an argument for info. Or the other way around.

comment:6 in reply to:  5 Changed 8 months ago by teor

This is the ticket for list generation. I opened #28986 so we don't forget your change.

Replying to teor:

We don't use check_existing mode when generating the list, so the final part of the patch shouldn't be needed.

But it would be good to be consistent.

Maybe we should make warn the default, and add an argument for info. Or the other way around.

Let's make info-level the default, so we don't have to remember extra options.

comment:7 Changed 2 months ago by nickm

Keywords: 034-backport removed

Removing 034-backport from all open tickets: 034 has reached EOL.

comment:8 Changed 2 months ago by teor

Keywords: 041-backport added
Owner: set to teor
Points: 1
Status: newassigned

comment:9 Changed 2 months ago by teor

Status: assignedneeds_review

comment:10 Changed 2 months ago by teor

I attached a draft fallback list and log, based on this branch:
https://github.com/teor2345/fallback-scripts/tree/tickets29100_28794

I still need to:

  • activate info-level logging (#28986)
  • work out why there are only 127 entries in the list (#30952)

comment:11 Changed 2 months ago by teor

Actual Points: 0.5
Keywords: 041-should added
Milestone: Tor: unspecifiedTor: 0.4.1.x-final

comment:12 Changed 2 months ago by teor

Status: needs_reviewneeds_revision

Changed 8 weeks ago by teor

fallback dir list 3

Changed 8 weeks ago by teor

fallback dir list 3 log with info

comment:13 Changed 8 weeks ago by teor

List 3 is based on this branch, which merges #29100 and #28794:
https://github.com/torproject/fallback-scripts/pull/4

I'm going to run the script from the US in the morning, and maybe Europe or another continent if I can find a trusted volunteer.

Changed 8 weeks ago by teor

fallback dir list 4

Changed 8 weeks ago by teor

fallback dir list 4 log

Changed 8 weeks ago by teor

fallback dir list 4 DE

Changed 8 weeks ago by teor

fallback dir list 4 DE log

Changed 8 weeks ago by teor

fallback dir list 4 US log

Changed 8 weeks ago by teor

fallback dir list 4 US

comment:14 Changed 8 weeks ago by teor

Actual Points: 0.51.5
Status: needs_revisionneeds_review

See my pull request:

And clean merges:

This is a merge of the AU lists 3 & 4. I tried merging the DE and US lists, but they didn't add any more fallbacks. (It looks like timing makes the biggest difference to the results, not location. At least at my current location. So that's good to know.)

comment:15 Changed 7 weeks ago by asn

Reviewer: nickm

comment:16 Changed 7 weeks ago by nickm

Resolution: implemented
Status: needs_reviewclosed

Merged to 0.2.9 and forward. Hooray!

comment:17 Changed 7 weeks ago by nickm

(I did the full merge because this is one that historically we have given an automatic backport, on the theory that it is a good idea to keep all the branches together, and there are unlikely to be bugs here.)

comment:18 in reply to:  17 Changed 7 weeks ago by teor

Replying to nickm:

(I did the full merge because this is one that historically we have given an automatic backport, on the theory that it is a good idea to keep all the branches together, and there are unlikely to be bugs here.)

Thanks!

I've documented this process on the wiki on our backports page:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Backports#ImmediateBackports

Some things worth noting:

  • backports happen after CI passes
  • GeoIP files, fallbacks, and authorities get backported immediately, feel free to add more things to the list

comment:19 Changed 13 days ago by teor

Milestone: Tor: 0.4.1.x-finalTor: 0.2.9.x-final
Parent ID: #28793
Note: See TracTickets for help on using tickets.