Opened 12 months ago

Closed 5 months ago

Last modified 4 months ago

#28795 closed task (implemented)

Generate a new fallback list in 2019 and backport it to all supported Tor versions

Reported by: teor Owned by: teor
Priority: Medium Milestone: Tor: 0.2.9.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: fallback, 029-backport, 035-backport, 040-backport, 041-backport, 041-should
Cc: teor, starlight@… Actual Points: 1.5
Parent ID: Points: 1
Reviewer: nickm Sponsor:

Description (last modified by teor)

We usually get two people to run:

scripts/maint/updateFallbackDirs.py > src/or/fallback_dirs.inc 2> fallback_dirs.log

from two different locations, then merge the lists.

Please attach the logs to this ticket.

Child Tickets

Attachments (8)

fallback_dirs_2019-06-25-11-49-10_AU_a37adb956fbb5cd2.inc (21.6 KB) - added by teor 6 months ago.
fallback dir list 3
fallback_dirs_2019-06-25-11-49-10_AU_a37adb956fbb5cd2.log (337.3 KB) - added by teor 6 months ago.
fallback dir list 3 log with info
fallback_dirs_2019-06-28-08-58-39_AU_f0437a39ddbc8459.inc (21.6 KB) - added by teor 5 months ago.
fallback dir list 4
fallback_dirs_2019-06-28-08-58-39_AU_f0437a39ddbc8459.log (337.2 KB) - added by teor 5 months ago.
fallback dir list 4 log
fallback_dirs_2019-06-28-09-00-11_DE_f0437a39ddbc8459.inc (21.6 KB) - added by teor 5 months ago.
fallback dir list 4 DE
fallback_dirs_2019-06-28-09-00-11_DE_f0437a39ddbc8459.log (337.7 KB) - added by teor 5 months ago.
fallback dir list 4 DE log
fallback_dirs_2019-06-28-10-26-11_US_246ed3b179f1b237.log (338.2 KB) - added by teor 5 months ago.
fallback dir list 4 US log
fallback_dirs_2019-06-28-10-26-11_US_246ed3b179f1b237.inc (21.5 KB) - added by teor 5 months ago.
fallback dir list 4 US

Download all attachments as: .zip

Change History (27)

comment:1 Changed 12 months ago by teor

Component: Core Tor/Fallback ScriptsCore Tor/Tor
Description: modified (diff)
Keywords: 029-backport 034-backport 035-backport 040-backport added

comment:2 Changed 12 months ago by starlight

Cc: starlight@… added

comment:3 Changed 12 months ago by teor

Type: defecttask

comment:4 Changed 12 months ago by starlight

to view full explanations for reasons why relays are excluded this edit is needed

@@ -2214,4 +2219,4 @@                                                           
 def process_default():                                                         
-  logging.basicConfig(level=logging.WARNING)
-  logging.getLogger('stem').setLevel(logging.WARNING)
+  logging.basicConfig(level=logging.INFO)
+  logging.getLogger('stem').setLevel(logging.INFO)
   whitelist = {'data': read_from_file(WHITELIST_FILE_NAME, MAX_LIST_FILE_SIZE),
@@ -2235,3 +2240,3 @@
   if get_command() == 'check_existing':
-    logging.warning(msg, *args)
+    logging.info(msg, *args)
   else:

otherwise little is revealed

comment:5 Changed 11 months ago by teor

We don't use check_existing mode when generating the list, so the final part of the patch shouldn't be needed.

Maybe we should make warn the default, and add an argument for info. Or the other way around.

comment:6 in reply to:  5 Changed 11 months ago by teor

This is the ticket for list generation. I opened #28986 so we don't forget your change.

Replying to teor:

We don't use check_existing mode when generating the list, so the final part of the patch shouldn't be needed.

But it would be good to be consistent.

Maybe we should make warn the default, and add an argument for info. Or the other way around.

Let's make info-level the default, so we don't have to remember extra options.

comment:7 Changed 6 months ago by nickm

Keywords: 034-backport removed

Removing 034-backport from all open tickets: 034 has reached EOL.

comment:8 Changed 6 months ago by teor

Keywords: 041-backport added
Owner: set to teor
Points: 1
Status: newassigned

comment:9 Changed 6 months ago by teor

Status: assignedneeds_review

comment:10 Changed 6 months ago by teor

I attached a draft fallback list and log, based on this branch:
https://github.com/teor2345/fallback-scripts/tree/tickets29100_28794

I still need to:

  • activate info-level logging (#28986)
  • work out why there are only 127 entries in the list (#30952)

comment:11 Changed 6 months ago by teor

Actual Points: 0.5
Keywords: 041-should added
Milestone: Tor: unspecifiedTor: 0.4.1.x-final

comment:12 Changed 6 months ago by teor

Status: needs_reviewneeds_revision

Changed 6 months ago by teor

fallback dir list 3

Changed 6 months ago by teor

fallback dir list 3 log with info

comment:13 Changed 6 months ago by teor

List 3 is based on this branch, which merges #29100 and #28794:
https://github.com/torproject/fallback-scripts/pull/4

I'm going to run the script from the US in the morning, and maybe Europe or another continent if I can find a trusted volunteer.

Changed 5 months ago by teor

fallback dir list 4

Changed 5 months ago by teor

fallback dir list 4 log

Changed 5 months ago by teor

fallback dir list 4 DE

Changed 5 months ago by teor

fallback dir list 4 DE log

Changed 5 months ago by teor

fallback dir list 4 US log

Changed 5 months ago by teor

fallback dir list 4 US

comment:14 Changed 5 months ago by teor

Actual Points: 0.51.5
Status: needs_revisionneeds_review

See my pull request:

And clean merges:

This is a merge of the AU lists 3 & 4. I tried merging the DE and US lists, but they didn't add any more fallbacks. (It looks like timing makes the biggest difference to the results, not location. At least at my current location. So that's good to know.)

comment:15 Changed 5 months ago by asn

Reviewer: nickm

comment:16 Changed 5 months ago by nickm

Resolution: implemented
Status: needs_reviewclosed

Merged to 0.2.9 and forward. Hooray!

comment:17 Changed 5 months ago by nickm

(I did the full merge because this is one that historically we have given an automatic backport, on the theory that it is a good idea to keep all the branches together, and there are unlikely to be bugs here.)

comment:18 in reply to:  17 Changed 5 months ago by teor

Replying to nickm:

(I did the full merge because this is one that historically we have given an automatic backport, on the theory that it is a good idea to keep all the branches together, and there are unlikely to be bugs here.)

Thanks!

I've documented this process on the wiki on our backports page:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Backports#ImmediateBackports

Some things worth noting:

  • backports happen after CI passes
  • GeoIP files, fallbacks, and authorities get backported immediately, feel free to add more things to the list

comment:19 Changed 4 months ago by teor

Milestone: Tor: 0.4.1.x-finalTor: 0.2.9.x-final
Parent ID: #28793
Note: See TracTickets for help on using tickets.