Opened 2 years ago

Closed 12 months ago

#24805 closed enhancement (fixed)

Update fallback whitelist in late 2018

Reported by: teor Owned by: phoul
Priority: Medium Milestone: Tor: 0.4.0.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: s8-bootstrap, fallback, 034-triage-20180328, 034-removed-20180328, 035-removed-20180711, 035-roadmap
Cc: phoul Actual Points:
Parent ID: #24786 Points: 0.5
Reviewer: Sponsor: Sponsor8-can

Description (last modified by teor)

We need to update the list using:

  • operator emails from 2018
  • the opt-in process in #24804
  • running the script, finding change warnings, contacting operators, and updating their details we'll do #24838 instead

Child Tickets

TicketTypeStatusOwnerSummary
#25169defectclosedFallback removal for 0BEA4A88D069753218EAAAD6D22EA87B9A1319D6
#25414defectclosedFallback details changed for 9FBEB75E8BC142565F12CBBE078D63310236A334
#25436taskclosedRemove from fallback.whitelist, add to fallback.blacklist 328E54981C6DDD7D89B89E418724A4A7881E3192
#25798defectclosedRemove from fallback list:
#27297enhancementclosednew fallbackdir: 113143469021882C3A4B82F084F8125B08EE471E
#27737defectclosedteorManually remove 64.113.32.29 from the generated fallback list

Change History (61)

comment:1 Changed 2 years ago by teor

Points: 2
Type: defectenhancement

comment:2 Changed 2 years ago by teor

An operator sent me an email asking me to add this relay to the whitelist:

https://atlas.torproject.org/#details/BC6B2E2F62ACC5EDECBABE64DA1E48F84DD98B78

It's just changed DirPort, so it wouldn't be in the January 2018 list anyway.
(Fallbacks need to have the same address and port for 90 days.)

comment:3 Changed 2 years ago by teor

An operator moved from 217.79.190.25 to 37.157.255.35 and sent this email:
https://lists.torproject.org/pipermail/tor-relays/2018-January/014055.html

comment:4 Changed 2 years ago by teor

Three operators sent emails to tor-relays:

https://lists.torproject.org/pipermail/tor-relays/2018-January/014063.html

92412EA1B9AA887D462B51D816777002F4D58907
360CBA08D1E24F513162047BDB54A1015E531534

https://lists.torproject.org/pipermail/tor-relays/2018-January/014064.html

A2A6616723B511D8E068BB71705191763191F6B2

https://lists.torproject.org/pipermail/tor-relays/2018-January/014069.html

E51620B90DCB310138ED89EDEDD0A5C361AAE24E

And one sent an email to me directly, saying their relay's details had changed to:

BC7ACFAC04854C77167C7D66B7E471314ED8C410
144.76.75.184:9001
[2a01:4f8:191:93a2::4]:9001

comment:5 Changed 2 years ago by teor

An operator sent an email to me directly, opting-in this relay:

F2DFE5FA1E4CF54F8E761A6D304B9B4EC69BDAE8

comment:6 Changed 2 years ago by teor

An operator wants this relay taken off the whitelist and put on the blacklist:

855BC2DABE24C861CD887DB9B2E950424B49FC34

I moved it from the blacklist to the whitelist last release by mistake.

comment:7 Changed 23 months ago by teor

An operator wants this relay taken off the whitelist and put on the blacklist:

3B33F6FCA645AD4E91428A3AF7DC736AD9FB727B

It is being shut down.

comment:8 Changed 23 months ago by teor

An operator wants to opt-in:
https://lists.torproject.org/pipermail/tor-relays/2018-January/014267.html

D122094E396DF8BA560843E7B983B0EA649B7DF9

comment:9 Changed 23 months ago by teor

An operator wants to opt-in:
https://lists.torproject.org/pipermail/tor-relays/2018-January/014276.html

32828476F4F84E15C42B4C360A5CD8DE4C3C2BE7

comment:10 Changed 22 months ago by teor

An operator wants this relay taken off the whitelist and put on the blacklist:
0BEA4A88D069753218EAAAD6D22EA87B9A1319D6
It is being shut down.
See #25169.

comment:11 Changed 22 months ago by teor

An operator wants this relay taken off the whitelist and put on the blacklist:
9FBEB75E8BC142565F12CBBE078D63310236A334
It is being shut down.
See #25414.

comment:12 Changed 21 months ago by teor

Removal for 328E54981C6DDD7D89B89E418724A4A7881E3192 , see #25436

comment:13 Changed 21 months ago by nickm

Keywords: 034-triage-20180328 added

comment:14 Changed 21 months ago by nickm

Keywords: 034-removed-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

comment:15 Changed 21 months ago by teor

Owner: set to teor
Status: newassigned

This ticket is part of the minimal fallback directory refresh for 0.3.4 or 0.3.5.

I can do it, or pastly can do it, or some volunteer may do it.

Edit: autocorrect

Last edited 21 months ago by teor (previous) (diff)

comment:16 Changed 21 months ago by teor

An operator wants the IP address of 6A7551EEE18F78A9813096E82BF84F740D32B911 changed to 94.130.186.5.

They posted to tor-relays, but somehow the message didn't make it into the archives, possibly because the subject was very long.

comment:17 Changed 21 months ago by teor

A relay operator is shutting down these two fallbacks:
175921396C7C426309AB03775A9930B6F611F794
1DBAED235E3957DE1ABD25B4206BE71406FB61F8

https://lists.torproject.org/pipermail/tor-relays/2018-April/014944.html

comment:18 Changed 20 months ago by nickm

Milestone: Tor: 0.3.4.x-finalTor: unspecified

These tickets, tagged with 034-removed-*, are no longer in-scope for 0.3.4. We can reconsider any of them, if time permits.

comment:19 Changed 20 months ago by crazysane

comment:20 Changed 20 months ago by teor

An operator wants treadstone (185.129.249.124) removed from the whitelist, see #25798.

comment:21 Changed 20 months ago by teor

Milestone: Tor: unspecifiedTor: 0.3.4.x-final
Points: 20.5

This ticket is in scope for 0.3.4 or 0.3.5 as part of the minimal fallback rebuild task.

We will do a rebuild when 25% of fallbacks are down, which is 37/150:
https://consensus-health.torproject.org/graphs.html#fallbackdirstatus

comment:22 Changed 20 months ago by teor

Description: modified (diff)

Instead of updating fallback details in the whitelist, we should do #24838 and ignore them. We believe onionoo, and onionoo checks stability anyway.

comment:23 Changed 20 months ago by teor

Owner: teor deleted

Anyone can do this

comment:24 Changed 20 months ago by teor

Milestone: Tor: 0.3.4.x-finalTor: 0.3.5.x-final

8/150 = 5% of fallbacks are currently down.
https://consensus-health.torproject.org/graphs.html
At this rate, it will be 2019 before we reach our 25% failure threshold.
So we don't need to do a fallback rebuild in 0.3.4.

comment:25 Changed 20 months ago by teor

An operator wants 774555642FDC1E1D4FDF2E0C31B7CA9501C5C9C7 removed from the whitelist:
https://lists.torproject.org/pipermail/tor-relays/2018-May/015146.html

comment:26 Changed 19 months ago by teor

An operator wants the following relay removed:

Nickname: DigiGesTor2e1
Fingerprint: 88487BDD980BF6E72092EE690E8C51C0AA4A538C
IP: 176.10.104.243

...
It will be shutdown end of June 2018.

They want to add this relay instead:

Nickname: DigiGesTor3e1
Fingerprint:9AA3FF35E7A549D2337E962333D366E102FE4D50
IP:94.230.208.147
IPv6:2a02:418:6017::147

comment:27 Changed 19 months ago by teor

Summary: Update fallback whitelist and blacklistUpdate fallback whitelist and blacklist in late 2018

comment:28 Changed 19 months ago by phoul

Cc: phoul added

comment:29 Changed 18 months ago by teor

An operator wants their relay removed from the whitelist:
https://lists.torproject.org/pipermail/tor-dev/2018-June/013207.html

comment:30 Changed 18 months ago by teor

An operator wants E947C029087FA1C3499BEF5D4372947C51223D8F added to the fallback list.

comment:31 Changed 18 months ago by teor

Phoul has a draft commit that includes all of the relays above (except for those that don't have DirPorts):
https://github.com/Phoul/tor/commit/306ab8ae17b8f39a6b83bcb24fe02afd8389dc13

comment:32 Changed 18 months ago by phoul

Another draft has been created with the current operators who have responded to the mailing list post at https://lists.torproject.org/pipermail/tor-relays/2018-June/015521.html and Twitter postings:

https://github.com/Phoul/tor/commit/23d9ffcdf4877f574278b0f0b39ce511072898af

comment:33 Changed 18 months ago by teor

That branch has 20 new and 2 changed IPv4 addresses.

Our target is >30% of the final count having new or changed IPv4 addresses, we currently have a best case scenario of:
(2 + 20)/(150 + 20) = 13%.

If we have 170 fallbacks, we can also increase the limit per operator to 170 x 5% = 8 relays.
That should gain us 5 more relays from existing large operators, which gives us 15% new or changed IPv4 addresses.

comment:34 Changed 17 months ago by teor

Cc: teor@… removed

Remove useless CC

comment:35 in reply to:  33 Changed 17 months ago by phoul

I have just pushed another commit with 9 more relays added to the fallback list. I've also followed up with large relay operators today to see if they have any more to offer us.

https://github.com/Phoul/tor/commit/d7e7c0d20d15d90882bc0b963f897a99b5d7585a

(there is a spacing issue in this commit on the final line, the following commit fixes it)

Replying to teor:

That branch has 20 new and 2 changed IPv4 addresses.

Our target is >30% of the final count having new or changed IPv4 addresses, we currently have a best case scenario of:
(2 + 20)/(150 + 20) = 13%.

If we have 170 fallbacks, we can also increase the limit per operator to 170 x 5% = 8 relays.
That should gain us 5 more relays from existing large operators, which gives us 15% new or changed IPv4 addresses.

comment:36 Changed 17 months ago by nickm

Keywords: 035-removed-20180711 added
Milestone: Tor: 0.3.5.x-finalTor: unspecified

These tickets are being triaged out of 0.3.5. The ones marked "035-roadmap-proposed" may return.

comment:37 Changed 17 months ago by phoul

https://github.com/Phoul/tor/commit/855feab70d35dd5093aae1f27d6cf19ee6cb3338 3 more fallback directory mirrors added, a number of operators have been contacted to see if they would like any others from their relay families added.

comment:39 in reply to:  36 Changed 17 months ago by teor

Keywords: 035-roadmap added

Replying to nickm:

These tickets are being triaged out of 0.3.5. The ones marked "035-roadmap-proposed" may return.

Oops, this is on the 0.3.5 roadmap, if:

  • we reach the 30% changed/new relays threshold, so the list can help with censorship
  • we reach the 25% down relays threshold in consensus health

Here's how we're progressing:

The branch now has ~35 new and ~2 changed IPv4 addresses.

Our target is >30% of the final count having new or changed IPv4 addresses, we currently have a best case scenario of:
(2 + 35)/(150 + 35) = 20%.

If we have 185 fallbacks, we can also increase the limit per operator to 185 x 5% = 9 relays.
That should gain us 10 more relays from existing large operators, which gives us 24% new or changed IPv4 addresses.

20/150 = 13% of fallbacks are down.

comment:40 Changed 17 months ago by teor

Status: assignednew

Make everything that is assigned to no-one new again.

comment:41 Changed 16 months ago by phoul

Owner: set to phoul
Status: newassigned

On Monday I sent out another round of emails to large operators asking for fallback mirrors, today I received 1 more and heard from other operators they are considering which to add.

I have added the new one to https://github.com/Phoul/tor/commit/0260b720053717cdc348a51ede1d7e51e7ea2741.

comment:43 Changed 16 months ago by teor

One fallback has been terminated by the network provider:
https://lists.torproject.org/pipermail/tor-relays/2018-August/015886.html

comment:44 in reply to:  43 Changed 16 months ago by phoul

Replying to teor:

One fallback has been terminated by the network provider:
https://lists.torproject.org/pipermail/tor-relays/2018-August/015886.html

Thanks for adding this to the ticket, I have updated the repo at https://github.com/Phoul/tor/commit/51f8db1d92d64331d000d2847edb8b66e9fb4c0c

comment:45 Changed 16 months ago by teor

Milestone: Tor: unspecifiedTor: 0.3.5.x-final
Status: assignedneeds_revision
Summary: Update fallback whitelist and blacklist in late 2018Update fallback whitelist in late 2018

Since #26502, there is no blacklist any more.

comment:46 Changed 16 months ago by teor

25% of fallbacks are failing, so these tickets need to be done in 0.3.5.

comment:47 Changed 16 months ago by teor

Phoul, see also #27297.

comment:48 in reply to:  47 Changed 16 months ago by phoul

Replying to teor:

Phoul, see also #27297.

This relay has been added at https://github.com/Phoul/tor/commit/f0792537e9cf25e76b5cab16635de7b78bfbf8b7

comment:49 Changed 15 months ago by teor

Hi Phoul,

An operator just told me that 30C19B81981F450C402306E2E7CFB6C3F79CB6B2 is going to change from 64.113.32.29 to 198.232.165.2 . You can update the list if you'd like, but #24838 removes all the IP address checks from the fallback script.

After #26502 and #24838, we can't say "don't use this relay until it changes to this IP address". That's ok, it's a very rare case. It not worth the hassle of trying to keep all the addresses up to date.

I opened #27737 so we can manually remove 64.113.32.29 if it appears in the generated list.

comment:50 in reply to:  49 Changed 15 months ago by phoul

Replying to teor:

Hi Phoul,

An operator just told me that 30C19B81981F450C402306E2E7CFB6C3F79CB6B2 is going to change from 64.113.32.29 to 198.232.165.2 . You can update the list if you'd like, but #24838 removes all the IP address checks from the fallback script.

This has been changed at https://github.com/Phoul/tor/commit/1663a1dd633213b750fe72b72b8c1bca7e15cd0a

comment:52 Changed 14 months ago by phoul

Removed 2 relays from the fallback list at https://github.com/Phoul/tor/commit/57bba19bbeb74d97f3e429259a19b5b6b1f8be53 (operators requested removal on tor-relays)

&

Added 1 relay to the fallback list at https://github.com/Phoul/tor/commit/5033e950ccd0e0f37134f5c4124e1f79d4d784b3

comment:54 Changed 14 months ago by phoul

Removed 1 relay from fallback list at operators request: https://github.com/Phoul/tor/commit/17f4388b0fa377f7bbdf6f9df23ff429a717d4f8

comment:56 Changed 12 months ago by teor

Keywords: s8-bootstrap added
Milestone: Tor: 0.3.5.x-finalTor: 0.4.0.x-final
Sponsor: Sponsor8-can
Status: needs_revisionneeds_review

We're creating the new fallback list over the next few days.

I created a pull request for the whitelist updates in:
https://github.com/torproject/tor/pull/572

I based it on Phoul's branch, and added a changes file.

Phoul, if you can update the actual points in the ticket with the number of days you spent on this task, that would be great!

comment:57 Changed 12 months ago by teor

Component: Core Tor/Fallback ScriptsCore Tor/Tor

This ticket makes changes to tor, so it goes in the tor component for review.

comment:58 Changed 12 months ago by dgoulet

Status: needs_reviewmerge_ready

CI is happy.

comment:59 Changed 12 months ago by teor

CI doesn't run on the fallback whitelist, but Phoul and I both used this branch in #24803 to run the fallback generation script.

comment:60 Changed 12 months ago by nickm

Merged!

Please close or un-parent child tickets as appropriate, and then close this ticket? :)

comment:61 Changed 12 months ago by teor

Resolution: fixed
Status: merge_readyclosed

The new fallback whitelist is merged, so these tickets are done.

Note: See TracTickets for help on using tickets.