Regenerate fallback list for 0.2.9

We need to regenerate the fallback directory mirror list in 0.2.9 in case any of the 0.2.8 fallbacks have changed details or gone down.

This should not require another opt-in mailout, as we had ~70 additional fallbacks in 0.2.8 that were suitable but not selected.

We should also:

• restore the 120 day stability period and 99% uptime requirement that were reduced in 0.2.8 due to #18050 (moved)
• check the bandwdith range in the script's generated C comments
• check the IP version, netblock, port, and Exit flag proportions in the script's stderr output

Over the longer term, we could:

• reconsider whether to allow 2 fallbacks per operator (contact, family), while keeping 1 per IP
• decide whether to change to an opt-out system, where we includes fallbacks unless operators specifically opt-out

changed milestone to %Tor: 0.3.0.x-final

added 029-accepted TorCoreTeam201612 actualpoints::5 component::core tor/tor milestone::Tor: 0.3.0.x-final owner::teor parent::20172 points::3 priority::medium resolution::fixed severity::normal sponsor::U-can status::closed type::enhancement labels

Trac:
Description: We need to regenerate the fallback directory mirror list in 0.2.9 in case any of the 0.2.8 fallbacks have changed details or gone down.

This should not require another opt-in mailout, as we had ~70 additional fallbacks in 0.2.8 that were suitable but not selected.

We should also:

• check the bandwdith range in the script's generated C comments
• check the IP version, netblock, port, and Exit flag proportions in the script's stderr output

Over the longer term, we could:

• reconsider whether to allow 2 fallbacks per operator (contact, family), while keeping 1 per IP
• decide whether to change to an opt-out system, where we includes fallbacks unless operators specifically opt-out

to

We need to regenerate the fallback directory mirror list in 0.2.9 in case any of the 0.2.8 fallbacks have changed details or gone down.

This should not require another opt-in mailout, as we had ~70 additional fallbacks in 0.2.8 that were suitable but not selected.

We should also:

• restore the 120 day stability period and 99% uptime requirement that were reduced in 0.2.8 due to #18050 (moved)
• check the bandwdith range in the script's generated C comments
• check the IP version, netblock, port, and Exit flag proportions in the script's stderr output

Over the longer term, we could:

• reconsider whether to allow 2 fallbacks per operator (contact, family), while keeping 1 per IP
• decide whether to change to an opt-out system, where we includes fallbacks unless operators specifically opt-out

agreed; we should do this about once per release series. (Also perhaps you can recruit somebody else to do it together with you this time, so that two people have experience doing it?)

Trac:
Keywords: 029-proposed deleted, 029-accepted added
Milestone: Tor: 0.2.??? to Tor: 0.2.9.x-final

Replying to nickm:

agreed; we should do this about once per release series. (Also perhaps you can recruit somebody else to do it together with you this time, so that two people have experience doing it?)

Sure. It would also help if that person reviewed #17158 (moved), #17905 (moved), and #18749 (moved) in 0.2.8.

I've started a branch with some more fallback whitelist / blacklist changes: fallbacks-201605 on https://github/com/teor2345/tor.git

We should merge it when we next regenerate the hard-coded list.

Trac:
Points: medium to 3

The latest fallback whitelist / blacklist changes are based on maint-0.2.8 (shortly after 0.2.8.4-rc) and are in fallbacks-201606 on ​https://github/com/teor2345/tor.git

I've merged the changes in the above branch into bug19071 (targeted at 0.2.8), because there's no way I'm keeping two concurrent whitelist/blacklist versions.

So there's nothing extra that needs to be merged into 0.2.9 at this point in time.

I'm now collecting new fallback whitelist entries on my branch fallbacks-201607 on https://github.com/teor2345/tor.git

We need to rebuild the fallback list entirely in 0.2.9 to make sure we've fixed #19163 (moved) - as of July 2016, every recommended tor version supports ntor.

Since #19610 (moved) causes IPv6-only clients to ask 15 IPv6 fallback directories for microdescriptors, we should increase the fallback numbers to 200 or 300, so we have a reasonable number of IPv6 fallbacks.

What I'd like to do in August is:

• generate a draft list of 200 fallbacks,
• find relays that are not whitelisted, but have high enough bandwidth and stability to have been included in the list, and
• if the operators of these relays have not been contacted already, email them an opt-in request

What I'd like to do in September is:

• add any new entries to the whitelist/blacklist,
• generate an alpha list of 200 fallbacks,

What I'd like to do as our alphas start to stabilise (December 2016?) is:

• check each fallback on the existing list to see if it's still working (same key, IP, and delivers a consensus)
• comment-out the ones that aren't working

Trac:
Keywords: TorCoreTeam201609 deleted, TorCoreTeam201608 added

It's likely that #19989 (moved) applies to IPv6-only clients fetching microdescriptors from fallback directories (#19608 (moved)). So we should make sure there are some non-Exit IPv6-capable relays in the set.

I've updated the fallbacks-201607 branch on https://github.com/teor2345/tor.git based on #20010 (moved) and an operator email.

[fallbacks-201607 e7ed8bb] Update fallback addresses based on operator emails and tickets

We have extended the 0.2.9 release deadline, and so I can afford to do some of this in September.

I also want to:

• make a wiki page so someone else can update fallbacks if needed (and so I don't forget)
• change the fallback script so it has an "exclude existing" mode (exclude both the whitelist and blacklist), to make finding new potential fallbacks easier

Trac:
Keywords: TorCoreTeam201608 deleted, TorCoreTeam201609 added

Trac:
Status: new to assigned

I wrote a wiki page here: https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors

Trac:
Parent: N/A to #20172 (moved)

And I made an updated whitelist and blacklist branch: fallbacks-029 on my github. The changes to the 0.2.8 list are in #20170 (moved).

I am up to step 2 of https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors , but I can't seem to get a good OnionOO uptime document (#20193 (moved)), so I'm kind of stuck.