Ship list of fallback directory mirrors on long-term fixed IPv6 addresses

changed milestone to %Tor: 0.2.8.x-final

added component::core tor/tor ipv6 milestone::Tor: 0.2.8.x-final owner::teor parent::17158 points::medium priority::high resolution::fixed severity::normal sponsor::U status::closed type::enhancement labels

The ticket this ticket is referring to is #6027 (moved) (Directory authorities on IPv6) rather than # 6072 (OONI proxy).

(Just to avoid confusion.)

Karsten, if you make such a list it would be easy for me (and I do mean easy, trivially so) to just add a makefile target to each of the bundles that wgets it and puts it in the right place each time I make a new one.

Replying to erinn:

Karsten, if you make such a list it would be easy for me (and I do mean easy, trivially so) to just add a makefile target to each of the bundles that wgets it and puts it in the right place each time I make a new one.

Sounds great! Thanks!

nickm, how's that file supposed to look like when it's shipped with bundles? Would it simply contain a few dozen FallbackDir lines with ipv6 values?

I think that should work; make sure that they have IPv4 addresses too and that they're really likely to be stable for a long time. Also, test before shipping. :)

Okay. We should avoid writing yet another service for this if we can. I think Onionoo has almost everything we need, and the missing parts are quite easy to implement. What is missing is that Onionoo tracks relay address changes, and we'd also need a small script that converts Onionoo's output into a list of fallback addresses. Here's how this could work:

We add a new field to Onionoo to track address changes:
- The spec for that field would be: "last_changed_addresses": UTC timestamp (YYYY-MM-DD hh:mm:ss) when this relay last announced a different IPv4 or IPv6 address where it accepts OR or directory connections. Note that this timestamp is reset when a relay drops out of the consensus for more than 7 days, so that whenever that relay rejoins, it would seem as if it changed its address(es) at that time; this is a known limitation of the current Onionoo design. Required field.
- Note that the special case of leaving and rejoining relays is not relevant here, because we'd only look for stable relays that don't leave the network for more than a week.
- We'll want to feed Onionoo with consensuses from the past month or so to initialize these field values.
The script that would be run whenever a new 0.2.5.x (or higher) bundle is made would do these steps:
- Download the list of all currently running relays from https://onionoo.torproject.org/details?type=relay&running=true
- Discard all relays that don't have an IPv6 address in "or_addresses", that don't have a "dir_address" field, that have a "first_seen" or "last_changed_addresses" date younger than 4 weeks (or more/less depending on results), or that have "Authority" in their "flags" field.
- Sort the remaining relays by "consensus_weight" in descending order.
- Format the top-20 (or more/less depending on results) relays as: FallbackDir "dir_address" orport="or_addresses(ipv4 port)" id="fingerprint" weight="consensus_weight" ipv6="or_addresses(ipv6 address)"

Here's an example relay that might make it into the fallback directories list:

{"nickname":"Kiruna",
"fingerprint":"980D326017CEF4CBBF4089FBABE767DC83D059AF",
"or_addresses":["193.11.164.242:9001","[2001:6b0:7:125::242]:9001"],
"dir_address":"193.11.164.242:9030",
"last_seen":"2013-03-13 14:00:00",
"first_seen":"2013-01-21 15:00:00",
"running":true,
"flags":["Fast","Named","Running","V2Dir","Valid"],
"country":"se",
"latitude":62.0000,
"longitude":15.0000,
"country_name":"Sweden",
"as_number":"AS1653",
"as_name":"SUNET Swedish University Network",
"consensus_weight":16500,
[...]

The fallback directory line for this relay would be:

FallbackDir 193.11.164.242:9030 orport=9001 id=980D326017CEF4CBBF4089FBABE767DC83D059AF
  weight=16500 ipv6=[2001:6b0:7:125::242]

I can look into making the Onionoo changes, but I'd appreciate if someone else would work on the script. In theory, work on the script can begin now, except for the "last_changed_addresses" check that will have to be added once that field is present. I'll ask around.

Trac:
dir_list.py

Replying to karsten:

I can look into making the Onionoo changes, but I'd appreciate if someone else would work on the script. In theory, work on the script can begin now, except for the "last_changed_addresses" check that will have to be added once that field is present. I'll ask around.

Check attached script.

Trac:
Status: new to needs_review

Replying to gsathya:

Replying to karsten:

I can look into making the Onionoo changes, but I'd appreciate if someone else would work on the script. In theory, work on the script can begin now, except for the "last_changed_addresses" check that will have to be added once that field is present. I'll ask around.

Check attached script.

Cool! Looks correct. Thanks!

Moving this ticket to the Onionoo component to add the "last_changed_addresses" field.

Trac:
Status: needs_review to accepted
Owner: erinn to karsten
Component: Tor bundles/installation to Onionoo

Onionoo now has a new "last_changed_address_or_port" field which is a tweaked version of what I suggested above.

Using Sathya's script, I created a list of 10 fallback directories which I think can be shipped with 0.2.5.x bundles. Each relay in that list matches the following requirements:

has both an IPv4 and an IPv6 address,
has a non-zero DirPort,
is around for at least 180 days,
has not changed one of its IP addresses or ORPort/DirPort in the past 180 days, and
doesn't have the Authority flag.

Here's the list:

FallbackDir 77.247.181.162:80 orport=443 id=253DFF1838A2B7782BE7735F74E50090D46CA1BC weight=72700 ipv6=[2a00:1768:1001:21:1:0:32a3:201a]
FallbackDir 171.25.193.20:80 orport=443 id=DD8BD7307017407FCC36F8D04A688F74A0774C02 weight=50600 ipv6=[2001:67c:289c::20]
FallbackDir 128.6.224.107:9030 orport=9001 id=D67B28212377617448A2AC192E11372AD951FD13 weight=18000 ipv6=[2620:0:d60:401::2]
FallbackDir 82.94.251.204:80 orport=443 id=9B02AA745B22B3FAB37C84B5E695623DD107A74D weight=15100 ipv6=[2001:888:2133:0:82:94:251:204]
FallbackDir 188.40.51.232:80 orport=443 id=CAF7986ECF1FBF903E68155531F8930C9ECC3A0D weight=13900 ipv6=[2a01:4f8:100:24e1:ffff::1]
FallbackDir 193.11.164.242:9030 orport=9001 id=980D326017CEF4CBBF4089FBABE767DC83D059AF weight=13800 ipv6=[2001:6b0:7:125::242]
FallbackDir 171.25.193.21:80 orport=443 id=A10C4F666D27364036B562823E5830BC448E046A weight=13300 ipv6=[2001:67c:289c::21]
FallbackDir 149.20.52.51:9030 orport=5251 id=09C0E63BD41FE86A31CB3FB27C4D54F7D49A1F7C weight=12500 ipv6=[2001:4f8:3:2e::51]
FallbackDir 91.121.245.171:80 orport=443 id=85670C66276B84F956FC9F2407DAFF9774104522 weight=2550 ipv6=[2001:41d0:2:90a8::3]
FallbackDir 78.47.134.6:3480 orport=3451 id=26220AEA188B8D0E47BB541E1A616EB3AD70295F weight=2360 ipv6=[2a01:4f8:d13:1602::2012]

Does this list look sane?

I'm attaching the updated script that I used to generate this list. Run it like this:

curl "https://onionoo.torproject.org/details?type=relay&running=true" > details.json
python dir_list.py details.json fallback-dirs 180 180 20

Erinn, can you integrate this script and instructions into the build process for 0.2.5.x bundles? Of course, as Nick says above, we'll want to test those new bundles before shipping.

Trac:
Component: Onionoo to Tor bundles/installation

Trac:
dir_list.2.py

Updated dir_list.py script.

I can, but we don't currently have any 0.2.5.x bundles. What is the ETA for 0.2.5.x? And just so we're clear, do we only want this in TBB? Or do we want it for all of them? And if I make a test bundle (based on master, presumably), which one would be best for me to make so that you can check it out?

I think we want this in all bundles, not just TBB.

I wonder, would it make things easier if we put this file in src/config/ in the tor repo? I could update this file once per month whenever I update the geoip file and send a joint pull request. That would reduce a couple of potential problems from generating this file for various bundles and platforms. I could clean up the dir_list.2.py script to include it in src/config/, too.

With respect to testing, I guess either an OS X TBB or a Debian package would be easiest for me. Testing should be easy: enable info-level logging, check for warnings (or success messages if there are any) when parsing the fallback dir file, and check if the first directory requests go to fallback directories and not to directory authorities. Well, the IPv4 testing part is easy, the IPv6 part requires doing the same on an IPv6-only client.

Putting it in src/config is probably better because it doesn't rely on various network services or other machines functioning. If once a month is frequent enough to keep fallback descriptors fresh then that sounds like a great solution to me.

Are there any deadlines for this? Let me know how to prioritize the testing package for you. I can get you one by the end of the month, unless it's more urgent in which case I can try to get it for you this week or sooner. I will make an OSX TBB.

Replying to erinn:

Putting it in src/config is probably better because it doesn't rely on various network services or other machines functioning. If once a month is frequent enough to keep fallback descriptors fresh then that sounds like a great solution to me.

nickm, is this a workable solution? If so, I could prepare a branch similar to the geoip update branch once per month for you to merge. I could even do two commits in a single branch if that makes things simpler. Let me know what works best.

Are there any deadlines for this? Let me know how to prioritize the testing package for you. I can get you one by the end of the month, unless it's more urgent in which case I can try to get it for you this week or sooner. I will make an OSX TBB.

No deadlines in the next weeks. Take your time. Thanks!

Replying to karsten:

Replying to erinn:

Putting it in src/config is probably better because it doesn't rely on various network services or other machines functioning. If once a month is frequent enough to keep fallback descriptors fresh then that sounds like a great solution to me.

nickm, is this a workable solution? If so, I could prepare a branch similar to the geoip update branch once per month for you to merge. I could even do two commits in a single branch if that makes things simpler. Let me know what works best.

That ought to work. (If a set of fallback directory sources is not going to be a good choice for several months, it is probably not a good set of fallback directory sources after all.)

Replying to karsten:

Replying to erinn:

Are there any deadlines for this? Let me know how to prioritize the testing package for you. I can get you one by the end of the month, unless it's more urgent in which case I can try to get it for you this week or sooner. I will make an OSX TBB.

No deadlines in the next weeks. Take your time. Thanks!

In fact, let's first test and merge #6027 (moved) before making testing packages. I'll ping you. Thanks.

Replying to karsten:

In fact, let's first test and merge #6027 (moved) before making testing packages. I'll ping you. Thanks.

Noted. Thanks!

Ship list of fallback directory mirrors on long-term fixed IPv6 addresses

Child items 0

Activity