Bw Auths should have a backup bw server

We have not had bw measurements in the consensus since Sept 30th, because the VM hosting our measurement files has been down. We need to set up a backup server and have the bw authorities automatically switch over to it when they detect the primary is down.

added MikePerryIterationFires20111120 actualpoints::2 component::core tor/torflow owner::mikeperry points::2 priority::high resolution::fixed status::closed type::enhancement labels

Instructions for setting up a bw server are in appendix A of the bw auth readme:

https://gitweb.torproject.org/torflow.git/blob_plain/master:/NetworkScanners/BwAuthority/README.BwAuthorities

We may need more than 40Mbit burst upstream nowadays though, and possibly more monthly capacity, too.

Trac:
Cc: N/A to linus

Trac:
Cc: linus to linus, ln5

https://38.229.70.2 is up for me. What other services does it need to host?

Maybe we should have more than one server as the serving point for the test files. the bwauths can either have a failover setup, or just randomly pull from one of the list for their testing.

phobos set up this alternate server for us: https://38.229.70.19/

We need to set that as the primary server and have the bw auths fall back to .2 when .19 is down. This will require some code changes.

Tomb suggested that it would in fact be better to randomly alternate between the multiple file servers. This would make it so that when one scanner fails, the deviation from the mean measurements should be smaller. We should still alert just as noisily when one fails, though.

Happy to set one of these up on moria whenever somebody wants it

Trac:
Points: N/A to N/A
Type: defect to enhancement
Actualpoints: N/A to N/A

Trac:
Priority: normal to major

The bwauth support for this is implemented in origin/master. We are just waiting on one to actually exist that is listening on an IP.

There is one on archive.tp.o, but it is not exposed by IP... Also, is archive.tp.o the best machine? Isn't it at the same ISP as the current host (https://38.229.70.2/)?

Trac:
Cc: linus, ln5 to linus, ln5, phobos
Keywords: N/A deleted, MikePerryIterationFires20111120 added

38.229.70.2 has been dead for a year or so. The IP you've been using for the past year or so is aroides.torproject.org, which also hosts archive.torproject.org. The migration of aroides to our own VM server changed the IP. I don't understand why bwauth.torproject.org can't be used.

Can we fix them to do proper http/1.1 with a hostname? Or, if we can't, is it easy to have them fetch stuff not from / but from a directory below like /bwauth/ ?

Replying to phobos:

38.229.70.2 has been dead for a year or so. The IP you've been using for the past year or so is aroides.torproject.org, which also hosts archive.torproject.org. The migration of aroides to our own VM server changed the IP. I don't understand why bwauth.torproject.org can't be used.

Actually, https://38.229.70.2/ is currently in use by the bw auths as we speak. The plan was for it to stay in use and have the additional IP of https://38.229.72.16/ used for redundancy.

Again, I don't want to involve DNS because exits can fail at it, and I'd prefer we still scan them in that case.

We can figure out how to special-case DNS failures later, but right now my opinion is they are SoaT-domain failures. The bw auths should not need to worry about them at all.

Can we fix them to do proper http/1.1 with a hostname? Or, if we can't, is it easy to have them fetch stuff not from / but from a directory below like /bwauth/ ?

The subdirectory is fine. http/1.1 is not.

Trac:
Actualpoints: N/A to 2
Resolution: N/A to fixed
Status: new to closed
Points: N/A to 2

closed

changed time estimate to 16h

added 16h of time spent

moved to tpo/network-health/torflow#2014 (closed)