Opened 2 months ago

Closed 12 days ago

#31232 closed defect (fixed)

Migrate default snowflake broker (and bridge?) to TPA machines

Reported by: cohosh Owned by: tpa
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords: snowflake
Cc: anarcat, hiro, cohosh, phw, dcf, arlolra Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We've talked about this off and on in the past and there are a few good reasons for doing it:

  • Have redundant access and permissions to the DNS/hosts for the default broker and bridge URLs
  • Right now we are having an issue with the existing bamsoftware.com subdomains due to a Google safe browsing service blacklist (#31230)

Child Tickets

Change History (17)

comment:1 Changed 2 months ago by cohosh

Noting that migration to a TPO domain doesn't necessarily solve the problem in #31230 and might make our lives more difficult if Google decides torproject.org subdomains are unsafe due to snowflake.

comment:2 Changed 2 months ago by arlolra

Also note that the current broker doesn't have an ipv6 address, so we might have to do some migration anyways. See comment:8:ticket:29258

comment:3 Changed 7 weeks ago by anarcat

Status: newneeds_information

i'm not exactly sure how to act on this ticket, unfortunately, so it would help if you could provide me us with a little more information.. :)

  1. what, exactly, do you want to move? how many machines, with how much memory, disk and bandwidth do you need?
  2. when do you need this to happen?
  3. what kind of access do you require on those machines? who will need those accesses?

I also wonder how this interacts with your (separate) requests for monitoring in #29863 and #31159: there you asked us to monitor external resources and we started setting up an external monitoring server for that purpose. Now, if we move those machines under TPA's wing, do we still need the external monitoring server?

Thanks for the clarifications!

comment:4 Changed 6 weeks ago by cohosh

There are a few different scenarios here. Basically what we want in the end is for us to be able to handle outages and maintenance to key snowflake infrastructure as an organization without relying solely on dcf. The key infrastructure here is the snowflake bridge (which is hard-coded into the proxies and therefore difficult to update a change of IP address quickly), and the snowflake broker (think of this as bridgeDB but for Snowflake).

Right now both the bridge and the broker are on hosts and domains owned by dcf. We as the anti-censorship team have access to the hosts, but if something goes wrong with the domain (as happened in #31230), our hands are still tied. We hacked together a temporary fix by pointing freehaven.net domains to the bridge and broker machines but that shouldn't be our permanent solution.

So to answer your questions:

1) We definitely need a tp.org (or tp.net)? domain to point to the broker and bridge IP addresses. There is a rumour going around that we only want to use these domains for hosts we control. If that is true, then we need a domain-fronted snowflake broker host and a snowflake bridge host. I suppose it's up to the sysadmin team as to whether these are each their own machines or not. The resource usage doesn't seem too bad at the moment but the bridge will need a lot of data transfer. I can be more specific about resource numbers if needed. I think dcf said he'd be happy to keep hosting these services as long as we're happy to point our domains at his machines but I'll let him confirm that.

2) It's not urgent because things are back up and running now but I think it's a good idea to keep the ball rolling on this. Now if Roger and dcf are unavailable we're in a tough spot again until we hack together another quick fix.

3) The anti-censorship team will need root access to both the bridge and broker for monitoring/logs/installation/update abilities.

As far as #29863 and #31159, we'll still want prometheus monitoring. I'm not sure whether this is something the anti-censorship team or the sysadmin team will "own" though. I suppose if TPA absorbs the snowflake infrastructure, then it is in a similar category as bridgedb or gettor and I'm not clear on where we are with who is in charge of monitoring this infrastructure at the moment.

comment:5 Changed 6 weeks ago by anarcat

  1. if you just need a domain, and not a machine, that is fast and quick. i *think* the policy is that non-TPA machines get a torproject.net hostname, but I can verify that. if you *do* want us to host the machine as well, you will definitely need to be most specific on this.
  1. yes, it seems like a good idea to figure out a fix for this in the long term
  1. allowing root on TPA machines is a problem. we don't normally allow that: we handle the OS-level stuff and grant you access to an account (with the sudo access you need, e.g. restart apache or something) on a case-by-case basis

Regarding "monitoring", if the machine is TPA, you get a ton of monitoring for free. If it's non-TPA, you get the experimental/external Prometheus server that doesn't do alerts.

So I guess the next step is to actually decide if we absorb this or not, and for that we need more precise numbers. I need to know how much memory (in GB), disk (size and HDD or SSD or NVMe too), CPU (count and type) and bandwidth usage (in TB/mth or gbps 95th percentile) we're talking about.

Thanks!

comment:6 Changed 6 weeks ago by phw

At today's anti-censorship meeting, we talked about snowflake migration. One argument against The Tor Project being involved in running snowflake infrastructure is our "we don't run the network" principle. Conceptually, the broker is similar to our bridge authority (not run by TPO) and BridgeDB (run by TPO). Bridges report themselves to the bridge authority, and their descriptors are then copied over to BridgeDB, which takes care of bridge distribution. The broker combines these two tasks – but it does so for snowflake proxies, and not bridges. We're therefore in a bit of a grey area here. (FWIW, I don't want to steer this discussion in either direction – just wanted to provide a bit of context.)

comment:7 Changed 6 weeks ago by anarcat

would it be possible to split the snowflake architecture in two parts, the same way bridgedb and bridges authority are separate, and would that help us here?

also: i'm not sure what the next step is, but if you need a torproject.net domain, just let me know. :)

comment:8 in reply to:  7 Changed 6 weeks ago by cohosh

Replying to anarcat:

would it be possible to split the snowflake architecture in two parts, the same way bridgedb and bridges authority are separate, and would that help us here?

That would be difficult to do with the way snowflake currently works. It won't happen in the near future and it seems unlikely overall.

also: i'm not sure what the next step is, but if you need a torproject.net domain, just let me know. :)

Yes, let's go ahead on this. Can we get a snowflake-broker.torproject.net domain and a snowflake.torproject.net domain for the broker and bridge, respectively?

comment:9 Changed 6 weeks ago by anarcat

Yes, let's go ahead on this. Can we get a snowflake-broker.torproject.net domain and a snowflake.torproject.net domain for the broker and bridge, respectively?

Sure, where do you want those to point at?

comment:10 in reply to:  9 Changed 6 weeks ago by cohosh

Replying to anarcat:

Yes, let's go ahead on this. Can we get a snowflake-broker.torproject.net domain and a snowflake.torproject.net domain for the broker and bridge, respectively?

Sure, where do you want those to point at?

$ host snowflake-broker.bamsoftware.com
snowflake-broker.bamsoftware.com has address 37.218.240.96
$ host snowflake.bamsoftware.com
snowflake.bamsoftware.com has address 37.218.242.151
snowflake.bamsoftware.com has IPv6 address 2a00:c6c0:0:151:4:8f94:69f5:7c01

comment:11 Changed 6 weeks ago by anarcat

sorry for my dumb questions, but should we make those CNAMEs to snowflake.bamsoftware.com and so on, or A/AAAA records? if the former, you need to ask bamsoftware to make changes, if the latter, you need to ask us.

i'd assume you want the latter, but wanted to double-check.

comment:12 in reply to:  11 Changed 6 weeks ago by cohosh

Replying to anarcat:

sorry for my dumb questions, but should we make those CNAMEs to snowflake.bamsoftware.com and so on, or A/AAAA records? if the former, you need to ask bamsoftware to make changes, if the latter, you need to ask us.

i'd assume you want the latter, but wanted to double-check.

No worries, thanks for asking. We want the latter so that if the IP addresses of the broker/bridge have to change for any reason we can make those changes.

comment:13 Changed 6 weeks ago by anarcat

alright, added the following to the torproject.net zone:

; https://trac.torproject.org/projects/tor/ticket/31232
snowflake-broker	IN	A	37.218.240.96
snowflake		IN	A	37.218.242.151
			IN	AAAA	2a00:c6c0:0:151:4:8f94:69f5:7c01

This should propagate through the DNS with time:

root@nevii:~# dig snowflake-broker.torproject.net +short
37.218.240.96
root@nevii:~# dig snowflake.torproject.net +short
37.218.242.151
root@nevii:~# dig snowflake.torproject.net +short AAAA
2a00:c6c0:0:151:4:8f94:69f5:7c01

Leaving this ticket open for the rest of the conversation, which is probably "do we move stuff to TPA now?"

comment:14 Changed 6 weeks ago by cohosh

Thank you anarchat! I just updated /etc/runit/snowflake-broker/run on the broker host and /etc/tor/torrc on the bridge host to get Let's Encrypt certificates for the torproject.net domains. Seems to be working :)

comment:15 Changed 2 weeks ago by dcf

Keywords: snowflake added

comment:16 Changed 12 days ago by anarcat

excellent, so is this done? is there more infrastructure that should be moved inside TPA somehow?

thanks!

comment:17 Changed 12 days ago by cohosh

Resolution: fixed
Status: needs_informationclosed

Yes, let's say it's done for now. I think dcf is content with hosting these machines for the moment and it's nice to not have Tor Project "run the network" as explained above. If we want to revisit this, we can reopen this ticket or make a new one.

Thank you anarcat for setting up these domains!

Note: See TracTickets for help on using tickets.