Opened 10 months ago

Last modified 2 weeks ago

#29258 reopened task

Provide an IPv6 address for the Snowflake broker

Reported by: ahf Owned by: dcf
Priority: Medium Milestone:
Component: Circumvention/Snowflake Version:
Severity: Normal Keywords: anti-censorship-roadmap-august
Cc: dcf, arlolra, cohosh Actual Points:
Parent ID: Points:
Reviewer: Sponsor: Sponsor28-must

Description

We have a bit of a tendency to forget to test IPv6 solutions properly and in a structured way. We should make sure that IPv6 is working properly with Snowflake.

Child Tickets

Change History (19)

comment:1 Changed 7 months ago by phw

Keywords: anti-censorship-roadmap-maybe added

A brief summary from an anti-censorship meeting in which we discussed snowflake and IPv6:

  • Clients already can connect to snowflake proxies over IPv6.
  • Our broker currently has no IPv6 address.
  • We should have a way to ensure that an IPv6-only Tor Browser can use snowflake (see #29259).

comment:2 Changed 6 months ago by gaba

Keywords: anti-censorship-roadmap added

Adding this tickets to the backlog.

comment:3 Changed 6 months ago by gaba

Keywords: anti-censorship-roadmap-maybe removed

comment:4 Changed 6 months ago by phw

Sponsor: Sponsor19Sponsor28-must

Moving from Sponsor 19 to Sponsor 28.

comment:5 Changed 5 months ago by arlolra

Milestone: Tor: unspecified
Version: Tor: unspecified

comment:6 Changed 5 months ago by dcf

Owner: set to dcf
Status: newassigned
  • Our broker currently has no IPv6 address.

I'm claiming responsibility for getting an IPv6 address for the broker.

comment:7 Changed 5 months ago by gaba

Keywords: anti-censorship-roadmap-august added; anti-censorship-roadmap removed

thanks!

comment:8 Changed 5 months ago by dcf

Status: assignedneeds_information

Back in 2017, I inquired about IPv6 addresses. The reply is that IPv6 is only supported in one of the Greenhost data centers, namely Amsterdam

...instances on our Amsterdam location we can give you an ipv6 prefix. Other locations don't have ipv6 available yet.

The bridge is in the Amsterdam location, so I activated IPv6 for it back then. But the broker is in the Hong Kong location. I sent another support request this week to ask whether anything had changed, but IPv6 is still not available in Hong Kong.

Unfortunately there are no ipv6 block available yet for our Hong Kong customers.

My proposed solution is to migrate the broker to the Amsterdam data center.

  1. Provision a new VM in Amsterdam.
  2. Set it up just as the current broker and rsync past logs to it.
  3. Change the snowflake-broker.bamsoftware.com DNS record to point to the new broker.
    1. Restart our proxy-go instances. Web badge and WebExtension instances should restart automatically.
  4. Run the two brokers in parallel for a while.
  5. Shut down the Hong Kong broker.

If all goes well, this plan means no required downtime. The downside I see is that during step 4, there will be two separate sets of logs (snowflake.log and metrics.log) being kept. We will need to either merge them, or ignore one copy during the transition.

comment:9 Changed 4 months ago by arlolra

My proposed solution is to migrate the broker to the Amsterdam data center.

Maybe get some resolution on #31232 first

comment:10 in reply to:  8 Changed 8 weeks ago by dcf

Replying to dcf:

My proposed solution is to migrate the broker to the Amsterdam data center.

I've set up a new host in the Amsterdam data center and documented the installation instructions at org/teams/AntiCensorshipTeam/SnowflakeBrokerInstallationGuide. I copied over usernames, passwords, and ssh/authorized_keys, but not the contents of home directories or logs. You should be able to SSH and sudo as before.

IP: 37.218.245.111
RSA:  2048 SHA256:dp0Xo/oN1qZfMuZnqgKEbeOsbU2qpDR60B5MLIRaAgg
DSA:  1024 SHA256:DF5ofogjGur02gv8/ciU3wFA+YHNuAhUlel9Uv2KBlo
ECDSA: 256 SHA256:6cskO6ch/kv2RbIMhTdwqpsd9vB8npzlZTlkWZJLoek

One problem: the IPv6 doesn't work :) I did the same thing I did last time with the bridge, but I cannot send nor receive IPv6 on the network interface. I'm going to contact support to see if I'm doing something wrong.

comment:11 Changed 7 weeks ago by dcf

Status: needs_informationneeds_review
Summary: What is the IPv6 story with SnowflakeProvide an IPv6 address for the Snowflake broker

I got the IPv6 situation sorted out. (Just needed a different prefix.) Here's the information. After we've switched to this new host, I'll update the SSH fingerprints in org/teams/AntiCensorshipTeam/SnowflakeBrokerSurvivalGuide.

37.218.245.111
2a00:c6c0:0:154:4:d8aa:b4e6:c89f

RSA:    2048 SHA256:dp0Xo/oN1qZfMuZnqgKEbeOsbU2qpDR60B5MLIRaAgg
DSA:    1024 SHA256:DF5ofogjGur02gv8/ciU3wFA+YHNuAhUlel9Uv2KBlo
ECDSA:   256 SHA256:6cskO6ch/kv2RbIMhTdwqpsd9vB8npzlZTlkWZJLoek
ED25519: 256 SHA256:fEkLvmu5woTvU6162I8Jxd2eHzsTnBshumtWICclWA4

There's a PGP-signed version of this information at
https://lists.torproject.org/pipermail/anti-censorship-team/2019-October/000040.html.

Now the question is what to do about handling the migration. We can talk about this at the next meeting on Thursday. All that's needed is to point the following DNS names to the new IPv4 and IPv6 addresses:

  • snowflake-broker.bamsoftware.com
  • snowflake-broker.freehaven.net (currently a CNAME for snowflake-broker.bamsoftware.com)
  • snowflake-broker.torproject.net

About logs, I'm thinking we just let the log files happen in parallel on the old and new hosts. Then after we've made the switch, we check the old logs for sanitization and publish them. Logs we publish in the future from the new broker will partially temporally overlap those from the old, but that should be no problem.

I've copied over the contents of people's home directories.

Last edited 7 weeks ago by dcf (previous) (diff)

comment:12 in reply to:  11 ; Changed 7 weeks ago by dcf

Replying to dcf:

Now the question is what to do about handling the migration. We can talk about this at the next meeting on Thursday.

Today we decided to start by pointing the snowflake-broker.torproject.net DNS, which is currently unused, at the new broker, so we can test it ourselves.

#32128 is for that.

comment:13 in reply to:  12 Changed 7 weeks ago by dcf

Replying to dcf:

Today we decided to start by pointing the snowflake-broker.torproject.net DNS, which is currently unused, at the new broker, so we can test it ourselves.

#32128 is for that.

snowflake-broker.torproject.net is now set up for us. Using the following proxy-go command and torrc I was able (using an IPv6 connection to the broker) to connect to myself and bootstrap to 100%.

./proxy-go -broker https://snowflake-broker.torproject.net
UseBridges 1
DataDirectory datadir

ClientTransportPlugin snowflake exec ./client \
-url https://snowflake-broker.torproject.net/ \
-ice stun:stun.l.google.com:19302 \
-log snowflake.log \
-max 3

Bridge snowflake 0.0.3.0:1

I did have to first upgrade the version of golang.org/x/crypto/acme/autocert compiled into the broker, for a protocol change:

go get -u golang.org/x/crypto/acme/autocert

Before doing this, I was getting these errors in the broker logs, linking to https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430.

2019/10/17 19:37:32 http: TLS handshake error from [scrubbed]: 403 urn:acme:error:unauthorized: Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.
2019/10/17 19:37:37 http: TLS handshake error from [scrubbed]: acme/autocert: missing certificate
2019/10/17 19:37:41 http: TLS handshake error from [scrubbed]: acme/autocert: missing certificate

comment:14 Changed 6 weeks ago by cohosh

Uh oh, just realized we've been losing metrics. CollecTor uses the torproject.new domain for the broker. Created this ticket: #32231

comment:15 Changed 6 weeks ago by cohosh

Status: needs_reviewmerge_ready

Okay I think we can go ahead and finish switching hosts now.

About logs, I'm thinking we just let the log files happen in parallel on the old and new hosts. Then after we've made the switch, we check the old logs for sanitization and publish them. Logs we publish in the future from the new broker will partially temporally overlap those from the old, but that should be no problem.

The metrics logs will be the largest problem (see #322131). I propose this for the switch:

  • stop the broker process on the new and old host
  • copy over all metrics log files from the old host to the new host
  • start the new host

We're going to lose partial metrics for the collection period that overlaps with the switch, but that actually happens every time we restart the broker since metrics for the time period (which is one) are stored in memory until the time period ends at which point they are written to a file.

So, maybe the better question to ask here is: is that okay and if not how do we solve it more generally?

comment:16 Changed 3 weeks ago by dcf

At 2019-11-14 21:23:00, applied these changes:

domain name record type old address new address
snowflake-broker.bamsoftware.com A 37.218.240.96 37.218.245.111
snowflake-broker.bamsoftware.com AAAA none 2a00:c6c0:0:154:4:d8aa:b4e6:c89f

After the DNS changes propagate, I need to restart snowflake-proxy-restartless. If I'm not mistaken, all other proxies will restart themselves and update on their own.

comment:17 in reply to:  16 Changed 3 weeks ago by dcf

Replying to dcf:

After the DNS changes propagate, I need to restart snowflake-proxy-restartless.

I just did sv restart snowflake-proxy-restartless. I'm planning to let the others just restart themselves naturally. One of them must have already done so, because https://snowflake-broker.torproject.net/debug is currently (2019-11-14 22:10:00) showing 2 standalone proxies:

current snowflakes available: 7
	standalone proxies: 2
	browser proxies: 5

comment:18 Changed 2 weeks ago by cohosh

Resolution: fixed
Status: merge_readyclosed

I'm going to close this because everything looks like it's switched over :)

comment:19 Changed 2 weeks ago by cohosh

Resolution: fixed
Status: closedreopened

Reopening because I think we still need to think about logs?

Note: See TracTickets for help on using tickets.