Domain fronting to App Engine stopped working

added component::circumvention/snowflake moat priority::medium resolution::wontfix severity::normal status::closed type::defect labels

Trac:

https://metrics.torproject.org/rs.html#details/5481936581E23D2D178105D44DB6915AB06BFB7F

I am estimating the time by looking at the Relay Search page for the Snowflake bridge. It shows nonzero bandwidth at 2018-04-13 14:00:00 and zero bandwidth starting at 2018-04-13 18:00:00.

This HTTP request has a Host header that is not covered \ by the TLS certificate used. Due to an infrastructure change, \ this request cannot be processed.

Due to an infrastructure change Reason: Zello app (amazon then google fronting) versus censorship. Global business sold all users. Nothing personal just a business.

Seems like Signal was affected by this as well (and they seem to have been aware of it in advance - from Mar 27, 2018): https://github.com/signalapp/Signal-Android/pull/7584 https://github.com/signalapp/Signal-Android/commit/a573ab7c7668360c3ab411627bbb23109ef9facc

*.appspot.com still works as domain fronting.

Replying to cypherpunks:

This HTTP request has a Host header that is not covered \ by the TLS certificate used. Due to an infrastructure change, \ this request cannot be processed.

Replying to dcf:

Here is a cheesy proof of concept. It's not suitable because it disable certificate verification (InsecureSkipVerify). What's needed is another parameter to verify the certificate as if we had accessed www.google.com (or other specific domain).

https://golang.org/pkg/crypto/tls/#Config (VerifyPeerCertificate) https://golang.org/pkg/crypto/x509/#Certificate.Verify

Actually, as #25807 (moved) shows, this is not only affecting snowflake but moat, too.

Trac:
Cc: dcf, arlolra to dcf, arlolra, gk

I corrected the month in the ticket description (April instead of March).

Trac:
Cc: dcf, arlolra, gk to dcf, arlolra, gk, brade, mcs
Description: On or about 2018-03-13 16:00:00 UTC, domain-fronted requests for snowflake-reg.appspot.com stopped working. It appears to affect fronting to all appspot.com domains, not only ours. This leaves all currently deployed clients unable to register themselves.

Requests now fail with status code 502:

$ wget -q -O - --content-on-error -S https://www.google.com/ --header 'Host: snowflake-reg.appspot.com'
  HTTP/1.1 502 Bad Gateway
  Date: Sun, 15 Apr 2018 04:58:49 GMT
  Content-Type: text/html
  Server: HTTP server (unknown)
  Content-Length: 209
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Alt-Svc: hq=":443"; ma=2592000; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="42,41,39,35"
<html><body><h1>502 Bad Gateway</h1>\
<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

This ticket is to document the issue; I'm not sure we can do anything about it directly.

Other related tickets:

#22782 (moved), use non-Google domain fronts
#25594 (moved), use non-fronting-based registration

to

On or about 2018-04-13 16:00:00 UTC, domain-fronted requests for snowflake-reg.appspot.com stopped working. It appears to affect fronting to all appspot.com domains, not only ours. This leaves all currently deployed clients unable to register themselves.

Requests now fail with status code 502:

$ wget -q -O - --content-on-error -S https://www.google.com/ --header 'Host: snowflake-reg.appspot.com'
  HTTP/1.1 502 Bad Gateway
  Date: Sun, 15 Apr 2018 04:58:49 GMT
  Content-Type: text/html
  Server: HTTP server (unknown)
  Content-Length: 209
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Alt-Svc: hq=":443"; ma=2592000; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="42,41,39,35"
<html><body><h1>502 Bad Gateway</h1>\
<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

This ticket is to document the issue; I'm not sure we can do anything about it directly.

Other related tickets:

#22782 (moved), use non-Google domain fronts
#25594 (moved), use non-fronting-based registration

Trac:
Keywords: N/A deleted, moat added
Description: On or about 2018-04-13 16:00:00 UTC, domain-fronted requests for snowflake-reg.appspot.com stopped working. It appears to affect fronting to all appspot.com domains, not only ours. This leaves all currently deployed clients unable to register themselves.

Requests now fail with status code 502:

$ wget -q -O - --content-on-error -S https://www.google.com/ --header 'Host: snowflake-reg.appspot.com'
  HTTP/1.1 502 Bad Gateway
  Date: Sun, 15 Apr 2018 04:58:49 GMT
  Content-Type: text/html
  Server: HTTP server (unknown)
  Content-Length: 209
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Alt-Svc: hq=":443"; ma=2592000; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="42,41,39,35"
<html><body><h1>502 Bad Gateway</h1>\
<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

This ticket is to document the issue; I'm not sure we can do anything about it directly.

Other related tickets:

#22782 (moved), use non-Google domain fronts
#25594 (moved), use non-fronting-based registration

to

On or about 2018-04-13 16:00:00 UTC, domain-fronted requests for *.appspot.com stopped working. It appears to affect fronting to all appspot.com domains, not only ours. This has broken Snowflake client registration and Moat (#25807 (moved)).

Requests now fail with status code 502:

$ wget -q -O - --content-on-error -S https://www.google.com/ --header 'Host: snowflake-reg.appspot.com'
  HTTP/1.1 502 Bad Gateway
  Date: Sun, 15 Apr 2018 04:58:49 GMT
  Content-Type: text/html
  Server: HTTP server (unknown)
  Content-Length: 209
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Alt-Svc: hq=":443"; ma=2592000; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="42,41,39,35"
<html><body><h1>502 Bad Gateway</h1>\
<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

This ticket is to document the issue; I'm not sure we can do anything about it directly.

Other related tickets:

#22782 (moved), use non-Google domain fronts
#25594 (moved), use non-fronting-based registration

It seems worthwhile in a general sense to try to salvage domain fronting with appengine if it's easy to do.

In addition, it seems smart to move to a world where the Snowflake client ships with more than one potential domain fronting domain. Then it should be able to survive one of them going away without us needing to ship an updated Tor Browser to the affected users (who suddenly also find themselves without a working privacy tool to fetch the updates).

It seems worthwhile in a general sense to try to salvage domain fronting with appengine if it's easy to do.

SNI-less or *.appspot.com (like generated randomly)?

A Google update just created a big problem for anti-censorship tools

Reached by The Verge, Google said the changes were the result of a long-planned network update. “Domain fronting has never been a supported feature at Google,” a company representative said, “but until recently it worked because of a quirk of our software stack. We’re constantly evolving our network, and as part of a planned software update, domain fronting no longer works. We don’t have any plans to offer it as a feature.”

We’re constantly evolving our network, and as part of a planned software update, domain fronting no longer works

Some google's code diff (insight job)

+ if Host == appspot && SNI != appspot { // Let's break it

youtube (or anything else) didn't affected.

still works via plain http

Trac:
Username: djsf

https solution: use $WHATEVER.appspot.com (generated name or some popular application) instead of google.com to circumvent DNS censorship:

wget -q -O - --content-on-error -S https://innocent-app.appspot.com/ --header 'Host: snowflake-reg.appspot.com'

Trac:
Username: djsf

Domain fronting to App Engine stopped working

Child items ...

Activity