Opened 5 days ago

Last modified 9 hours ago

#25804 new defect

Domain fronting to App Engine stopped working

Reported by: dcf Owned by:
Priority: Medium Milestone:
Component: Obfuscation/Snowflake Version:
Severity: Normal Keywords: moat
Cc: dcf, arlolra, gk, brade, mcs Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by dcf)

On or about 2018-04-13 16:00:00 UTC, domain-fronted requests for *.appspot.com stopped working. It appears to affect fronting to all appspot.com domains, not only ours. This has broken Snowflake client registration and Moat (#25807).

Requests now fail with status code 502:

$ wget -q -O - --content-on-error -S https://www.google.com/ --header 'Host: snowflake-reg.appspot.com'
  HTTP/1.1 502 Bad Gateway
  Date: Sun, 15 Apr 2018 04:58:49 GMT
  Content-Type: text/html
  Server: HTTP server (unknown)
  Content-Length: 209
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Alt-Svc: hq=":443"; ma=2592000; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="42,41,39,35"
<html><body><h1>502 Bad Gateway</h1>\
<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

This ticket is to document the issue; I'm not sure we can do anything about it directly.

Other related tickets:

  • #22782, use non-Google domain fronts
  • #25594, use non-fronting-based registration

Child Tickets

Change History (21)

comment:1 Changed 5 days ago by dcf

I am estimating the time by looking at the Relay Search page for the Snowflake bridge. It shows nonzero bandwidth at 2018-04-13 14:00:00 and zero bandwidth starting at 2018-04-13 18:00:00.

https://metrics.torproject.org/rs.html#details/5481936581E23D2D178105D44DB6915AB06BFB7F

comment:2 Changed 5 days ago by cypherpunks

<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

No domain fronting to App Engine but works without SNI

comment:3 Changed 5 days ago by cypherpunks

Due to an infrastructure change

Reason: Zello app (amazon then google fronting) versus censorship.
Global business sold all users. Nothing personal just a business.

comment:4 Changed 5 days ago by cypherpunks

Seems like Signal was affected by this as well (and they seem to have been aware of it in advance - from Mar 27, 2018): https://github.com/signalapp/Signal-Android/pull/7584 https://github.com/signalapp/Signal-Android/commit/a573ab7c7668360c3ab411627bbb23109ef9facc

Last edited 5 days ago by cypherpunks (previous) (diff)

comment:5 Changed 5 days ago by cypherpunks

*.appspot.com still works as domain fronting.

comment:6 in reply to:  2 ; Changed 4 days ago by dcf

Replying to cypherpunks:

<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

No domain fronting to App Engine but works without SNI

I confirm that this is the case. Resolve www.google.com to an IP address, access the server via its IP address (need to override the certificate check) and pass a Host header:

$ dig +short www.google.com
172.217.11.164
$ wget --content-on-error --save-header --no-check-certificate -q -O- https://172.217.11.164/ip --header 'Host: snowflake-reg.appspot.com'
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
X-Cloud-Trace-Context: b0805cfcb7d0d60a3f5352c65879afaa
Date: Sun, 15 Apr 2018 22:18:54 GMT
Server: Google Frontend
Content-Length: 13
Alt-Svc: hq=":443"; ma=2592000; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="42,41,39,35"

X.X.X.X

Related meek ticket (not implemented):

  • #12208: Make it possible to use an IP address as a front

If someone has a ticket for SNI-less Snowflake rendezvous, it would be very welcome. The relevant code is here:
https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/rendezvous.go?id=c61336c897b5d21cc94a21241e98b33df5dcbf78#n61

Here is a cheesy proof of concept. It's not suitable because it disable certificate verification (InsecureSkipVerify). What's needed is another parameter to verify the certificate as if we had accessed www.google.com (or other specific domain).

  • client/rendezvous.go

    diff --git a/client/rendezvous.go b/client/rendezvous.go
    index cab7f5a..c74e041 100644
    a b package main 
    1414import (
    1515        "bufio"
    1616        "bytes"
     17        "crypto/tls"
    1718        "errors"
    1819        "io/ioutil"
    1920        "log"
     21        "net"
    2022        "net/http"
    2123        "net/url"
    2224        "os"
    type BrokerChannel struct { 
    4648func CreateBrokerTransport() http.RoundTripper {
    4749        transport := http.DefaultTransport.(*http.Transport)
    4850        transport.Proxy = nil
     51        // haxxx
     52        transport.TLSClientConfig = &tls.Config{
     53                InsecureSkipVerify: true,
     54        }
    4955        return transport
    5056}
    5157
    func NewBrokerChannel(broker string, front string, transport http.RoundTripper) 
    6167        bc := new(BrokerChannel)
    6268        bc.url = targetURL
    6369        if "" != front { // Optional front domain.
    64                 log.Println("Domain fronting using:", front)
     70                var addr net.Addr
     71                addr, err = net.ResolveIPAddr("ip", front)
     72                if nil != err {
     73                        addr, err = net.ResolveTCPAddr("tcp", front)
     74                        if nil != err {
     75                                return nil
     76                        }
     77                }
     78                log.Println("Domain fronting using:", addr)
    6579                bc.Host = bc.url.Host
    66                 bc.url.Host = front
     80                bc.url.Host = addr.String()
    6781        }
    6882
    6983        bc.transport = transport

comment:7 in reply to:  6 Changed 4 days ago by yawning

Replying to dcf:

Here is a cheesy proof of concept. It's not suitable because it disable certificate verification (InsecureSkipVerify). What's needed is another parameter to verify the certificate as if we had accessed www.google.com (or other specific domain).

https://golang.org/pkg/crypto/tls/#Config (VerifyPeerCertificate)
https://golang.org/pkg/crypto/x509/#Certificate.Verify

comment:8 Changed 4 days ago by gk

Actually, as #25807 shows, this is not only affecting snowflake but moat, too.

comment:9 Changed 4 days ago by gk

Cc: gk added

comment:10 Changed 4 days ago by mcs

Cc: brade mcs added
Description: modified (diff)

I corrected the month in the ticket description (April instead of March).

comment:11 Changed 3 days ago by dcf

Description: modified (diff)
Keywords: moat added

comment:12 Changed 2 days ago by arma

It seems worthwhile in a general sense to try to salvage domain fronting with appengine if it's easy to do.

In addition, it seems smart to move to a world where the Snowflake client ships with more than one potential domain fronting domain. Then it should be able to survive one of them going away without us needing to ship an updated Tor Browser to the affected users (who suddenly also find themselves without a working privacy tool to fetch the updates).

comment:13 Changed 38 hours ago by cypherpunks

It seems worthwhile in a general sense to try to salvage domain fronting with appengine if it's easy to do.

SNI-less or *.appspot.com (like generated randomly)?

comment:14 Changed 20 hours ago by cypherpunks

A Google update just created a big problem for anti-censorship tools

Reached by The Verge, Google said the changes were the result of a long-planned network update. “Domain fronting has never been a supported feature at Google,” a company representative said, “but until recently it worked because of a quirk of our software stack. We’re constantly evolving our network, and as part of a planned software update, domain fronting no longer works. We don’t have any plans to offer it as a feature.”

comment:15 Changed 19 hours ago by cypherpunks

We’re constantly evolving our network, and as part of a planned software update, domain fronting no longer works

Some google's code diff (insight job)

+ if Host == appspot && SNI != appspot { // Let's break it

youtube (or anything else) didn't affected.

comment:16 Changed 17 hours ago by djsf

still works via plain http

comment:17 Changed 16 hours ago by djsf

https solution: use $WHATEVER.appspot.com (generated name or some popular application) instead of google.com to circumvent DNS censorship:

wget -q -O - --content-on-error -S https://innocent-app.appspot.com/ --header 'Host: snowflake-reg.appspot.com'

Last edited 16 hours ago by djsf (previous) (diff)

comment:18 in reply to:  17 Changed 16 hours ago by cypherpunks

Replying to djsf:

https solution: use $WHATEVER.appspot.com (generated name or some popular application) instead of google.com to circumvent DNS censorship:

wget -q -O - --content-on-error -S https://innocent-app.appsppot.com/ --header 'Host: snowflake-reg.appspot.com'

Censors will just block the front innocent-app.appspot.com (it's already blocked in China FWIW). ¯\_(ツ)_/¯

And anyways, because of #22782 migrating from Google fronts is a necessity. DNS-over-HTTPS with https://1.1.1.1/dns-query, https://1.0.0.1/dns-query (fallback in case 1.1.1.1 isn't reachable), https://mozilla.cloudflare-dns.com/dns-query and https://cloudflare-dns.com/dns-query (not blocked in China according to [1][2]) seems promising tho. So this seems like the best time to upgrade with both of those two solutions with the former acting as fallback (because 1.1.1.1 is free and Amazon isn't).


[1] : https://www.greatfirewallofchina.org/index.php?siteurl=cloudflare-dns.com

[2] : https://www.comparitech.com/privacy-security-tools/blockedinchina/

Last edited 16 hours ago by cypherpunks (previous) (diff)

comment:19 Changed 15 hours ago by cypherpunks

Censors will just block the front innocent-app.appspot.com (it's already blocked in China FWIW). ¯\_(ツ)_/¯

Google blocked by range of IP addresses.

comment:20 in reply to:  19 Changed 9 hours ago by neel

Replying to cypherpunks:

Censors will just block the front innocent-app.appspot.com (it's already blocked in China FWIW). ¯\_(ツ)_/¯

Google blocked by range of IP addresses.

However, I can front a non-appspot.com domain with another non-appspot.com domain. For instance, I can front youtube.com with google.com.

neel@xb2:~ % wget -q -O - --content-on-error -S https://www.google.com/ --header 'Host: www.youtube.com'

And it fetches YouTube's homepage.

But this probably won't work for us because we have to use appspot in order to be able to front.

Note: See TracTickets for help on using tickets.