Opened 7 months ago

Closed 3 months ago

#25804 closed defect (wontfix)

Domain fronting to App Engine stopped working

Reported by: dcf Owned by:
Priority: Medium Milestone:
Component: Obfuscation/Snowflake Version:
Severity: Normal Keywords: moat
Cc: dcf, arlolra, gk, brade, mcs Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by dcf)

On or about 2018-04-13 16:00:00 UTC, domain-fronted requests for *.appspot.com stopped working. It appears to affect fronting to all appspot.com domains, not only ours. This has broken Snowflake client registration and Moat (#25807).

Requests now fail with status code 502:

$ wget -q -O - --content-on-error -S https://www.google.com/ --header 'Host: snowflake-reg.appspot.com'
  HTTP/1.1 502 Bad Gateway
  Date: Sun, 15 Apr 2018 04:58:49 GMT
  Content-Type: text/html
  Server: HTTP server (unknown)
  Content-Length: 209
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Alt-Svc: hq=":443"; ma=2592000; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="42,41,39,35"
<html><body><h1>502 Bad Gateway</h1>\
<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

This ticket is to document the issue; I'm not sure we can do anything about it directly.

Other related tickets:

  • #22782, use non-Google domain fronts
  • #25594, use non-fronting-based registration

Child Tickets

Change History (43)

comment:1 Changed 7 months ago by dcf

I am estimating the time by looking at the Relay Search page for the Snowflake bridge. It shows nonzero bandwidth at 2018-04-13 14:00:00 and zero bandwidth starting at 2018-04-13 18:00:00.

https://metrics.torproject.org/rs.html#details/5481936581E23D2D178105D44DB6915AB06BFB7F

comment:2 Changed 7 months ago by cypherpunks

<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

No domain fronting to App Engine but works without SNI

comment:3 Changed 7 months ago by cypherpunks

Due to an infrastructure change

Reason: Zello app (amazon then google fronting) versus censorship.
Global business sold all users. Nothing personal just a business.

comment:4 Changed 7 months ago by cypherpunks

Seems like Signal was affected by this as well (and they seem to have been aware of it in advance - from Mar 27, 2018): https://github.com/signalapp/Signal-Android/pull/7584 https://github.com/signalapp/Signal-Android/commit/a573ab7c7668360c3ab411627bbb23109ef9facc

Last edited 7 months ago by cypherpunks (previous) (diff)

comment:5 Changed 7 months ago by cypherpunks

*.appspot.com still works as domain fronting.

comment:6 in reply to:  2 ; Changed 7 months ago by dcf

Replying to cypherpunks:

<p>This HTTP request has a Host header that is not covered \
by the TLS certificate used. Due to an infrastructure change, \
this request cannot be processed.</p></body></html>

No domain fronting to App Engine but works without SNI

I confirm that this is the case. Resolve www.google.com to an IP address, access the server via its IP address (need to override the certificate check) and pass a Host header:

$ dig +short www.google.com
172.217.11.164
$ wget --content-on-error --save-header --no-check-certificate -q -O- https://172.217.11.164/ip --header 'Host: snowflake-reg.appspot.com'
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
X-Cloud-Trace-Context: b0805cfcb7d0d60a3f5352c65879afaa
Date: Sun, 15 Apr 2018 22:18:54 GMT
Server: Google Frontend
Content-Length: 13
Alt-Svc: hq=":443"; ma=2592000; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="42,41,39,35"

X.X.X.X

Related meek ticket (not implemented):

  • #12208: Make it possible to use an IP address as a front

If someone has a ticket for SNI-less Snowflake rendezvous, it would be very welcome. The relevant code is here:
https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/rendezvous.go?id=c61336c897b5d21cc94a21241e98b33df5dcbf78#n61

Here is a cheesy proof of concept. It's not suitable because it disable certificate verification (InsecureSkipVerify). What's needed is another parameter to verify the certificate as if we had accessed www.google.com (or other specific domain).

  • client/rendezvous.go

    diff --git a/client/rendezvous.go b/client/rendezvous.go
    index cab7f5a..c74e041 100644
    a b package main 
    1414import (
    1515        "bufio"
    1616        "bytes"
     17        "crypto/tls"
    1718        "errors"
    1819        "io/ioutil"
    1920        "log"
     21        "net"
    2022        "net/http"
    2123        "net/url"
    2224        "os"
    type BrokerChannel struct { 
    4648func CreateBrokerTransport() http.RoundTripper {
    4749        transport := http.DefaultTransport.(*http.Transport)
    4850        transport.Proxy = nil
     51        // haxxx
     52        transport.TLSClientConfig = &tls.Config{
     53                InsecureSkipVerify: true,
     54        }
    4955        return transport
    5056}
    5157
    func NewBrokerChannel(broker string, front string, transport http.RoundTripper) 
    6167        bc := new(BrokerChannel)
    6268        bc.url = targetURL
    6369        if "" != front { // Optional front domain.
    64                 log.Println("Domain fronting using:", front)
     70                var addr net.Addr
     71                addr, err = net.ResolveIPAddr("ip", front)
     72                if nil != err {
     73                        addr, err = net.ResolveTCPAddr("tcp", front)
     74                        if nil != err {
     75                                return nil
     76                        }
     77                }
     78                log.Println("Domain fronting using:", addr)
    6579                bc.Host = bc.url.Host
    66                 bc.url.Host = front
     80                bc.url.Host = addr.String()
    6781        }
    6882
    6983        bc.transport = transport

comment:7 in reply to:  6 ; Changed 7 months ago by yawning

Replying to dcf:

Here is a cheesy proof of concept. It's not suitable because it disable certificate verification (InsecureSkipVerify). What's needed is another parameter to verify the certificate as if we had accessed www.google.com (or other specific domain).

https://golang.org/pkg/crypto/tls/#Config (VerifyPeerCertificate)
https://golang.org/pkg/crypto/x509/#Certificate.Verify

comment:8 Changed 7 months ago by gk

Actually, as #25807 shows, this is not only affecting snowflake but moat, too.

comment:9 Changed 7 months ago by gk

Cc: gk added

comment:10 Changed 7 months ago by mcs

Cc: brade mcs added
Description: modified (diff)

I corrected the month in the ticket description (April instead of March).

comment:11 Changed 7 months ago by dcf

Description: modified (diff)
Keywords: moat added

comment:12 Changed 7 months ago by arma

It seems worthwhile in a general sense to try to salvage domain fronting with appengine if it's easy to do.

In addition, it seems smart to move to a world where the Snowflake client ships with more than one potential domain fronting domain. Then it should be able to survive one of them going away without us needing to ship an updated Tor Browser to the affected users (who suddenly also find themselves without a working privacy tool to fetch the updates).

comment:13 Changed 7 months ago by cypherpunks

It seems worthwhile in a general sense to try to salvage domain fronting with appengine if it's easy to do.

SNI-less or *.appspot.com (like generated randomly)?

comment:14 Changed 7 months ago by cypherpunks

A Google update just created a big problem for anti-censorship tools

Reached by The Verge, Google said the changes were the result of a long-planned network update. “Domain fronting has never been a supported feature at Google,” a company representative said, “but until recently it worked because of a quirk of our software stack. We’re constantly evolving our network, and as part of a planned software update, domain fronting no longer works. We don’t have any plans to offer it as a feature.”

comment:15 Changed 7 months ago by cypherpunks

We’re constantly evolving our network, and as part of a planned software update, domain fronting no longer works

Some google's code diff (insight job)

+ if Host == appspot && SNI != appspot { // Let's break it

youtube (or anything else) didn't affected.

comment:16 Changed 7 months ago by djsf

still works via plain http

comment:17 Changed 7 months ago by djsf

https solution: use $WHATEVER.appspot.com (generated name or some popular application) instead of google.com to circumvent DNS censorship:

wget -q -O - --content-on-error -S https://innocent-app.appspot.com/ --header 'Host: snowflake-reg.appspot.com'

Last edited 7 months ago by djsf (previous) (diff)

comment:18 in reply to:  17 Changed 7 months ago by cypherpunks

Replying to djsf:

https solution: use $WHATEVER.appspot.com (generated name or some popular application) instead of google.com to circumvent DNS censorship:

wget -q -O - --content-on-error -S https://innocent-app.appsppot.com/ --header 'Host: snowflake-reg.appspot.com'

Censors will just block the front innocent-app.appspot.com (it's already blocked in China FWIW). ¯\_(ツ)_/¯

And anyways, because of #22782 migrating from Google fronts is a necessity. DNS-over-HTTPS with https://1.1.1.1/dns-query, https://1.0.0.1/dns-query (fallback in case 1.1.1.1 isn't reachable), https://mozilla.cloudflare-dns.com/dns-query and https://cloudflare-dns.com/dns-query (not blocked in China according to [1][2]) seems promising tho. So this seems like the best time to upgrade with both of those two solutions with the former acting as fallback (because 1.1.1.1 is free and Amazon isn't).


[1] : https://www.greatfirewallofchina.org/index.php?siteurl=cloudflare-dns.com

[2] : https://www.comparitech.com/privacy-security-tools/blockedinchina/

Last edited 7 months ago by cypherpunks (previous) (diff)

comment:19 Changed 7 months ago by cypherpunks

Censors will just block the front innocent-app.appspot.com (it's already blocked in China FWIW). ¯\_(ツ)_/¯

Google blocked by range of IP addresses.

comment:20 in reply to:  19 Changed 7 months ago by neel

Replying to cypherpunks:

Censors will just block the front innocent-app.appspot.com (it's already blocked in China FWIW). ¯\_(ツ)_/¯

Google blocked by range of IP addresses.

However, I can front a non-appspot.com domain with another non-appspot.com domain. For instance, I can front youtube.com with google.com.

neel@xb2:~ % wget -q -O - --content-on-error -S https://www.google.com/ --header 'Host: www.youtube.com'

And it fetches YouTube's homepage.

But this probably won't work for us because we have to use appspot in order to be able to front.

comment:21 Changed 7 months ago by coldsauce

Why not use Souq or some other CDN as the domain front like Signal is doing?

comment:22 in reply to:  21 Changed 7 months ago by cypherpunks

Replying to coldsauce:

Why not use Souq or some other CDN as the domain front like Signal is doing?

It's already pointed out in the ticket's post:

Other related tickets:

  • #22782, use non-Google domain fronts
  • #25594, use non-fronting-based registration

comment:23 Changed 7 months ago by cypherpunks

Seems like Signal was affected by this as well (and they seem to have been aware of it in advance - from Mar 27, 2018)

Rumors:

Hey, everyone. We spent a decent amount of time at Signal trying to come up with alternatives when we first heard rumors that Google was disabling domain fronting on GAE.

comment:24 in reply to:  23 ; Changed 7 months ago by cypherpunks

Replying to cypherpunks:

Seems like Signal was affected by this as well (and they seem to have been aware of it in advance - from Mar 27, 2018)

Rumors:

Hey, everyone. We spent a decent amount of time at Signal trying to come up with alternatives when we first heard rumors that Google was disabling domain fronting on GAE.

Wow, maybe sometime in the future will start learning about Google deprecating this or that from SecureDrop leakers at the nytimes/bezos post. Truly epic that they didn't even bother to put a notice or something, not even a two line blog post. Maybe people should no longer base things off anything Google related, and maybe RMS was right.

comment:25 Changed 7 months ago by twim

Hi there,

It turns out that AppEngine is not the only option for domain fronting with Google.
Google also provides a service called AMP cache for AMP pages. What it basically does is proxying random pages on the Internet and making them load faster (e.g. on Google search results). It requires pages to comply with some format though and also strips invisible content, resizes images, etc.
Despite it is being served via different domain names (one per real domain) it is still hosted at Google infrastructure which can be fronted.

I wrote a library that implements wrappers around AMP cache for tunneling traffic through it.
I've also made a hacky pluggable transport thing as a PoC and managed to bootstrap tor using it. Have to say that no one should ever use AMP cache as an actual PT because it makes tons of requests (so you will probably be banned by Google) and it is incredibly slow.

I guess that this can be a pretty good fit for both Moat and Snowflake use cases.

comment:26 in reply to:  25 Changed 7 months ago by dcf

Replying to twim:

It turns out that AppEngine is not the only option for domain fronting with Google.
Google also provides a service called AMP cache for AMP pages. What it basically does is proxying random pages on the Internet and making them load faster (e.g. on Google search results). It requires pages to comply with some format though and also strips invisible content, resizes images, etc.
Despite it is being served via different domain names (one per real domain) it is still hosted at Google infrastructure which can be fronted.

Thanks, twim, this is good work. Would you create a new ticket for the Snowflake part and link it from #25594?

I think, for the broker side, all we would need to do is add a new route, /amp/client or whatever, which is the same as the existing /client except that it adds the AMP header and trailer.

comment:28 Changed 7 months ago by cypherpunks

comment:29 in reply to:  7 Changed 7 months ago by dcf

Replying to yawning:

Replying to dcf:

Here is a cheesy proof of concept. It's not suitable because it disable certificate verification (InsecureSkipVerify). What's needed is another parameter to verify the certificate as if we had accessed www.google.com (or other specific domain).

https://golang.org/pkg/crypto/tls/#Config (VerifyPeerCertificate)
https://golang.org/pkg/crypto/x509/#Certificate.Verify

I posted some prototype code in comment:11:ticket:12208. I would appreciate some review on it. It handles our use case of doing a TLS handshake without SNI, but still verifying the certificate.

comment:30 Changed 7 months ago by cypherpunks

Why not use Souq or some other CDN as the domain front like Signal is doing?

Amazon threatens to suspend Signal's AWS account over censorship circumvention

Yesterday AWS became aware of your Github and Hacker News/ycombinator posts describing how Signal plans to make its traffic look like traffic from another site

They reads?!
RedTeamers is like censors, censors is like nazi.
Punch a nazi in the face.

comment:31 in reply to:  30 Changed 7 months ago by cypherpunks

Replying to cypherpunks:

Why not use Souq or some other CDN as the domain front like Signal is doing?

Amazon threatens to suspend Signal's AWS account over censorship circumvention

Glad m0xie didn't shut up about this thing and voiced it loudly! It's receiving enough coverage now on Hacker News which, unfortunately, has a lot of pro-censorship pro-Bezos apologia (Russian Internet Agency trolls?) https://news.ycombinator.com/item?id=16970199

Yesterday AWS became aware of your Github and Hacker News/ycombinator posts describing...

"became aware" lol

Last edited 7 months ago by cypherpunks (previous) (diff)

comment:32 in reply to:  24 Changed 7 months ago by cypherpunks

Replying to cypherpunks:

Replying to cypherpunks:

Rumors:

Hey, everyone. We spent a decent amount of time at Signal trying to come up with alternatives when we first heard rumors that Google was disabling domain fronting on GAE.

Wow, maybe sometime in the future will start learning about Google deprecating this or that from SecureDrop leakers at the nytimes/bezos post. Truly epic that they didn't even bother to put a notice or something, not even a two line blog post. Maybe people should no longer base things off anything Google related, and maybe RMS was right.

This is no longer accurate, Moxie did receive a 30 day notice and the reason isn't related to Telegram but following lobbying efforts to not block requests from the fine people in Iran,

In early 2018, a number of policy organizations increased pressure on Google to change their position on how they were interpreting US sanction law so that domain fronting would be possible from Iran. Sadly, these lobbying efforts seem to have had the opposite effect. When Google’s leadership became more aware of domain fronting, it generated internal conversations about whether they wanted to put themselves in the situation of providing cover for sites that entire countries wished to block.

A month later, we received 30-day advance notice from Google that they would be making internal changes to stop domain fronting from working entirely.

comment:33 Changed 7 months ago by cypherpunks

Loud Amazon were given the choice between dictator's $$$ and reputation. Amazon chose $$$...
Learn google's history, amazon. Dictator would steal your $$$ and kill you later. Do stuff silently next time.

comment:34 in reply to:  28 Changed 7 months ago by dcf

Replying to cypherpunks:

Make Meek Great Again?

Yes, there is some discussion about encrypted SNI and other related topics here:

IETF draft: SNI Encryption in TLS Through Tunneling
Secondary Cert Authentication.

Unfortunately I haven't thought about them very much or how they may be implemented. This is a good place for someone to get involved. There are more ideas than there are people to go after them.

About Secondary Cert Authentication, Nick Sullivan of Cloudflare gave a (fairly non-technical) talk about it at USENIX Enigma 2018: https://www.youtube.com/watch?v=xZN0H3jzwys

comment:35 Changed 7 months ago by cypherpunks

Crazy idea perhaps, but could HTTPS Everywhere collect a list of real-world *.appspot.com subdomains from people who have enabled the certificate observatory?

comment:36 Changed 7 months ago by cypherpunks

Don't know how much data Moat and Snowflake need, but if it's only a tiny amount an alternative for the AMP proxy could be Google's favicon retrieval service, which allows to retrieve one 16×16 PNG at the time. Could perhaps be combined with wildcard DNS so you get <some random id>.some-endpoint.torproject.org/favicon.ico, <another random id>.some-endpoint.torproject.org/favicon.ifo

curl --output example.png -H 'Host: www.google.com' https://ssl.google-analytics.com/s2/favicons?domain=torproject.org

8 bits for every layer (red, green, blue and alpha) makes for 16*16*4 = 1024 bytes returned per request.

Last edited 7 months ago by cypherpunks (previous) (diff)

comment:37 in reply to:  35 Changed 6 months ago by dcf

Replying to cypherpunks:

Crazy idea perhaps, but could HTTPS Everywhere collect a list of real-world *.appspot.com subdomains from people who have enabled the certificate observatory?

Yes; but I think it would have limited benefit. *.appspot.com is blocked in China already (I don't know about elsewhere). If someone found some really solid *.appspot.com domains it might be worth doing.

comment:38 in reply to:  36 Changed 6 months ago by dcf

Replying to cypherpunks:

Don't know how much data Moat and Snowflake need, but if it's only a tiny amount an alternative for the AMP proxy could be Google's favicon retrieval service, which allows to retrieve one 16×16 PNG at the time.

That's neat, I like it :) I briefly ran the numbers on Snowflake for #25874. The client needs to send about 700 bytes, or about 500 bytes if compressed; and receive a similar amount. Unfortunately that's too long for a single DNS name (max 255 bytes). Sending an entire client offer will take multiple DNS requests, so it probably won't work in the favicon service.

Moat is even harder, at least as currently implemented. It doesn't use a single request/response; it uses a tunneled TLS connection atop multiple serialized requests and responses. Making it work over a single request/response would require rearchitecting the protocol so that Moat messages have their own confidentiality and integrity protection, independent of TLS.

comment:39 Changed 6 months ago by cypherpunks

Don't know how much data Moat and Snowflake need, but if it's only a tiny amount an alternative for the AMP proxy could be Google's favicon retrieval service, which allows to retrieve one 16×16 PNG at the time. Could perhaps be combined with wildcard DNS so you get <some random id>.some-endpoint.torproject.org/favicon.ico, <another random id>.some-endpoint.torproject.org/favicon.ifo

That's neat for sure.
But Google support DNS-over-HTTPS (Beta version).

comment:40 Changed 6 months ago by cypherpunks

Google's favicon retrieval service, which allows to retrieve one 16×16 PNG at the time.

Output PNG up to 256px, but you need to specify size.

duckduckgo.com&sz=64

comment:42 Changed 3 months ago by dcf

Resolution: wontfix
Status: newclosed

Closing this as it is well understood by now.

Note: See TracTickets for help on using tickets.