Make it possible to use an IP address as a front (no DNS request and no SNI)
meek puts one domain name on the "outside" of your connection (the DNS request and SNI), and a different name on the "inside" (the HTTP Host header). It would be good for some uses if the outside could be just to an IP address rather than a domain name, so that there were no DNS request, and no server_name extension in the CLientHello. Kind of like if you were to browse to https://38.229.72.16/ instead of https://www.torproject.org/.
The motivating use case is using a CDN as a front instead of www.google.com. A CDN has many domains behind it, but if we choose just one of them as the front, that domain might get blocked (because the collateral damage would be limited to just one domain). Such blocking would break the transport and also incidentally get the innocent third-party domain, who has nothing to do with any of this, censored even for non-circumventors. What we want is to use one of the CDN's frontend IP addresses as a front, so that the censor has to block the whole IP and the thousands of domains behind it, not just a single domain.
Activity
I didn't find an obvious way to reach this goal.
In the naked golang HTTPS code (without the browser extension), the first thing I tried was putting an IP address in place of www.google.com for the --front option.
-ClientTransportPlugin meek exec ./meek-client --url=https://meek-reflect.appspot.com/ --front=www.google.com +ClientTransportPlugin meek exec ./meek-client --url=https://meek-reflect.appspot.com/ --front=74.125.224.116It works fine up until certificate verification, then exits with this error:
error in handling request: x509: cannot validate certificate for 74.125.224.116 because it doesn't contain any IP SANsWhat we would want is to connect to an IP without a domain name, but still let the TLS code know that it should use "www.google.com" for verification purposes. It doesn't seem possible to do it with the crypto/tls library. The clientHandshake function builds a ClientHello using the value c.config.ServerName, and then later does verification using the same value c.config.ServerName. In other words, it doesn't look possible to use one value for the SNI and another during verification.
Everything works fine if you set InsecureSkipVerify on the TLSClientConfig of the Transport, but then of course you're not doing any verification.
The golang HTTPS library, when you connect to an IP address, sends a server_name extension whose value is the IP address as a string. When I tried the same thing in Firefox 29 just now, it simply leaves off the server_name extension.
How [[doc/GoAgent|GoAgent]] does it is it cooks up its own TLS connection using Python's ssl library, Python's ssl library doesn't do any verification, and then GoAgent itself does some nonstandard verification, not checking any certs but either looking for an organizationName starting with "Google " or looking for ".google" or ".appspot.com" in the commonName. It's essentially the same as InsecureSkipVerify.
I don't know yet if there's a way to do what we want in the Firefox and Chrome helpers.
A way to do this in Firefox may be nsICertOverrideService, specifically hasMatchingOverride. nsiCertOverrideService is the service that manages the overrides that get added when you click through a certificate warning (which are visible through Preferences→Advanced→Certificates→View Certificates, Lifetime=Temporary).
It looks like we can define hasMatchingOverride and request HTTPS to a CloudFront IP (for example). hasMatchingOverride will get called back. We check that the error is only ERROR_MISMATCH (and not ERROR_UNTRUSTED or ERROR_TIME). We check that the commonName in the cert is "*.cloudfront.net", and return true. In effect, we defer to the normal verification mechanisms for everything but hostname verification, and for that we check that the wildcard is what we expect.
skipCertError is a third-party Firefox extension to automatically ignore certificate errors.
Selenium's docs on hasMatchingOverride.
Psiphon and Lantern both independently discovered this issue. They found that connecting with no SNI is necessary when using the Fastly CDN. They both worked around it by patching the Go crypto/tls library.
Psiphon:
- Go http.Transport modified: no SNI in TSL client hello, new meek client binary\ https://bitbucket.org/psiphon/psiphon-circumvention-system/commits/c93134904eb67b5c36e80ee2b1cdf2e264113c22
Lantern:
- Bad news - I had to fork Go's tls library.\ https://groups.google.com/forum/#!topic/lantern-devel/ya1_kjzwWzw\ https://github.com/getlantern/tls/commit/c26f6d9c4b17213b1bd78a734465f2e33be3d12f
I contacted the ICSI Certificate Notary (notary.icsi.berkeley.edu), which monitors SSL traffic at some big Internet sites, asking what fraction of their monitored connections use SNI. The answer:
In the last month:
- 83.5% of connections had SNI.
- 80.1% of connections where certificates were exchanged (excluding e.g. resumed sessions) had SNI.
I wrote a post to the golang-nuts mailing list asking about this use case:
- https://groups.google.com/forum/#!topic/golang-nuts/bt9vBbZLYgU Brad Fitzpatrick replied to say that what we want should be possible once this in-progress issue is resolved:
- https://code.google.com/p/go/issues/detail?id=8522
Following discussion in the thread at https://groups.google.com/forum/#!topic/golang-nuts/bt9vBbZLYgU, Ox found a workaround that works with an unmodified crypto/tls library. It uses manual dialing and verification.
This is supposed to be possible now by providing a DialTLS callback.
https://groups.google.com/forum/#!msg/golang-nuts/bt9vBbZLYgU/_jvH4tPKcfkJ https://code.google.com/p/go/source/detail?r=455042166f13
Replying to cypherpunks:
When I tried the same thing in Firefox 29 just now, it simply leaves off the server_name extension.
Amazon CDN doesn't support for such requests now. Seems like it terminates TLS session if ClientHello lack of SNI?
Thanks for checking this. Could you add a note to [[doc/meek#AmazonCloudFront]]?
The past couple of days I've been experimenting with ways to set or omit the SNI. There's a demo program in names.go and I'd appreciate any review or questions about design decisions. I took inspiration from Ox's domainfront.go demo from 2014, but changes in the standard library (especially since Go 1.8) provide better ways to do some things now. If this looks good, I'm going to use it as the basis of new code in meek-client.
The basic situation is that we are managing four domain names:
-
urlName: this is the name we resolve and actually establish a TCP connection with. -
hostName: this is the name that goes in the HTTP Host header. -
sniName: this is the name that goes in the SNI extension. -
verifyName: this is the name that the client verifies the server certificate against. In normal everyday HTTPS, all four of these names are the same. Domain fronting allowshostNameto differ, but the other three names are the same. names.go shows how to make all four names independent, so you can, for example, send no SNI but still verify the server certificate against a hostname, while still fronting a different domain in the Host header.
An explanation of some decisions:
- The way to set the SNI is to modify
TLSClientConfig.ServerName; the way to set the verification name is to modify theTLSClientConfig.VerifyPeerCertificatecallback. Both of these are properties of http.Transport, which is a long-lived data structure that manages multiple HTTP roundtrips. We therefore need a separatehttp.Transportfor each unique (sniName,verifyName) pair. The only exception is whenurlName=sniName=verifyName: that's the built-in behavior of the net/http package, so we can share anhttp.Transportin that case. The functiongetHTTPTransportForNamescreates and caches newhttp.Transports as needed. (And note thathostNamedoes not enter this logic at all: whether or not you are domain fronting is orthogonal to certificate verification.) - The way to omit the SNI extension is to set
TLSClientConfig.ServerNameto an IP address. The IP address is not actually used for anything else, as long asTLSClientConfig.InsecureSkipVerifyis set. The trick of settingTLSClientConfig.TLSDial, and then pulling the certificates from thetls.ConnusingConnectionState, doesn't work when a proxy is set, because the proxy bypasses the dialer (which is the reason why VerifyPeerCertificate was introduced). - We need to use multiple simultaneous
http.Transports, so we can't just modify the global http.DefaultTransport like we do currently. Buthttp.DefaultTransporthas some nice default settings for things like timeouts. Go doesn't provide any method for creating a newhttp.Transportthat has the default settings, and you can't just copy the struct literal because it contains mutexes. So I used a reflection trick I found on a mailing list to copy just the public struct members. Also if we naively setTLSClientConfig, it disables HTTP/2 support; because we plan to modifyTLSClientConfig, we have to first callhttp2.ConfigureTransport.
Here are examples use cases. all names equal (ordinary HTTPS):: {{{ $ ./names -host example.com -sni example.com -verify example.com https://example.com/ urlName: "example.com" hostName: "example.com" sniName: "example.com" verifyName: "example.com" HTTP/2.0 200 OK }}} omit SNI, but still verify:: {{{ $ ./names -host example.com -sni "" -verify example.com https://example.com/ urlName: "example.com" hostName: "example.com" sniName: "" verifyName: "example.com" HTTP/2.0 200 OK }}} omit DNS request and SNI, but still verify:: {{{ $ dig +short example.com 93.184.216.34 $ ./names -host example.com -sni "" -verify example.com https://93.184.216.34/ urlName: "93.184.216.34" hostName: "example.com" sniName: "" verifyName: "example.com" HTTP/2.0 200 OK }}} ask for one name in the SNI but verify against another (still valid) name:: {{{ $ ./names -host example.com -sni example.com -verify www.example.net https://example.com/ urlName: "example.com" hostName: "example.com" sniName: "example.com" verifyName: "www.example.net" HTTP/2.0 200 OK }}} try verifying against a name that's not valid for the certificate:: {{{ $ ./names -host example.com -sni example.com -verify microsoft.com https://example.com/ urlName: "example.com" hostName: "example.com" sniName: "example.com" verifyName: "microsoft.com" error: x509: certificate is valid for www.example.org, example.com, example.edu, example.net, example.org, www.example.com, www.example.edu, www.example.net, not microsoft.com }}} domain fronting:: {{{ $ ./names -host maps.google.com -sni www.google.com -verify www.google.com https://www.google.com/ urlName: "www.google.com" hostName: "maps.google.com" sniName: "www.google.com" verifyName: "www.google.com" HTTP/2.0 302 Found }}} domain fronting [ticket:25804 thwarted]:: {{{ $ ./names -host html5-demos.appspot.com -sni www.google.com -verify www.google.com https://www.google.com/ urlName: "www.google.com" hostName: "html5-demos.appspot.com" sniName: "www.google.com" verifyName: "www.google.com" HTTP/2.0 502 Bad Gateway }}} but works if you leave off the SNI:: {{{ $ ./names -host html5-demos.appspot.com -sni "" -verify www.google.com https://www.google.com/ urlName: "www.google.com" hostName: "html5-demos.appspot.com" sniName: "" verifyName: "www.google.com" HTTP/2.0 200 OK }}}
Trac:
Status: new to needs_review-
Will it be easier for a censor to block the SNI-less domain fronting or it's of similar difficulty as the "original" domain fronting implementation?
Depends censorship level.
https://en.wikipedia.org/wiki/Server_Name_Indication#Support
Replying to cypherpunks:
Will it be easier for a censor to block the SNI-less domain fronting or it's of similar difficulty as the "original" domain fronting implementation?
Depends censorship level. https://en.wikipedia.org/wiki/Server_Name_Indication#Support
Ya it depends. Back in June 2014 (ctrl+f for "domainless"), about 16% of observed TLS connections didn't have SNI. I don't know what it is now.
But the TLS fingerprint also matters. If the fingerprint looks exactly like a specific version of Firefox, except that it lacks SNI, that's probably unusual enough to block. It would only happen in normal use when someone browses to an IP address, which is unusual except for rare cases like https://1.1.1.1/. For this reason I'm thinking of adopting the utls library which allows modifying the TLS fingerprint from ordinary Go code. In any case, using the Firefox helper won't be possible when making SNI-less requests, because I'm not aware of any way to control behavior like that from a browser extension.
But another issue is potential blocking by the intermediary services. Maybe a CDN decides they want to always require SNI and they stop dropping SNI-less connections. Cloudflare did this in 2015 on all of their edge servers except for a few special ones, requiring SNI and enforcing a match between SNI and Host header.
Here are the four use cases I want to support. Cases !#1 and !#2 (closed) are already supported; this ticket is about adding !#3 (closed) and !#4 (closed). In the table, I separated
urlNameinto "DNS name" and "connect to".- cdn.ex is at IP address 1.2.3.4
- serves a default certificate for CN=cdn.ex in absence of SNI
- meek.ex is at IP address 5.5.5.5
= DNS query = = connect to = = hostName== sniName== verifyName=!#1 direct meek.ex 5.5.5.5 meek.ex meek.ex meek.ex !#2 (closed) domain fronting cdn.ex 1.2.3.4 meek.ex cdn.ex cdn.ex !#3 (closed) DNS, no SNI cdn.ex 1.2.3.4 meek.ex ^_none_^ cdn.ex !#4 (closed) no DNS, no SNI ^_none_^ 1.2.3.4 meek.ex ^_none_^ cdn.ex meek-client takes its configuration in two ways: on a per-connection basis via PT SOCKS arguments, or globally via command-line options. SOCKS arguments take precedence over command-line options. Here is how cases !#1 and !#2 (closed) are represented:
=== !#1 direct SOCKS args::
url=https://meek.ex/command line::-url https://meek.ex/=== !#2 (closed) domain fronting SOCKS args::url=https://meek.ex/ front=cdn.excommand line::-url https://meek.ex/ -front cdn.exWe have to decide how to represent use cases !#3 (closed) and !#4 (closed).
Observations:
-
hostNameis the name of the final destination, always.- It comes from the
urlargument and I like that design.
- It comes from the
- The only time
sniName≠verifyNameis whensniName=_none_.- That is, there's no need to control
sniNameandverifyNamecompletely independently, only for an option to blank thesniName.
- That is, there's no need to control
- cdn.ex is at IP address 1.2.3.4
mentioned in issue #29293 (moved)
