Opened 4 years ago

Last modified 7 weeks ago

#12208 needs_review enhancement

Make it possible to use an IP address as a front (no DNS request and no SNI)

Reported by: dcf Owned by: dcf
Priority: Medium Milestone:
Component: Obfuscation/meek Version:
Severity: Normal Keywords:
Cc: brade, mcs, arlolra Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

meek puts one domain name on the "outside" of your connection (the DNS request and SNI), and a different name on the "inside" (the HTTP Host header). It would be good for some uses if the outside could be just to an IP address rather than a domain name, so that there were no DNS request, and no server_name extension in the CLientHello. Kind of like if you were to browse to https://38.229.72.16/ instead of https://www.torproject.org/.

The motivating use case is using a CDN as a front instead of www.google.com. A CDN has many domains behind it, but if we choose just one of them as the front, that domain might get blocked (because the collateral damage would be limited to just one domain). Such blocking would break the transport and also incidentally get the innocent third-party domain, who has nothing to do with any of this, censored even for non-circumventors. What we want is to use one of the CDN's frontend IP addresses as a front, so that the censor has to block the whole IP and the thousands of domains behind it, not just a single domain.

Child Tickets

Attachments (1)

names.go (7.9 KB) - added by dcf 8 weeks ago.
Demo allowing you to set the URL, Host, SNI, and verify names independently.

Download all attachments as: .zip

Change History (18)

comment:1 Changed 4 years ago by dcf

I didn't find an obvious way to reach this goal.

In the naked golang HTTPS code (without the browser extension), the first thing I tried was putting an IP address in place of www.google.com for the --front option.

-ClientTransportPlugin meek exec ./meek-client --url=https://meek-reflect.appspot.com/ --front=www.google.com
+ClientTransportPlugin meek exec ./meek-client --url=https://meek-reflect.appspot.com/ --front=74.125.224.116

It works fine up until certificate verification, then exits with this error:

error in handling request: x509: cannot validate certificate for 74.125.224.116 because it doesn't contain any IP SANs

What we would want is to connect to an IP without a domain name, but still let the TLS code know that it should use "www.google.com" for verification purposes. It doesn't seem possible to do it with the crypto/tls library. The clientHandshake function builds a ClientHello using the value c.config.ServerName, and then later does verification using the same value c.config.ServerName. In other words, it doesn't look possible to use one value for the SNI and another during verification.

Everything works fine if you set InsecureSkipVerify on the TLSClientConfig of the Transport, but then of course you're not doing any verification.

The golang HTTPS library, when you connect to an IP address, sends a server_name extension whose value is the IP address as a string. When I tried the same thing in Firefox 29 just now, it simply leaves off the server_name extension.

How GoAgent does it is it cooks up its own TLS connection using Python's ssl library, Python's ssl library doesn't do any verification, and then GoAgent itself does some nonstandard verification, not checking any certs but either looking for an organizationName starting with "Google " or looking for ".google" or ".appspot.com" in the commonName. It's essentially the same as InsecureSkipVerify.

I don't know yet if there's a way to do what we want in the Firefox and Chrome helpers.

Last edited 4 years ago by dcf (previous) (diff)

comment:2 Changed 4 years ago by dcf

A way to do this in Firefox may be nsICertOverrideService, specifically hasMatchingOverride. nsiCertOverrideService is the service that manages the overrides that get added when you click through a certificate warning (which are visible through Preferences→Advanced→Certificates→View Certificates, Lifetime=Temporary).

It looks like we can define hasMatchingOverride and request HTTPS to a CloudFront IP (for example). hasMatchingOverride will get called back. We check that the error is only ERROR_MISMATCH (and not ERROR_UNTRUSTED or ERROR_TIME). We check that the commonName in the cert is "*.cloudfront.net", and return true. In effect, we defer to the normal verification mechanisms for everything but hostname verification, and for that we check that the wildcard is what we expect.

skipCertError is a third-party Firefox extension to automatically ignore certificate errors.

Selenium's docs on hasMatchingOverride.

comment:3 Changed 4 years ago by dcf

A case where this ticket would be useful: In China, it apparently works to route through CloudFront, except that *.cloudfront.net domain names are DNS-poisoned. If you edit /etc/hosts, or use an IP in place of a domain name, then it works.

Last edited 4 years ago by dcf (previous) (diff)

comment:4 Changed 4 years ago by dcf

Psiphon and Lantern both independently discovered this issue. They found that connecting with no SNI is necessary when using the Fastly CDN. They both worked around it by patching the Go crypto/tls library.

Psiphon:

Lantern:

comment:5 Changed 4 years ago by dcf

I contacted the ICSI Certificate Notary (notary.icsi.berkeley.edu), which monitors SSL traffic at some big Internet sites, asking what fraction of their monitored connections use SNI. The answer:

In the last month:

  • 83.5% of connections had SNI.
  • 80.1% of connections where certificates were exchanged (excluding e.g. resumed sessions) had SNI.

comment:6 Changed 4 years ago by dcf

I wrote a post to the golang-nuts mailing list asking about this use case:

Brad Fitzpatrick replied to say that what we want should be possible once this in-progress issue is resolved:

comment:7 Changed 4 years ago by dcf

Following discussion in the thread at https://groups.google.com/forum/#!topic/golang-nuts/bt9vBbZLYgU, Ox found a workaround that works with an unmodified crypto/tls library. It uses manual dialing and verification.

comment:9 Changed 21 months ago by cypherpunks

Severity: Normal

When I tried the same thing in Firefox 29 just now, it simply leaves off the server_name extension.

Amazon CDN doesn't support for such requests now. Seems like it terminates TLS session if ClientHello lack of SNI?

comment:10 in reply to:  9 Changed 21 months ago by dcf

Replying to cypherpunks:

When I tried the same thing in Firefox 29 just now, it simply leaves off the server_name extension.

Amazon CDN doesn't support for such requests now. Seems like it terminates TLS session if ClientHello lack of SNI?

Thanks for checking this. Could you add a note to doc/meek#AmazonCloudFront?

Changed 8 weeks ago by dcf

Attachment: names.go added

Demo allowing you to set the URL, Host, SNI, and verify names independently.

comment:11 Changed 8 weeks ago by dcf

Status: newneeds_review

The past couple of days I've been experimenting with ways to set or omit the SNI. There's a demo program in attachment:names.go and I'd appreciate any review or questions about design decisions. I took inspiration from Ox's domainfront.go demo from 2014, but changes in the standard library (especially since Go 1.8) provide better ways to do some things now. If this looks good, I'm going to use it as the basis of new code in meek-client.

The basic situation is that we are managing four domain names:

  • urlName: this is the name we resolve and actually establish a TCP connection with.
  • hostName: this is the name that goes in the HTTP Host header.
  • sniName: this is the name that goes in the SNI extension.
  • verifyName: this is the name that the client verifies the server certificate against.

In normal everyday HTTPS, all four of these names are the same. Domain fronting allows hostName to differ, but the other three names are the same. attachment:names.go shows how to make all four names independent, so you can, for example, send no SNI but still verify the server certificate against a hostname, while still fronting a different domain in the Host header.

An explanation of some decisions:

  1. The way to set the SNI is to modify TLSClientConfig.ServerName; the way to set the verification name is to modify the TLSClientConfig.VerifyPeerCertificate callback. Both of these are properties of http.Transport, which is a long-lived data structure that manages multiple HTTP roundtrips. We therefore need a separate http.Transport for each unique (sniName, verifyName) pair. The only exception is when urlName = sniName = verifyName: that's the built-in behavior of the net/http package, so we can share an http.Transport in that case. The function getHTTPTransportForNames creates and caches new http.Transports as needed. (And note that hostName does not enter this logic at all: whether or not you are domain fronting is orthogonal to certificate verification.)
  2. The way to omit the SNI extension is to set TLSClientConfig.ServerName to an IP address. The IP address is not actually used for anything else, as long as TLSClientConfig.InsecureSkipVerify is set. The trick of setting TLSClientConfig.TLSDial, and then pulling the certificates from the tls.Conn using ConnectionState, doesn't work when a proxy is set, because the proxy bypasses the dialer (which is the reason why VerifyPeerCertificate was introduced).
  3. We need to use multiple simultaneous http.Transports, so we can't just modify the global http.DefaultTransport like we do currently. But http.DefaultTransport has some nice default settings for things like timeouts. Go doesn't provide any method for creating a new http.Transport that has the default settings, and you can't just copy the struct literal because it contains mutexes. So I used a reflection trick I found on a mailing list to copy just the public struct members. Also if we naively set TLSClientConfig, it disables HTTP/2 support; because we plan to modify TLSClientConfig, we have to first call http2.ConfigureTransport.

Here are examples use cases.

all names equal (ordinary HTTPS)

$ ./names -host example.com -sni example.com -verify example.com https://example.com/
urlName:    "example.com"
hostName:   "example.com"
sniName:    "example.com"
verifyName: "example.com"
HTTP/2.0 200 OK
omit SNI, but still verify
$ ./names -host example.com -sni "" -verify example.com https://example.com/
urlName:    "example.com"
hostName:   "example.com"
sniName:    ""
verifyName: "example.com"
HTTP/2.0 200 OK
omit DNS request and SNI, but still verify
$ dig +short example.com
93.184.216.34
$ ./names -host example.com -sni "" -verify example.com https://93.184.216.34/
urlName:    "93.184.216.34"
hostName:   "example.com"
sniName:    ""
verifyName: "example.com"
HTTP/2.0 200 OK
ask for one name in the SNI but verify against another (still valid) name
$ ./names -host example.com -sni example.com -verify www.example.net https://example.com/ 
urlName:    "example.com"
hostName:   "example.com"
sniName:    "example.com"
verifyName: "www.example.net"
HTTP/2.0 200 OK
try verifying against a name that's not valid for the certificate
$ ./names -host example.com -sni example.com -verify microsoft.com https://example.com/
urlName:    "example.com"
hostName:   "example.com"
sniName:    "example.com"
verifyName: "microsoft.com"
error: x509: certificate is valid for www.example.org, example.com, example.edu, example.net, example.org, www.example.com, www.example.edu, www.example.net, not microsoft.com
domain fronting
$ ./names -host maps.google.com -sni www.google.com -verify www.google.com https://www.google.com/
urlName:    "www.google.com"
hostName:   "maps.google.com"
sniName:    "www.google.com"
verifyName: "www.google.com"
HTTP/2.0 302 Found
domain fronting thwarted
$ ./names -host html5-demos.appspot.com -sni www.google.com -verify www.google.com https://www.google.com/
urlName:    "www.google.com"
hostName:   "html5-demos.appspot.com"
sniName:    "www.google.com"
verifyName: "www.google.com"
HTTP/2.0 502 Bad Gateway
but works if you leave off the SNI
$ ./names -host html5-demos.appspot.com -sni "" -verify www.google.com https://www.google.com/
urlName:    "www.google.com"
hostName:   "html5-demos.appspot.com"
sniName:    ""
verifyName: "www.google.com"
HTTP/2.0 200 OK

comment:12 Changed 8 weeks ago by cypherpunks

Will it be easier for a censor to block the SNI-less domain fronting or it's of similar difficulty as the "original" domain fronting implementation?

comment:13 Changed 8 weeks ago by cypherpunks

Will it be easier for a censor to block the SNI-less domain fronting or it's of similar difficulty as the "original" domain fronting implementation?

Depends censorship level.

https://en.wikipedia.org/wiki/Server_Name_Indication#Support

comment:14 in reply to:  13 Changed 8 weeks ago by dcf

Replying to cypherpunks:

Will it be easier for a censor to block the SNI-less domain fronting or it's of similar difficulty as the "original" domain fronting implementation?

Depends censorship level.
https://en.wikipedia.org/wiki/Server_Name_Indication#Support

Ya it depends. Back in June 2014 (ctrl+f for "domainless"), about 16% of observed TLS connections didn't have SNI. I don't know what it is now.

But the TLS fingerprint also matters. If the fingerprint looks exactly like a specific version of Firefox, except that it lacks SNI, that's probably unusual enough to block. It would only happen in normal use when someone browses to an IP address, which is unusual except for rare cases like https://1.1.1.1/. For this reason I'm thinking of adopting the utls library which allows modifying the TLS fingerprint from ordinary Go code. In any case, using the Firefox helper won't be possible when making SNI-less requests, because I'm not aware of any way to control behavior like that from a browser extension.

But another issue is potential blocking by the intermediary services. Maybe a CDN decides they want to always require SNI and they stop dropping SNI-less connections. Cloudflare did this in 2015 on all of their edge servers except for a few special ones, requiring SNI and enforcing a match between SNI and Host header.

comment:15 Changed 8 weeks ago by mcs

Cc: brade mcs added

comment:16 Changed 8 weeks ago by arlolra

Cc: arlolra added

comment:17 Changed 7 weeks ago by dcf

Here are the four use cases I want to support. Cases #1 and #2 are already supported; this ticket is about adding #3 and #4. In the table, I separated urlName into "DNS name" and "connect to".

  • cdn.ex is at IP address 1.2.3.4
    • serves a default certificate for CN=cdn.ex in absence of SNI
  • meek.ex is at IP address 5.5.5.5
DNS query connect to hostName sniName verifyName
#1 direct meek.ex 5.5.5.5 meek.ex meek.ex meek.ex
#2 domain fronting cdn.ex 1.2.3.4 meek.ex cdn.ex cdn.ex
#3 DNS, no SNI cdn.ex 1.2.3.4 meek.ex none cdn.ex
#4 no DNS, no SNI none 1.2.3.4 meek.ex none cdn.ex

meek-client takes its configuration in two ways: on a per-connection basis via PT SOCKS arguments, or globally via command-line options. SOCKS arguments take precedence over command-line options. Here is how cases #1 and #2 are represented:

#1 direct

SOCKS args
url=https://meek.ex/
command line
-url https://meek.ex/

#2 domain fronting

SOCKS args
url=https://meek.ex/ front=cdn.ex
command line
-url https://meek.ex/ -front cdn.ex

We have to decide how to represent use cases #3 and #4.

Observations:

  • hostName is the name of the final destination, always.
    • It comes from the url argument and I like that design.
  • The only time sniNameverifyName is when sniName=none.
    • That is, there's no need to control sniName and verifyName completely independently, only for an option to blank the sniName.
Note: See TracTickets for help on using tickets.