uTLS for meek-client camouflage

added component::circumvention/meek moat owner::dcf priority::medium resolution::implemented reviewer::ahf severity::normal status::closed type::enhancement utls labels

I started trying this out in

meek branch utls
tor-browser-build branch meek-client-utls

I built a browser with these changes. It works, but not well. It hangs and stalls frequently, and Moat often throws errors. There are two primary causes for these problems.

utls's interaction with Go's built-in HTTPS is awkward. In meek-client we use an http.Transport to enable connection reuse, manage idle timeouts, etc. To hook into utls, I'm setting the DialTLS callback--but setting DialTLS has the side effect of disabling automatic HTTP/2 support on the client. This wouldn't be a problem, except for the fact that with e.g. the HelloFirefox_63 and HelloChrome_70 fingerprint, utls sends a Client Hello with an ALPN that negotiates HTTP/2. The server starts speaking HTTP/2 while the net/http client, oblivious to what utls has done, continues speaking HTTP/1.1, and you get an error. There are more details on this problem on GitHub ticket #16.

As a workaround for the ALPN mismatch issue, I tried specifying the HelloRandomizedNoALPN client. Randomized means it picks fingerprint features randomly, and NoALPN means it won't negotiate HTTP/2. The loss of HTTP/2 is a bummer, but the bigger problem is

The randomized fingerprint is re-randomized for every new TLS connection, and occasionally it generates a Client Hello that either (a) the server rejects or (b) causes the server to select a feature that the client advertised in its fake fingerprint but doesn't actually support. When this happens, you get an error in the meek-client log like {{{ error in handling request: tls: server selected unsupported group }}} At the tor level, you get a stall until tor decides to try the bridge again. With Moat, you get an error dialog like "Unable to obtain a bridge from BridgeDB. NS_ERROR_NET_INTERRUPT", unless you are lucky that none of the TLS connections for the duration of the Moat session had the randomization error. The re-randomization of fingerprints normally wouldn't a severe problem because of connection reuse, but the combination of DialTLS and the lack of HTTP/2 means that connections are reused less than normal.

In summary: the static fingerprints fail completely because of an ALPN mismatch; the randomized fingerprints fail sometimes because of server feature mismatches.

There is a partial workaround. Using http2.Transport in place of http.Transport seemingly solves all the above problems--with the caveat that it doesn't work with servers that don't support HTTP/2. Details here. So what I'm planning to do next is add utls and http2 controls to the PT configuration. utls would allow you to choose a utls fingerprint (with knowledge of the errors described above). http2 would let meek-client assume that the front server always supports HTTP/2, solving the errors described above as long as the server actually does support HTTP/2.

Trac:
Cc: mcs, brade to mcs, brade, sf

Trac:
Cc: mcs, brade, sf to mcs, brade, sf, gk

https://gitlab.com/yawning/obfs4/commit/4d453dab2120082b00bf6e63ab4aaeeda6b8d8a3

Replying to dcf:

(b) causes the server to select a feature that the client advertised in its fake fingerprint but doesn't actually support. When this happens, you get an error in the meek-client log like {{{ error in handling request: tls: server selected unsupported group }}}

Probably old news, but since this happens with the Firefox_Auto preset on the Azure bridge, I looked into it a bit, and it is primarily caused by utls's support for curves other than X25519 being non-existent in certain cases.

This is code inherited from crypto/tls, and the issue arises from the assumption that the server will always negotiate X25519 (ajax.aspnetcdn.com will pick secp256r1) if it is present in the ClientHello's Supported Curves extension.

Yeah I did some tests today with HelloRandomizedNoALPN, 400 trials against each of 3 servers. ||=ajax.aspnetcdn.com =| ||-------------------------| | 284|no error | | 80|remote error: tls: handshake failure | | 36|tls: server selected unsupported group | ||=golang.org =| | 335|no error | | 65|remote error: tls: handshake failure | ||=www.cloudflare.com =| | 400|no error |

HelloRandomizedNoALPN unconditionally includes sec256r1 and sec384r1, but it doesn't seem to cause a problem unless it also happens to choose TLSv1.3 (which ~~Firefox_60~~HelloFirefox_63 always does). 100% of the failures used a fingerprint with TLSv1.3, while only 28% of the successes did. I haven't dug deeper to see what the difference between the success and failure TLSv1.3 handshakes is.

Replying to dcf:

TLSv1.3 (which Firefox_60 always does). Firefox_60 doesn't support TLSv1.3.

Trac:
Username: watt

Replying to watt:

Replying to dcf:

TLSv1.3 (which Firefox_60 always does). Firefox_60 doesn't support TLSv1.3.

My bad, I actually meant HelloFirefox_63. You can see where the fingerprint specifies TLSv1.3 here. HelloFirefox_Auto currently means HelloFirefox_63.

Replying to dcf:

what the difference between the success and failure TLSv1.3 handshakes is. draft vs final TLSv1.3?

Trac:
Username: watt

100% of the failures used a fingerprint with TLSv1.3, while only 28% of the successes did. I haven't dug deeper to see what the difference between the success and failure TLSv1.3 handshakes is.

The kludges inherited from crypto/tls only apply when TLS 1.3 is requested.

Making the library support the NIST curves is trivial, and would fix this problem.

Replying to yawning:

Making the library support the NIST curves is trivial, and would fix this problem.

Okay; maybe it's better to have that discussion on the utls tracker?

Here is a new candidate:

meek branch utls_2
tor-browser-build branch meek-client-utls

This is a rewrite using the obfs4proxy-inspired technique (comment:4), with a few implementation differences. Instead of dialTLS being attached to the RoundTripper wrapper with a distinguised error code, use a standalone dialUTLS function. Store the state for the dynamically created Transport in a closure rather than in the parent struct. Raise an error if the ALPN changes.

You control which fingerprint to use with a SOCKS arg, like utls=HelloChrome_Auto. With the SOCKS arg, it uses the stdlib net/http as before. Using utls= with --helper is an error.

Currently this breaks proxy support, because previously we were using the built-in proxy support of net/http, and we can't do that anymore with uTLS; we'll have to make our own proxy connections. I'll restore proxy support separately.

I've removed HelloRandomized and HelloGolang from the table of allowed TLS fingerprints. HelloRandomized because it can negotiate different ALPN, and HelloGolang because that's ideally equivalent to omitting the utls= arg. I'm open to having it recognize utls=HelloGolang as an alias for omitting the utls= arg, because compatibility with meek_lite is the most important thing here.

When creating the internal http.Transport, I think I'd like to make it have the same default settings as http.DefaultTransport with respect to timeouts, idle connections, etc. So I'm thinking of cloning the public fields of http.DefaultTransport using the reflection trick from comment:11:ticket:12208. Unfortunately http2.Transport doesn't expose configuration options in the same way. Maybe it doesn't matter much? My main concern here is not having infinite timeouts.

I tested the TLS fingerprint with a few different configurations.

=configuration =	=fingerprint =	= seen (all time)=
no camouflage	c4b0fe116abff001 archive	0.01%
`--helper` (Tor Browser 8.0.4 / Firefox 60.4.0esr)	bb94e801f7aee52b archive	0.58%
`utls=HelloChrome_70`	bc4c7e42f4961cd7 archive	3.54%
`utls=HelloFirefox_63`	6bfedc5d5c740d58 archive	1.66%

Browsing and Moat, using meek-azure, seem to be working fine.

Trac:
Status: new to needs_review

Here is a new candidate

Looks ok at a first glance. I guess I should add the timeout values to meek_lite too.

Edit: Done, though I just copy the values out of http.DefaulTransport by hand instead of doing anything with reflection.

As far as interoperability/compatibility goes, I'm open to changing meek_lite as well.

Note that I treat not specifying the argument as "use a sensible default", and have none special cased as "use the built in TLS stack", under the rationale that once the code is solid the only reason to not use utls is if something breaks, and "Add utls=none" is a easier interim workaround to convey to the unwashed masses than "Edit an existing argument".

Let me know if you would rather I change this to match meek's behavior.

There's a number of other things that I'm tempted to do with the code (assuming I have the spare time) that would increase compatibility and blocking resistance, but it would require forking utls, so no promises.

Trac:
Cc: mcs, brade, sf, gk to mcs, brade, sf, gk, yawning

Here's a merge candidate for the utls_2 branch. Changes since comment:12:

Support for socks5, http, and https proxies. (The same proxy types net/http supports.)
utls=none and utls=HelloGolang mean "no uTLS".
Copying default http.Transport timeouts and other settings.
Rudimentary integration tests for the http and https proxies.

Replying to yawning:

Note that I treat not specifying the argument as "use a sensible default", and have none special cased as "use the built in TLS stack", under the rationale that once the code is solid the only reason to not use utls is if something breaks, and "Add utls=none" is a easier interim workaround to convey to the unwashed masses than "Edit an existing argument".

This sounds like a good design. I think for at least the next release I'll leave no utls meaning native TLS, and then a future release can compatibly upgrade it to mean some sensible uTLS configuration by default.

This sounds like a good design. I think for at least the next release I'll leave no utls meaning native TLS, and then a future release can compatibly upgrade it to mean some sensible uTLS configuration by default.

Ok, I'll let you know if behavior there changes, though I don't anticipate much beyond potentially changing which fingerprint meek_lite will use as a "sensible default".

Making the library support the NIST curves is trivial, and would fix this problem.

arnhem :: Documents/Development/utls-test % ./fuck --utls HelloFirefox_Auto https://ajax.aspnetcdn.com
utls: &{Firefox 63}
 0 HTTP/2.0 301 Moved Permanently
 1 HTTP/2.0 301 Moved Permanently
 2 HTTP/2.0 301 Moved Permanently
 3 HTTP/2.0 301 Moved Permanently

Not sure if I want to make the fork public yet. The test suite doesn't pass with my changes applied, and while I think I know why, I haven't dug into it enough to prove it to my satisfaction.

Edit: Paste the correct example output.

I'm convinced (after wasting an afternoon trying to get it to work) that HelloChrome_Auto is an acceptable default assuming the front is not on Google infrastructure. At some point in the future when more servers use BoringSSL with multiple draft and or non-standard extensions, that stance will need to be revisited.

I have changed meek_lite to use Firefox_Auto and a forked version of utls that adds a number of compatibility fixes. Additionally I believe that the Firefox fingerprint is unchanged in 65. There was one ciphersuite difference between FF 65 as packaged and ran on Fedora and that produced by utls, but that is due to Fedora disabling 3DES.

There's one more feature I want to add to meek_lite before tagging, but unless anything comes up, this is close to what I'm currently leaning towards tagging as 0.0.9.

https://gitweb.torproject.org/pluggable-transports/meek.git/log/?h=utls_3&id=21abea57d28abe614887135ab380bfb4355de9fc

Here's another merge candidate, rebased and cleaned up. It's the same utls_2 from comment:12, except for squashed commits, more tests, and fixes for two small bugs I found through testing: when the utls.Config doesn't have a ServerName set, take the SNI from the dial address; and if someone happens to make an http request using UTLSRoundTripper, honor the proxy setting on the UTLSRoundTripper.

Trac:
Reviewer: N/A to ahf

Hey a did a quick look, here are some comments:

meek-client now requires go version 1.11 or later because of utls. I looked but didn't find in the documentation or READMEs the required go version for building the client. It's possible I missed it, just wanted to point out the required version update.

Otherwise, I only have some minor copy edit notes for one of the comments:

in meek-client/utls.go:

+// otherwise, create an internal http.Transport. In either case, set DialTLS on
+// the created Transport do a function that dials using uTLS As a special case,

do-->to uTLS-->uTLS.

Replying to cohosh:

Hey a did a quick look, here are some comments:

meek-client now requires go version 1.11 or later because of utls. I looked but didn't find in the documentation or READMEs the required go version for building the client. It's possible I missed it, just wanted to point out the required version update.

Thank you for reviewing.

Yes, it's a bit of a bummer to require such a recent go... I think it's because uTLS is a copy of the stdlib crypto/tls with changes applied on top, and the stdlib is not shy about requiring a recent compiler. Unfortunately uTLS will put us on more of an update and maintenance treadmill. Not only will we have to keep an eye on uTLS to pick up new fingerprints, but uTLS will have to stay on top of crypto/tls updates (like this). There's a fair amount of uncertainty yet, I feel.

{{{ +// otherwise, create an internal http.Transport. In either case, set DialTLS on +// the created Transport do a function that dials using uTLS As a special case, }}} do-->to uTLS-->uTLS.

Thanks, fixed these in dd26d797b15b8a05262209e07846f7204420e81a.

The code also looks good to me. The uTLS library is neat and I love the idea of profiles. Didn't look at the wire part of the uTLS library, but I assume others have looked at that?

Let's get the code in.

Trac:
Status: needs_review to merge_ready

Merged in 0.33.

Trac:
Status: merge_ready to closed
Resolution: N/A to implemented

Replying to dcf:

When creating the internal http.Transport, I think I'd like to make it have the same default settings as http.DefaultTransport with respect to timeouts, idle connections, etc. So I'm thinking of cloning the public fields of http.DefaultTransport using the reflection trick from comment:11:ticket:12208.

Since go1.13 there is a Transport.Clone function that can replace the use of reflection to copy a Transport. See https://github.com/golang/go/issues/26013. I made meek use the new function here.

closed

mentioned in issue #29143 (moved)

mentioned in issue #29205 (moved)

mentioned in issue #29430 (moved)

moved to tpo/anti-censorship/pluggable-transports/meek#29077 (closed)

mentioned in issue tpo/applications/tor-browser#29430 (closed)

uTLS for meek-client camouflage

Child items 0

Activity