Opened 9 months ago

Closed 5 months ago

#29077 closed enhancement (implemented)

uTLS for meek-client camouflage

Reported by: dcf Owned by: dcf
Priority: Medium Milestone:
Component: Circumvention/meek Version:
Severity: Normal Keywords: moat utls
Cc: mcs, brade, sf, gk, yawning Actual Points:
Parent ID: Points:
Reviewer: ahf Sponsor:

Description

Use utls (paper) for TLS camouflage instead of a headless Firefox helper. The camouflage will not be as robust as a headless browser, but the complexity will be lower. It will be portable to platforms that don't have a convenient Firefox instance to use. It will fix #12774, which is caused by tor starting multiple instances of meek-client-torbrowser, which in turn start multiple instances of Firefox, which error because they are using the same profile directory.

Child Tickets

Change History (25)

comment:1 Changed 9 months ago by dcf

I started trying this out in

I built a browser with these changes. It works, but not well. It hangs and stalls frequently, and Moat often throws errors. There are two primary causes for these problems.

  1. utls's interaction with Go's built-in HTTPS is awkward. In meek-client we use an http.Transport to enable connection reuse, manage idle timeouts, etc. To hook into utls, I'm setting the DialTLS callback--but setting DialTLS has the side effect of disabling automatic HTTP/2 support on the client. This wouldn't be a problem, except for the fact that with e.g. the HelloFirefox_63 and HelloChrome_70 fingerprint, utls sends a Client Hello with an ALPN that negotiates HTTP/2. The server starts speaking HTTP/2 while the net/http client, oblivious to what utls has done, continues speaking HTTP/1.1, and you get an error. There are more details on this problem on GitHub ticket #16.

As a workaround for the ALPN mismatch issue, I tried specifying the HelloRandomizedNoALPN client. Randomized means it picks fingerprint features randomly, and NoALPN means it won't negotiate HTTP/2. The loss of HTTP/2 is a bummer, but the bigger problem is

  1. The randomized fingerprint is re-randomized for every new TLS connection, and occasionally it generates a Client Hello that either (a) the server rejects or (b) causes the server to select a feature that the client advertised in its fake fingerprint but doesn't actually support. When this happens, you get an error in the meek-client log like
    error in handling request: tls: server selected unsupported group
    
    At the tor level, you get a stall until tor decides to try the bridge again. With Moat, you get an error dialog like "Unable to obtain a bridge from BridgeDB. NS_ERROR_NET_INTERRUPT", unless you are lucky that none of the TLS connections for the duration of the Moat session had the randomization error. The re-randomization of fingerprints normally wouldn't a severe problem because of connection reuse, but the combination of DialTLS and the lack of HTTP/2 means that connections are reused less than normal.

In summary: the static fingerprints fail completely because of an ALPN mismatch; the randomized fingerprints fail sometimes because of server feature mismatches.

There is a partial workaround. Using http2.Transport in place of http.Transport seemingly solves all the above problems--with the caveat that it doesn't work with servers that don't support HTTP/2. Details here. So what I'm planning to do next is add utls and http2 controls to the PT configuration. utls would allow you to choose a utls fingerprint (with knowledge of the errors described above). http2 would let meek-client assume that the front server always supports HTTP/2, solving the errors described above as long as the server actually does support HTTP/2.

comment:2 Changed 9 months ago by dcf

Cc: sf added

comment:3 Changed 9 months ago by gk

Cc: gk added

comment:5 in reply to:  1 Changed 9 months ago by yawning

Replying to dcf:

(b) causes the server to select a feature that the client advertised in its fake fingerprint but doesn't actually support. When this happens, you get an error in the meek-client log like

error in handling request: tls: server selected unsupported group

Probably old news, but since this happens with the Firefox_Auto preset on the Azure bridge, I looked into it a bit, and it is primarily caused by utls's support for curves other than X25519 being non-existent in certain cases.

This is code inherited from crypto/tls, and the issue arises from the assumption that the server will always negotiate X25519 (ajax.aspnetcdn.com will pick secp256r1) if it is present in the ClientHello's Supported Curves extension.

comment:6 Changed 9 months ago by dcf

Yeah I did some tests today with HelloRandomizedNoALPN, 400 trials against each of 3 servers.

ajax.aspnetcdn.com
284no error
80remote error: tls: handshake failure
36tls: server selected unsupported group
golang.org
335no error
65remote error: tls: handshake failure
www.cloudflare.com
400no error

HelloRandomizedNoALPN unconditionally includes sec256r1 and sec384r1, but it doesn't seem to cause a problem unless it also happens to choose TLSv1.3 (which Firefox_60HelloFirefox_63 always does). 100% of the failures used a fingerprint with TLSv1.3, while only 28% of the successes did. I haven't dug deeper to see what the difference between the success and failure TLSv1.3 handshakes is.

Last edited 9 months ago by dcf (previous) (diff)

comment:7 in reply to:  6 ; Changed 9 months ago by watt

Replying to dcf:

TLSv1.3 (which Firefox_60 always does).

Firefox_60 doesn't support TLSv1.3.

comment:8 in reply to:  7 Changed 9 months ago by dcf

Replying to watt:

Replying to dcf:

TLSv1.3 (which Firefox_60 always does).

Firefox_60 doesn't support TLSv1.3.

My bad, I actually meant HelloFirefox_63. You can see where the fingerprint specifies TLSv1.3 here. HelloFirefox_Auto currently means HelloFirefox_63.

comment:9 in reply to:  6 Changed 9 months ago by watt

Replying to dcf:

what the difference between the success and failure TLSv1.3 handshakes is.

draft vs final TLSv1.3?

comment:10 Changed 9 months ago by yawning

100% of the failures used a fingerprint with TLSv1.3, while only 28% of the successes did. I haven't dug deeper to see what the difference between the success and failure TLSv1.3 handshakes is.

The kludges inherited from crypto/tls only apply when TLS 1.3 is requested.

Making the library support the NIST curves is trivial, and would fix this problem.

comment:11 in reply to:  10 Changed 9 months ago by dcf

Replying to yawning:

Making the library support the NIST curves is trivial, and would fix this problem.

Okay; maybe it's better to have that discussion on the utls tracker?

comment:12 Changed 9 months ago by dcf

Status: newneeds_review

Here is a new candidate:

This is a rewrite using the obfs4proxy-inspired technique (comment:4), with a few implementation differences. Instead of dialTLS being attached to the RoundTripper wrapper with a distinguised error code, use a standalone dialUTLS function. Store the state for the dynamically created Transport in a closure rather than in the parent struct. Raise an error if the ALPN changes.

You control which fingerprint to use with a SOCKS arg, like utls=HelloChrome_Auto. With the SOCKS arg, it uses the stdlib net/http as before. Using utls= with --helper is an error.

Currently this breaks proxy support, because previously we were using the built-in proxy support of net/http, and we can't do that anymore with uTLS; we'll have to make our own proxy connections. I'll restore proxy support separately.

I've removed HelloRandomized and HelloGolang from the table of allowed TLS fingerprints. HelloRandomized because it can negotiate different ALPN, and HelloGolang because that's ideally equivalent to omitting the utls= arg. I'm open to having it recognize utls=HelloGolang as an alias for omitting the utls= arg, because compatibility with meek_lite is the most important thing here.

When creating the internal http.Transport, I think I'd like to make it have the same default settings as http.DefaultTransport with respect to timeouts, idle connections, etc. So I'm thinking of cloning the public fields of http.DefaultTransport using the reflection trick from comment:11:ticket:12208. Unfortunately http2.Transport doesn't expose configuration options in the same way. Maybe it doesn't matter much? My main concern here is not having infinite timeouts.

I tested the TLS fingerprint with a few different configurations.

configuration fingerprint seen (all time)
no camouflage c4b0fe116abff001 archive 0.01%
--helper (Tor Browser 8.0.4 / Firefox 60.4.0esr) bb94e801f7aee52b archive 0.58%
utls=HelloChrome_70 bc4c7e42f4961cd7 archive 3.54%
utls=HelloFirefox_63 6bfedc5d5c740d58 archive 1.66%

Browsing and Moat, using meek-azure, seem to be working fine.

Last edited 9 months ago by dcf (previous) (diff)

comment:13 Changed 9 months ago by yawning

Here is a new candidate

Looks ok at a first glance. I guess I should add the timeout values to meek_lite too.

Version 0, edited 9 months ago by yawning (next)

comment:14 Changed 9 months ago by yawning

Cc: yawning added

comment:15 Changed 9 months ago by dcf

Here's a merge candidate for the utls_2 branch. Changes since comment:12:

  • Support for socks5, http, and https proxies. (The same proxy types net/http supports.)
  • utls=none and utls=HelloGolang mean "no uTLS".
  • Copying default http.Transport timeouts and other settings.
  • Rudimentary integration tests for the http and https proxies.

Replying to yawning:

Note that I treat not specifying the argument as "use a sensible default", and have none special cased as "use the built in TLS stack", under the rationale that once the code is solid the only reason to not use utls is if something breaks, and "Add utls=none" is a easier interim workaround to convey to the unwashed masses than "Edit an existing argument".

This sounds like a good design. I think for at least the next release I'll leave no utls meaning native TLS, and then a future release can compatibly upgrade it to mean some sensible uTLS configuration by default.

comment:16 Changed 9 months ago by yawning

This sounds like a good design. I think for at least the next release I'll leave no utls meaning native TLS, and then a future release can compatibly upgrade it to mean some sensible uTLS configuration by default.

Ok, I'll let you know if behavior there changes, though I don't anticipate much beyond potentially changing which fingerprint meek_lite will use as a "sensible default".

comment:17 Changed 9 months ago by yawning

Making the library support the NIST curves is trivial, and would fix this problem.

arnhem :: Documents/Development/utls-test % ./fuck --utls HelloFirefox_Auto https://ajax.aspnetcdn.com
utls: &{Firefox 63}
 0 HTTP/2.0 301 Moved Permanently
 1 HTTP/2.0 301 Moved Permanently
 2 HTTP/2.0 301 Moved Permanently
 3 HTTP/2.0 301 Moved Permanently

Not sure if I want to make the fork public yet. The test suite doesn't pass with my changes applied, and while I think I know why, I haven't dug into it enough to prove it to my satisfaction.

Edit: Paste the correct example output.

Last edited 9 months ago by yawning (previous) (diff)

comment:18 Changed 9 months ago by yawning

I'm convinced (after wasting an afternoon trying to get it to work) that HelloChrome_Auto is an acceptable default assuming the front is not on Google infrastructure. At some point in the future when more servers use BoringSSL with multiple draft and or non-standard extensions, that stance will need to be revisited.

comment:19 Changed 9 months ago by yawning

I have changed meek_lite to use Firefox_Auto and a forked version of utls that adds a number of compatibility fixes. Additionally I believe that the Firefox fingerprint is unchanged in 65. There was one ciphersuite difference between FF 65 as packaged and ran on Fedora and that produced by utls, but that is due to Fedora disabling 3DES.

There's one more feature I want to add to meek_lite before tagging, but unless anything comes up, this is close to what I'm currently leaning towards tagging as 0.0.9.

comment:20 Changed 8 months ago by dcf

Here's another merge candidate, rebased and cleaned up. It's the same utls_2 from comment:12, except for squashed commits, more tests, and fixes for two small bugs I found through testing: when the utls.Config doesn't have a ServerName set, take the SNI from the dial address; and if someone happens to make an http request using UTLSRoundTripper, honor the proxy setting on the UTLSRoundTripper.

comment:21 Changed 8 months ago by ahf

Reviewer: ahf

comment:22 Changed 8 months ago by cohosh

Hey a did a quick look, here are some comments:

  • meek-client now requires go version 1.11 or later because of utls. I looked but didn't find in the documentation or READMEs the required go version for building the client. It's possible I missed it, just wanted to point out the required version update.

Otherwise, I only have some minor copy edit notes for one of the comments:

in meek-client/utls.go:

+// otherwise, create an internal http.Transport. In either case, set DialTLS on
+// the created Transport do a function that dials using uTLS As a special case,

do-->to
uTLS-->uTLS.

comment:23 in reply to:  22 Changed 8 months ago by dcf

Replying to cohosh:

Hey a did a quick look, here are some comments:

  • meek-client now requires go version 1.11 or later because of utls. I looked but didn't find in the documentation or READMEs the required go version for building the client. It's possible I missed it, just wanted to point out the required version update.

Thank you for reviewing.

Yes, it's a bit of a bummer to require such a recent go... I think it's because uTLS is a copy of the stdlib crypto/tls with changes applied on top, and the stdlib is not shy about requiring a recent compiler. Unfortunately uTLS will put us on more of an update and maintenance treadmill. Not only will we have to keep an eye on uTLS to pick up new fingerprints, but uTLS will have to stay on top of crypto/tls updates (like this). There's a fair amount of uncertainty yet, I feel.

+// otherwise, create an internal http.Transport. In either case, set DialTLS on
+// the created Transport do a function that dials using uTLS As a special case,

do-->to
uTLS-->uTLS.

Thanks, fixed these in dd26d797b15b8a05262209e07846f7204420e81a.

comment:24 Changed 8 months ago by ahf

Status: needs_reviewmerge_ready

The code also looks good to me. The uTLS library is neat and I love the idea of profiles. Didn't look at the wire part of the uTLS library, but I assume others have looked at that?

Let's get the code in.

comment:25 Changed 5 months ago by dcf

Resolution: implemented
Status: merge_readyclosed

Merged in 0.33.

Note: See TracTickets for help on using tickets.