wiki:doc/meek

Version 61 (modified by dcf, 4 years ago) (diff)

Notes on GTor.

meek is a transport that uses HTTP for carrying bytes and TLS for obfuscation. Traffic is relayed through a third-party server (Google App Engine). It uses a trick to talk to the third party so that it looks like it is talking to an unblocked server.

Quick start

Download a browser bundle for your platform:

Extract and run it, and then configure these settings:

  1. Configure on the first screen.
  2. No to Does this computer need to use a proxy to access the Internet?
  3. No to Does this computer's Internet connection go through a firewall that only allows connections to certain ports?
  4. Yes to Does your Internet Service Provider (ISP) block or otherwise censor connections to the Tor Network?
  5. Connect with provided bridges and select meek from the dropdown box, then click Connect.

7.0.5 settings screen 1 7.0.5 settings screen 2 7.0.5 settings screen 3 7.0.5 settings screen 4 4.0-alpha-1 settings screen 4.

To build from source:

git clone https://git.torproject.org/pluggable-transports/meek.git
cd meek/meek-client
export GOPATH=~/go
go get
go build
tor -f torrc

FreeBSD port: http://www.freshports.org/security/meek/

Overview

meek sends traffic through a custom web app on Google App Engine. meek works in places where Google search (www.google.com) is unblocked, even though App Engine itself (appspot.com) may be blocked. The meek-client program builds a special HTTPS request and sends it to the Google frontend server--the server that dispatches requests to different Google services, like search, Google Drive, and App Engine. What's special about the HTTPS request is that from the outside it appears to be destined for www.google.com, so it is allowed by the censor. But under the encryption layer, the Host header is actually for meek-reflect.appspot.com, so the frontend server knows to forward the request to App Engine and the web app that we run. The web app in turn forwards the traffic to a Tor bridge, where the meek-server program decodes it and feeds it to Tor.

Redrawn diagram, no longer Google-specific.

See A Child's Garden of Pluggable Transports for details of how the protocol looks at the byte level, both at the TLS layer (the part visible to a censor), and at the HTTP layer (the invisible layer that carries the data).

General notes about App Engine

Quotas for unpaid apps:

You can pay to get higher quotas:

There is also a higher "premier" level of service that comes with support:

Paying for more bandwidth on a public proxy server would be nice in that users wouldn't have to set up their own instance, and we could hardcode the server inside a browser bundle. (It would be nice to know the cost per gigabyte of running App Engine and, say, an ordinary Tor relay.)

An idea to reduce overhead and eliminate polling is to use HTTP as a long-lived bidirectional channel, sending upstream data in a POST body and receiving data in the response body simultaneously. (That is, you send a POST with no Content-Length, the server reads your header and forwards the request to the relay, the server writes back a header, and after that you use the connection as an ordinary socket, with upstream and downstream data interleaved.) An implementation of this idea is at https://www.bamsoftware.com/git/meeker.git. The idea doesn't work with App Engine, for two reasons. 1) requests must be handled within 60 seconds, and 2) App Engine doesn't support streaming requests of this kind:

"App Engine calls the handler with a Request and a ResponseWriter, then waits for the handler to write to the ResponseWriter and return. When the handler returns, the data in the ResponseWriter's internal buffer is sent to the user. This is practically the same as when writing normal Go programs that use the http package. The one notable difference is that App Engine does not support streaming data in response to a single request."

App Engine doesn't even call your web app code until it has consumed the entire request body, and doesn't start flushing the response body until you close the output stream.

Using nginx instead of Google App Engine/PHP/etc

Any web server that serves a site over https using nginx can act as a reflector just by editing the config.

Requirements:

  • A web server running a sufficiently recent version of nginx serving a site over SSL/TLS.
  • A valid certificate signed by a "real" CA. Currently meek-client does not have a way to accept self-signed certificates or easily add CA certs to the trusted list.

Config:

   # The configuration for the normal HTTPS server
   server {
           # Set default_server so that real content gets served by default
   #       listen 443 default_server;
           listen [::]:443 default_server;
           server_name www.example.com;

           # Boilerplate for SSL, adjust as appropriate.
           ssl on;
           ssl_certificate /etc/ssl/nginx/example.com.crt;
           ssl_certificate_key /etc/ssl/nginx/example.com.key;
           ssl_session_timeout 5m;
           ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
           ssl_ciphers !RC4:HIGH:!MD5:!aNULL:!DH;
           ssl_prefer_server_ciphers on;

           # Blah blah blah, normal site specific config
           root /usr/share/nginx/www;
           index index.html index.htm;
           location / {
                   index index.html;
           }
   }

   # Configuration for the reflector
   server {
   #       listen 443;
           listen [::]:443;

           # This does not need to be a real name, but it MUST be distinct from
           # the host used for real content.
           server_name meek-reflect.example.com;

           # Boilerplate for SSL/TLS (copy/paste from the other server block)
           ssl on;
           ssl_certificate /etc/ssl/nginx/example.com.crt;
           ssl_certificate_key /etc/ssl/nginx/example.com.key;
           ssl_session_timeout 5m;
           ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
           ssl_ciphers !RC4:HIGH:!MD5:!aNULL:!DH;
           ssl_prefer_server_ciphers on;

           # This is where the magic happens
           location / {
                   # Proxy traffic all traffic received with the meek Host to
                   # a meek-server instance.
                   proxy_pass http://meek.bamsoftware.com:7002;

                   # Disable logging for the reflector.
                   access_log off;
                   error_log /dev/null;
           }
   }

ClientTransportPlugin Line:

ClientTransportPlugin meek exec ./meek-client \
  --url=https://meek-reflect.example.com \
  --front=www.example.com \
  --log meek-client.log

Ideas

Instead of sending TLS with a front SNI, think about sending TLS with no SNI at all. (It might look like a really old browser or a non-browser daemon or something.) Then the censor doesn't have an SNI to match on, and has the choice of blocking an entire IP address (which may virtually host many domains) instead of a single SNI. This idea could be useful in deployment with a CDN, which though it may have thousands of domains, is blockable if we choose just one of those domains as a front.

The App Engine Channel API provides a way to have long-lived push connections to the client, subject to a restricted interface. (HTTP handlers are otherwise required to finish within 60 seconds.) The client could use HTTP request bodies to send data, and a channel to receive, and remove the need for polling. It would require us to reimplement the client JavaScript channel API in order to make use of the particular Comet-based protocol.

Paid apps can create outbound sockets. I don't think it helps us because then the web app would be responsible for managing the session id mapping.

GoAgent is similar in that it also uses App Engine as a middleman.

GTor a.k.a. CloudEntry is based on GoAgent and uses App Engine. It uses App Engine's socket support to make two outgoing connections from App Engine: one to the relay and one back to the client. For that reason, the client can't be behind NAT (just like with flash proxy). The sockets can't live longer than 60 seconds because of App Engine limits, so after that your connection is broken and you have to start again. GTor works as an upstream HTTPProxy for Tor.

Flashlight from Lantern uses the Host header trick with CloudFlare. Like GoAgent, it uses local MITM for HTTPS connections and makes the actual HTTP requests from the remote server.

Users

https://metrics.torproject.org/userstats-bridge-transport.png
Link.

Distinguishability

Barriers to indistinguishability

  1. TLS ciphersuites
    Look like a browser. #4744 has the story of when tor changed its ciphersuite list to look like Firefox's in 2012. tor's list of ciphers is in src/common/ciphers.inc.
  2. TLS extensions
    Look like a browser.
  3. Packet lengths
    Do something to break up fixed-length cells.
  4. Interpacket times
  5. Upstream/downstream bandwidth ratio
  6. Polling interval
    When we have nothing to send, we start polling at 100 ms, and increase the interval by 50% every time no data is received, up to a maximum of 5 s. The growth pattern and the fixed cap is detectable.
    Here's what the fixed polling of 5 s looks like in the GNOME system monitor:
    Screenshot of the GNOME system monitor showing the 5 s polling interval of meek 0.1.
  7. Maximum payload lengths
    Payloads are limited to 65536 bytes. During bootstrapping and bulk downloads, a lot of bodies have exactly this size.
  8. Behavior on random drops
    Suppose the censor drops every hundredth connection to https://www.google.com/. Normal web users will just refresh; meek's stream will be broken.
  9. Number of simultaneous requests
    Browsers open many parallel connections to the same server; I think meek 0.4 opens just one.

Working in our advantage is that we are likely to be transporting web traffic, so we inherit some of its traffic characteristics.

How to look like browser HTTPS

We decided to use a browser extension to make all the HTTPS requests required by meek, so that the requests look just like those made by a browser. There's an extension for Firefox (which works with Tor Browser, so it can work in the browser bundle without shipping a second browser) and one for Chrome. The list below is a summary of a discussion that took place on the tor-dev mailing list and on IRC.

  1. Use your own HTTPS/TLS library, and take care to make sure your ciphersuites and extensions match those of a browser. There are Python bindings for NSS that might make it easier. Chromium is moving to OpenSSL in the future.
  2. Use a separate (headless) browser as an instrument for making HTTPS requests. This is what htpt plans to do.
    PhantomJS is a headless WebKit that is scriptable with JavaScript. Its compressed size is 7–13 MB. This postserver.js example shows it running its own web server, which we could use as a means of communication:

meek-client on localhost ←HTTP→ PhantomJS on localhost ←HTTPS→ www.google.com.

MozRepl (addons.mozilla.org) gives you a JavaScript REPL that allows you to control the browser. It looks like the in-browser JavaScript console, except accessible from outside. Firefox Puppeteer is a fork of MozRepl that is designed for machine-driven browser interaction. Another option is to write an extension for some other browser and communicate with it using some custom IPC.

  1. Use an extension in Tor Browser itself. The plugin bypasses Tor Browser's normal proxy settings in order to issue HTTPS requests directly to the front domain. GeKo says that nsISocketTransportService is what we want to look at. XPCOM is the Firefox API that includes the nsi* functions.
    #11183 is the progress of a browser extension.

Sample client hellos

A big list of client hellos from different applications was moved to meek/SampleClientHellos.

Here is a diff of the client hellos of Firefox 24.4.0 and Tor Browser with meek-http-helper, a browser extension that proxies the requests of meek-client. The only difference is in the client randomness.

--- firefox     2014-04-11 22:10:31.746597046 -0700
+++ meek-http-helper    2014-04-11 22:10:09.303527133 -0700
@@ -1,88 +1,88 @@
 Secure Sockets Layer
     SSL Record Layer: Handshake Protocol: Client Hello
         Content Type: Handshake (22)
         Version: TLS 1.0 (0x0301)
         Length: 169
         Handshake Protocol: Client Hello
             Handshake Type: Client Hello (1)
             Length: 165
             Version: TLS 1.0 (0x0301)
             Random
-                gmt_unix_time: Jul 12, 2089 08:23:06.000000000 PDT
-                random_bytes: f0b149a04ac4a554c5bda57030b17342cc1c0ab59c925cc8...
+                gmt_unix_time: Oct 23, 2081 13:09:42.000000000 PDT
+                random_bytes: 1608e4e50bbc5fb188ab87211ce29f35622d117a4829ebb2...
             Session ID Length: 0
             Cipher Suites Length: 70
             Cipher Suites (35 suites)
                 Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
                 Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
                 Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
                 Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
                 Cipher Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc008)
                 Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012)
                 Cipher Suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA (0xc007)
                 Cipher Suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA (0xc011)
                 Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
                 Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
                 Cipher Suite: TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (0x0045)
                 Cipher Suite: TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA (0x0044)
                 Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
                 Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
                 Cipher Suite: TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0088)
                 Cipher Suite: TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA (0x0087)
                 Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
                 Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
                 Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004)
                 Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e)
                 Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA (0xc005)
                 Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA (0xc00f)
                 Cipher Suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc003)
                 Cipher Suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA (0xc00d)
                 Cipher Suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA (0xc002)
                 Cipher Suite: TLS_ECDH_RSA_WITH_RC4_128_SHA (0xc00c)
                 Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                 Cipher Suite: TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (0x0041)
                 Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                 Cipher Suite: TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0084)
                 Cipher Suite: TLS_RSA_WITH_SEED_CBC_SHA (0x0096)
                 Cipher Suite: SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA (0xfeff)
                 Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
                 Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
                 Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
             Compression Methods Length: 1
             Compression Methods (1 method)
                 Compression Method: null (0)
             Extensions Length: 54
             Extension: server_name
                 Type: server_name (0x0000)
                 Length: 19
                 Server Name Indication extension
                     Server Name list length: 17
                     Server Name Type: host_name (0)
                     Server Name length: 14
                     Server Name: www.google.com
             Extension: renegotiation_info
                 Type: renegotiation_info (0xff01)
                 Length: 1
                 Renegotiation Info extension
                     Renegotiation info extension length: 0
             Extension: elliptic_curves
                 Type: elliptic_curves (0x000a)
                 Length: 8
                 Elliptic Curves Length: 6
                 Elliptic curves (3 curves)
                     Elliptic curve: secp256r1 (0x0017)
                     Elliptic curve: secp384r1 (0x0018)
                     Elliptic curve: secp521r1 (0x0019)
             Extension: ec_point_formats
                 Type: ec_point_formats (0x000b)
                 Length: 2
                 EC point formats Length: 1
                 Elliptic curves point formats (1)
                     EC point format: uncompressed (0)
             Extension: SessionTicket TLS
                 Type: SessionTicket TLS (0x0023)
                 Length: 0
                 Data (0 bytes)
             Extension: next_protocol_negotiation
                 Type: next_protocol_negotiation (0x3374)
                 Length: 0

Style guide

The word "meek" should be written in lower case, even when it is the first word in a sentence. Exception: when it is the last word in a sentence, it should instead be written in ALL CAPS. When printed on glossy paper, "meek" should be followed by a ™ symbol; when handwritten, it should be decisively underlined. You may use the abbrevation "M." in order to save space, but the first use in a document should be spelled out in full with the abbreviation in parentheses. Exception: for every use of an abbreviation after the first, if the number of uses so far is the description number of a non-halting Turing machine, then the "M." should be inverted to become a "W."

Tickets

#10935
Make bundles featuring meek
#10984
PHP relay for meek
#11183
Make an HTTP requestor Firefox extension for meek-client
#11184
Create meek repo
#11393
Make an HTTP requestor Chrome extension for meek-client
#11490
Include meek in userstats-bridge-transport
#12776
Move meek's URL and front configuration into bridge_prefs.js
#13106
Upgrade meek to 0.11
#13247
meek profile error after browser restarts (e.g., after update or add-on installation)
#13586
Use security.ssl.disable_session_identifiers pref in meek-http-helper to restore TLS session tickets
#14033
Upgrade meek to 0.15
#15428
Upgrade meek to 0.16
#15606
Upgrade meek to 0.17
#15872
Meek doesn't start in Tor Browser 4.5 on Windows 7
#15902
Upgrade meek to 0.18
#16014
Windows: staged update fails if Meek is enabled
#16281
Updating to 4.5.1 sets DisableNetwork
#16634
Use new CDN endpoint for meek-azure
#16662
Enable network.http.spdy.* prefs in meek-http-helper for a matching TLS fingerprint
#17330
Figure out what happens when a user's chosen transport is removed from bridge_prefs.js in an update
#17473
Update the meek-amazon fingerprint to B9E7141C594AF25699E0079C1F0146F409495296
#17476
Error console complaining it can't find meek helper
#18167
Don't trust "bridge-ips" blindly for user number estimates
#18517
meek is broken in Tor Browser 6.0a3
#18585
Cannot specify custom meek bridges
#18611
Improve semantics for pluggable transports with dummy addresses
#19646
Mac OS: wrong location for meek browser profile
#19732
"Tor circuit for this site" labels meek bridge as being in China
#20250
meek fails on macOS 10.12 when built with Go 1.4.3 or Go 1.6.3
#20290
Upgrade meek to 0.24
#20495
Unexplained drop in meek users, 2016-10-19 to 2016-11-10
#20781
Figure out how to sandbox meek in a sensible way.
#21342
Move meek-azure to the meek.azureedge.net backend and cymrubridge02 bridge.
#21732
Stop the Meek Tor Browser opening links or documents on macOS
#21918
Move meek-amazon to the d2cly7j4zqgua7.cloudfront.net backend
#22949
Add some IP-HOST pair for meek use
#24614
update to a newer Meek tag

Attachments (17)

Download all attachments as: .zip