Opened 10 months ago

Closed 4 months ago

#29347 closed enhancement (fixed)

Rewrite meek-http-helper as a WebExtension

Reported by: dcf Owned by: dcf
Priority: Medium Milestone:
Component: Circumvention/meek Version:
Severity: Normal Keywords: webextension
Cc: cohosh, brade, mcs, sukhbir, gk Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by dcf)

Firefox 60 ESR (the current basis of Tor Browser 8) officially doesn't support "legacy" browser extensions using XPCOM/XUL, only the newer WebExtension API.
https://www.mozilla.org/en-US/firefox/60.0esr/releasenotes/#changed
Tor Browser still includes some legacy extensions; apparently what makes them keep working is a extensions.legacy.exceptions pref (#26127; thanks sukhe for knowing that). I don't see where meek-http-helper@bamsoftware.com is being allowed (edit: probably a source patch, thanks mcs), but somehow it is still working too.

Assess whether it's possible to rewrite the helper as a WebExtension, and do it if so. Ideally it will be possible to keep 100% compatibility with the current helper interface; but changing meek-client and meek-client-torbrowser is also an option.

Child Tickets

Change History (21)

comment:1 Changed 10 months ago by dcf

Description: modified (diff)

comment:2 Changed 10 months ago by dcf

The basic domain fronting option seems to be possible. You can't override the Host header in a fetch or XMLHttpRequest because Host is a forbidden header name. I tried it, and any changes I made to Host were silently ignored. However you can set Host in webRequest.onBeforeSendHeaders, at least in Firefox 65, on which I was testing.

The following extension prints out the expected "I’m just a happy little web server." in the --jsconsole.

manifest.json

{
	"manifest_version": 2,
	"name": "Domain fronting demo",
	"version": "1.0",

	"background": {
		"scripts": ["main.js"]
	},

	"permissions": [
		"https://*/*",
		"webRequest",
		"webRequestBlocking"
	]
}

main.js

// https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onBeforeSendHeaders
browser.webRequest.onBeforeSendHeaders.addListener(
    function(details) {
        let requestHeaders = details.requestHeaders.filter(
            h => h.name.toLowerCase() !== "host"
        );
        requestHeaders.push({name: "Host", value: "meek.azureedge.net"});
        return {requestHeaders: requestHeaders};
    },
    {"urls": ["*://*/*"]},
    ["blocking", "requestHeaders"]
);

// https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch
let resp = fetch("https://ajax.aspnetcdn.com/")
    .then(resp => resp.text())
    .then(text => console.log(text));

It's a bit awkward because the listeners like onBeforeSendHeaders are global, not belonging to any single request. Ideally the extension will be able to handle different Host headers for different requests: we need a way to communicate from the outer code where we call fetch to the inner code where we modify the header. One way to do this would be to encode any per-request settings in a magic header or other metadata, which we strip and interpret in the callback. Another way would be to put a lock around the whole fetchonBeforeSendHeaders so that there can only be one in progress at a time, and use a global variable as shared memory. I don't think it would hurt performance because onBeforeSendHeaders is called before anything hits the network; i.e., it should happen almost immediately after fetch and then we can release the lock.

Version 0, edited 10 months ago by dcf (next)

comment:3 Changed 10 months ago by mcs

Cc: brade mcs added

comment:4 Changed 10 months ago by mcs

I think it is the following patch that allows meek-http-helper@bamsoftware.com to work in Tor Browser 8.x:
https://gitweb.torproject.org/tor-browser.git/commit/?h=tor-browser-60.5.0esr-8.5-1&id=10e590c79b01b2a30db1f01a2112c8808696a6cf

I might be pleasantly surprised, but I suspect it will be challenging to reimplement it as a Webextension while remaining compatible with the existing external interface. For example, I don't think there is an easy way to write to stdout. The Native Messaging API is another "tool" which you might need:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging

I was hoping the new direction for the Tor Browser meek client would be to stop using a real browser (but that assumes the TLS fingerprint spoofing works well enough).

comment:5 in reply to:  4 Changed 10 months ago by dcf

Replying to mcs:

I might be pleasantly surprised, but I suspect it will be challenging to reimplement it as a Webextension while remaining compatible with the existing external interface. For example, I don't think there is an easy way to write to stdout.

I haven't tried it all yet, but I've sketched it out mentally and I haven't found any showstoppers yet. (I found and appreciated you catalog of API differences at #17248.) window.dump still works for writing to stdout, as long as the pref browser.dom.window.dump.enabled is set. I'm not hung up on 100% compatibility, but fundamentally we need some way for meek-client to send specifications of HTTP requests into the browser and receive responses (currently we have the browser open a local TCP socket), so I think you're right, we will need the native messaging API. Instead of having the browser open its own socket, it will spawn a subprocess to open the socket and then pass information to/from the subprocess. This is the process tree I envision:

tor
└─meek-client-torbrowser
  ├─firefox --headless
  │ └─socket_shim (opens a socket on 127.0.0.1:XXXX)
  └─meek-client --helper 127.0.0.1:XXXX

meek-client and what I've here called socket_shim communicate over a local TCP socket. The socket_shim can communicate its unpredictable port number XXXX upward through the same native messaging API connection that it uses to exchange HTTP requests/responses.

I was hoping the new direction for the Tor Browser meek client would be to stop using a real browser (but that assumes the TLS fingerprint spoofing works well enough).

That is the plan—there's a near-merge-ready branch at comment:15:ticket:29077 that, if you have time, I'd appreciate you having a look at. Using obfs4proxy meek_lite is also an option, since obfs4proxy also integrated uTLS.

The headless Firefox WebExtension still has value for ESNI. Firefox has an ESNI implementation and Go doesn't. That's what led me to make this ticket—I wanted to try meek-client with ESNI in place of domain fronting, and found that a Firefox that is new enough to support ESNI is also too new to support the current browser extension 😅

comment:6 Changed 10 months ago by dcf

Description: modified (diff)

comment:7 Changed 10 months ago by dcf

Description: modified (diff)

comment:8 Changed 10 months ago by sukhbir

Cc: sukhbir added

comment:9 Changed 10 months ago by dcf

Here is a first version, using the scheme I described in comment:5. It doesn't support domain fronting or a proxy yet, but it is enough to bootstrap 100% with meek-client --helper.

Next I'm going to add domain fronting and proxy support, but feedback is welcome in the meantime.

Setup instructions

I used Firefox 65.

  1. Compile the native application.
    cd webextension/native && go build
    
  2. Edit meek.http.helper.json and set the "path" field to the path to the native application.
    "path": "/path/to/webextension/native/native",
    
  3. Copy the edited meek.http.helper.json file to the OS-appropriate location. The meek.http.helper.json file is called the "host manifest" or "app manifest" and it tells the browser where to find the native part of the WebExtension.
    • macOS
      mkdir -p ~/"Library/Application Support/Mozilla/NativeMessagingHosts/"
      cp meek.http.helper.json ~/"Library/Application Support/Mozilla/NativeMessagingHosts/"
      
    • other Unix
      mkdir -p ~/.mozilla/native-messaging-hosts/
      cp meek.http.helper.json ~/.mozilla/native-messaging-hosts/
      
    • no Windows yet, see below
  4. Run Firefox in a terminal so you can see its stdout. In Firefox, go to about:config and set
    browser.dom.window.dump.enabled=true
    
    This enables the extension to write to stdout.
  5. In Firefox, go to about:debugging and click Load Temporary Add-on.... Find webextension/manifest.json and click Open. In the terminal, you should see a line like this, with a random port number in place of XXXX:
    meek-http-helper: listen 127.0.0.1:XXXX
    

Test it out with this torrc (you need to replace XXXX):

UseBridges 1
ClientTransportPlugin meek exec ./meek-client --helper 127.0.0.1:XXXX --log meek-client.log
Bridge meek 0.0.2.0:1 url=https://meek.bamsoftware.com/

In the browser, open the browser console with Ctrl+Shift+J and you will see the requests being made.

Notes

The WebExtension is made of two parts: the extension and the native application. The extension itself is JavaScript, runs in the browser, and is responsible for making HTTP requests as instructed. The native application runs as a subprocess of the browser; its job is to open a localhost socket and act as an intermediary between the extension and meek-client, because the extension cannot open a socket by itself.

The in-browser part of the WebExtension is actually really really simple now. Most of the XPCOM extension was concerned with IPC, but now native messaging takes care of that. It will get a little more complicated once I add the ability to override the Host header and use a proxy.

The main complication in the native part of the WebExtension is that we have to share the stdio channel to the browser among many request–response pairs. Formerly, the XPCOM extension opened a server socket and did one request–response exchange for each new TCP connection. The TCP connection provided a natural binding between a request and its corresponding response. But now, every request and response must use the permanent stdio–stdout channel. To make this work, I had the native part of the extension tag every request with a random ID before sending it to the browser. When the browser sends back a response, it also tags the response with the same ID. That way, the native part can match them up and continue to provide the same old old-connection-per-roundtrip interface to meek-client.

Another slight complication is that meek-client needs to know the listening port of the native part, which is two levels down in the process hierarchy. To make this work, I added a "report-address" command that the native part can send to the in-browser part to inform it of its address. Then the in-browser part can write it to stdout as before.

Coincidentally, the WebExtension stdio protocol for transmitting JSON objects is almost exactly the same as the one I invented for meek-client --helper. The only difference is that meek-client uses a big-endian length prefix and WebExtension uses native-endian.

The need to install a host manifest (meek.http.helper.json) is a bit of a bummer, as is the requirement that it contain an absolute path to the native app. But I presume we can work around these by setting $HOME to be within the browser bundle, and perhaps rewrite some paths on first run. But the really awkward thing is that on Windows, there is no fixed location for the host manifest; you have to set a registry key pointing to it, one of:

HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\NativeMessagingHosts\meek.http.helper
HKEY_CURRENT_USER\SOFTWARE\Mozilla\NativeMessagingHosts\meek.http.helper

I'm pretty sure it's off-limits for Tor Browser to make such a persistent, global change as setting a registry key. I found the code responsible for doing the lookup. Maybe it's something we can patch.

comment:10 Changed 10 months ago by dcf

I added the ability to control the header (including the Host header) in b2b8b3af03b35d3c1fa794c229f77cc95db5cdf4. Now I can bootstrap with

Bridge meek 0.0.2.0:3 url=https://meek.azureedge.net/ front=ajax.aspnetcdn.com

In order to avoid conflicts with the shared onBeforeSendHeaders resource as described in comment:2, I'm using a mutex-like object that waits on a promise, while simultaneously creating a new promise for the next caller to wait on. We acquire a lock and register an onBeforeSendHeaders listener just before sending the HTTP request, and release the lock and unregister the listener (i.e., resolve the promise) inside the listener itself (i.e., before the HTTP request actually hits the network). I'm pretty new to this JavaScript async stuff so you may want to look at it.

comment:11 Changed 10 months ago by dcf

I added proxy support here:

Now I can bootstrap with

SOCKS4Proxy 127.0.0.1:1080
SOCKS5Proxy 127.0.0.1:1080
HTTPSProxy 127.0.0.1:3128

To test SOCKS4a/5 I used ssh -D 1080. To test http I used ncat -l --proxy-type http.

The strategy to support proxies is similar to the strategy to modify headers. We can only set the proxy in a proxy.onRequest event listener. We use a lock to ensure that only one request is affected at a time.

An unexpected complication was the error behavior of proxy.onRequest. If an error occurs in the event listener--say, you give it an unsupported type, or omit the host field--Firefox will log the error to the console but then proceed as if no proxy were configured. I didn't find a specification of what should happen, but it seems like it may be a bug, so I reported it here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1528873

It's not the kind of problem that can happen unpredictably at runtime: either your configuration is right or it's wrong. But silently ignoring an error when a user has tried to configure a proxy seems wrong, so I implemented a conservative safeguard: a proxy.onError listener checks for an error in proxy.onRequest, and a webRequest.onBeforeRequest listener cancels all requests if a proxy error has ever occurred. Unlike the listeners that modify headers and set the proxy per request, these listeners are static for the lifetime of the extension and don't need locks or unregistration logic.

Next, I'm planning to try integration with Tor Browser. As noted in comment:9, the likely difficulty here will be the static paths and registry entries used by the native messaging API.

comment:12 Changed 10 months ago by dcf

I want to mention an alternative architecture, in case the native messaging aspect turns out to be too awkward to deal with.

Fundamentally we need some kind of channel between meek-client and the browser, so that meek-client can send encoded HTTP requests to the browser, and the browser can send back encoded HTTP responses. With the old XPCOM extension, that channel was an nsiServerSocket that the extension opened itself. In the WebExtension Ihave been working on so far, with native messaging, the channel is a socket opened by the native shim, plus the stdio channel provided by the WebExtension API.

An alternative is to have meek-client run a local web server, and the browser communicate with it by making local HTTP or WebSocket requests. Let's say WebSocket, that's a little easier to explain. meek-client starts a WebSocket server. The browser extension establishes a connection to the server. They then exchange serialized requests and responses.

The thing that starts the local web server doesn't have to be meek-client itself; it could be a separate process. It could be part of meek-client-torbrowser. The important difference is that the separate process would not be a child of firefox. Compare with the diagram in comment:5:

tor
└─meek-client-torbrowser
  ├─meek-client-webserver (opens a WebSocket socket on port YYYY, and a helper server on port XXXX)
  ├─firefox --headless (connects over WebSocket to 127.0.0.1:YYYY)
  └─meek-client --helper 127.0.0.1:XXXX

The communications channel would be (keeping in mind that there are other options, like combining meek-client and meek-client-webserver):

[meek-client] <-- helper protocol --> :XXXX [meek-client-webserver] :YYYY <-- WebSocket --> [firefox]

One difficulty is how to inform the browser extension of the local web server's port number. Before the whole communications channel is set up, there are limited ways to get information into the browser extension. I don't think you can even read environment variables from a browser extension. One option is of course to run the web server on a consistent port, but then you have to deal with the case that the port number is already in use.

comment:13 Changed 10 months ago by dcf

Status: assignedneeds_review

I worked on integrating the WebExtension into Tor Browser. It's working now and ready to be looked at.

I tested it on linux-x86_64 and windows-x86_64, but I'm not set up to test on osx.

Recall that the native messaging API requires us to install a JSON "host manifest" for the native executable--this both authorizes the extension to run a native executable, and tells the browser the (absolute) path to the native executable. The absolute path inside the manifest means we cannot just use a static file; we need to know where the browser is installed. So now, meek-client-torbrowser writes a host manifest (taking into account platform-specific paths) before starting the browser.

Two things I'd specifically like feedback on:

  • I'm not able to test the osx version, which is slightly tricky because the data directory can be in different places depending on TOR_BROWSER_TOR_DATA_DIR (#18904). This is what I'm doing, but I'm not sure if it works:

if TOR_BROWSER_TOR_DATA_DIR is set:

install in $TOR_BROWSER_TOR_DATA_DIR/../Browser

else:

install in $PWD/../../../../TorBrowser-Data/Browser

The documentation says that the host manifest should be installed in $HOME/Library/Application Support/Mozilla/NativeMessagingHosts/, but the code actually does a Services.dirsvc.get for XRE_USER_NATIVE_MANIFESTS, which calls GetUserDataDirectoryHome and then into some Tor Browser–overriden code that replaces the home directory.

  • As noted in comment:9, on windows we cannot simply write the host manifest to a well-known path. You have to set a well-known registry key whose value is the path to the manifest. So what the code does now is write a registry key at HKEY_CURRENT_USER\SOFTWARE\Mozilla\NativeMessagingHosts\meek.http.helper. That works, but I don't like the fact that it leaves a permanent trace outside the installation directory. I'd like to know if there are any ideas for removing this step.

The tor-browser-build changes are minimal: just packaging the webextension directory instead of the firefox directory, building the native executable, and adding a dependency on golang.org/x/sys/windows/registry to write the registry key on windows.

comment:14 in reply to:  13 ; Changed 10 months ago by gk

Cc: gk added

Replying to dcf:

I worked on integrating the WebExtension into Tor Browser. It's working now and ready to be looked at.

I tested it on linux-x86_64 and windows-x86_64, but I'm not set up to test on osx.

If we go with meek_lite in 9.0 as planned in #29430 and given that the currently available extension is working in Tor Browser, then it seems to me there is no need to review your changes from a Tor Browser perspective AND you don't need to worry about the macOS testing, right?

comment:15 in reply to:  14 ; Changed 10 months ago by dcf

Replying to gk:

Replying to dcf:

I worked on integrating the WebExtension into Tor Browser. It's working now and ready to be looked at.

I tested it on linux-x86_64 and windows-x86_64, but I'm not set up to test on osx.

If we go with meek_lite in 9.0 as planned in #29430 and given that the currently available extension is working in Tor Browser, then it seems to me there is no need to review your changes from a Tor Browser perspective AND you don't need to worry about the macOS testing, right?

That's correct, but there are two reasons why it's still worth having a meek browser extension:

  1. to mitigate the risk caused by switching to uTLS--it is a fallback in case something goes catastrophically wrong and can't be fixed quickly
  2. a browser extension building on meek will be the easiest way to prototype a transport based on ESNI

comment:16 in reply to:  15 ; Changed 10 months ago by gk

Replying to dcf:

Replying to gk:

Replying to dcf:

I worked on integrating the WebExtension into Tor Browser. It's working now and ready to be looked at.

I tested it on linux-x86_64 and windows-x86_64, but I'm not set up to test on osx.

If we go with meek_lite in 9.0 as planned in #29430 and given that the currently available extension is working in Tor Browser, then it seems to me there is no need to review your changes from a Tor Browser perspective AND you don't need to worry about the macOS testing, right?

That's correct, but there are two reasons why it's still worth having a meek browser extension:

  1. to mitigate the risk caused by switching to uTLS--it is a fallback in case something goes catastrophically wrong and can't be fixed quickly
  2. a browser extension building on meek will be the easiest way to prototype a transport based on ESNI

Yes, those are good points. However, I'd like to understand what you think we should do for Tor Browser here. In particular, I was wondering whether to spend time on reviewing and testing your changes in a Tor Browser context *now*, with the aim to have all of that merged to the alpha series (so it will eventually be in stable at some point), given the current plan outlined in #29430.

I mean testing a transport based on ESNI in an alpha (which needs an extension) should not be a problem once that is ready and we could easily review the extension and tor-browser-build integration then.

comment:17 in reply to:  16 Changed 10 months ago by yawning

Replying to gk:

Yes, those are good points. However, I'd like to understand what you think we should do for Tor Browser here. In particular, I was wondering whether to spend time on reviewing and testing your changes in a Tor Browser context *now*, with the aim to have all of that merged to the alpha series (so it will eventually be in stable at some point), given the current plan outlined in #29430.

For my reference when should I have a new tag of utls and obfs4proxy by? There's a number of fixes I feel are required in the former, but my free time over the next few weeks will be even tighter than it usually is.

I mean testing a transport based on ESNI in an alpha (which needs an extension) should not be a problem once that is ready and we could easily review the extension and tor-browser-build integration then.

Adding ESNI to utls is likely fairly straight forward if that's the sort of thing people care about.

comment:18 in reply to:  16 Changed 10 months ago by dcf

Replying to gk:

Yes, those are good points. However, I'd like to understand what you think we should do for Tor Browser here. In particular, I was wondering whether to spend time on reviewing and testing your changes in a Tor Browser context *now*, with the aim to have all of that merged to the alpha series (so it will eventually be in stable at some point), given the current plan outlined in #29430.

No, there's no time-sensitive need to do any reviewing now.

Replaying to yawning:

For my reference when should I have a new tag of utls and obfs4proxy by?

Let's please keep other topics on their own tickets.

comment:19 in reply to:  13 Changed 9 months ago by dcf

Replying to dcf:

  • As noted in comment:9, on windows we cannot simply write the host manifest to a well-known path. You have to set a well-known registry key whose value is the path to the manifest. So what the code does now is write a registry key at HKEY_CURRENT_USER\SOFTWARE\Mozilla\NativeMessagingHosts\meek.http.helper. That works, but I don't like the fact that it leaves a permanent trace outside the installation directory. I'd like to know if there are any ideas for removing this step.

I decided to just do a DeleteKey on the registry key before exiting, as a best-effort attempt to clean up the global state. It doesn't attempt to clean any farther up the path, i.e. an empty HKEY_CURRENT_USER\SOFTWARE\Mozilla\NativeMessagingHosts will remain if it was missing to begin with.

comment:20 Changed 6 months ago by dcf

Status: needs_reviewmerge_ready

comment:21 Changed 4 months ago by dcf

Resolution: fixed
Status: merge_readyclosed

In tag 0.34 I've merged the WebExtension and deleted the legacy XPCOM extension. I also rebased the tor-browser-build branch on top of tbb-9.0a4-build2 as meek-webextension_2. The tor-browser-build branch is just for possible future reference, as the decision in #29430 was to use uTLS for TLS camouflage rather than a browser helper.

Two noteworthy points about the merge:

  • #28044 integrated Tor Launcher directly into the browser. This means that even the headless meek-http-helper browser includes a copy of Tor Launcher that (invisibly) tries to run a copy of tor. The additional copy of Tor Launcher quickly fails because it cannot run tor because of a port conflict. The meek-http-helper browser otherwise works for its purpose.
  • The extension uses proxy.settings.set to set network.proxy.socks_remote_dns=false, which is a workaround for one of Tor Browser's patches meant to prevent DNS leaks. In Firefox 67, the proxy.settings.set function will require special permission to work; specifically, it needs to have "Run in Private Windows" checked to work at all (not just in private windows). I've added a comment explaining the situation. To use the extension with Firefox 67+, it is likely that we will need to find a way to set the "Run in Private Windows" flag on the extension.
Note: See TracTickets for help on using tickets.