Opened 7 years ago

Closed 5 years ago

#6733 closed enhancement (fixed)

Patch Firefox to allow addons to set SOCKS username+password per request

Reported by: mikeperry Owned by: ben
Priority: High Milestone:
Component: Firefox Patch Issues Version:
Severity: Keywords: interview, tbb-linkability
Cc: ben.bucksch.news@…, arthuredelstein@… Actual Points:
Parent ID: #3455 Points:
Reviewer: Sponsor:

Description

We need to patch Firefox in order to allow us to set a custom username+password for each request, possibly as an attribute of the nsIHTTPChannel.

Child Tickets

Attachments (9)

socks-username-6733-proxyInfo-1.diff (40.2 KB) - added by ben 7 years ago.
proxyInfo API change
socks-username-6733-proxyInfo-2.diff (40.6 KB) - added by ben 7 years ago.
API change, v2. Slightly polished.
socks-username-6733-proxyInfo-5.diff (56.0 KB) - added by ben 7 years ago.
Implement SOCKS username and password, with pref
socks-username-6733-protocol-1.diff (15.9 KB) - added by ben 7 years ago.
Implement SOCKS username and password, with pref - separate patch, for easier review
socks-username-6733-proxyInfo-8.diff (56.4 KB) - added by ben 7 years ago.
Updated to current Firefox trunk
socks-username-6733-proxyInfo-7.diff (56.2 KB) - added by ben 7 years ago.
Same state, before Firefox update
socks-username-6733-torbutton-1.diff (993 bytes) - added by ben 7 years ago.
Test: Torbutton change to set username per connection
socks-username-6733-proxyInfo-9.diff (41.3 KB) - added by ben 6 years ago.
Updated to Firefox 24; 1 bug fixed
ext.xpi (49.7 KB) - added by ben 6 years ago.
Text extension, using the nsIChannel API to change proxy username/password

Download all attachments as: .zip

Change History (44)

comment:1 Changed 7 years ago by mikeperry

Keywords: interview added

comment:2 Changed 7 years ago by benb

Cc: com.tor.trac@… added

When and from where exactly would you set this?
In current Firefox code, the proxy is determined at the time the channel is created. This is logical, because the proxy determines which kind of physical channel is opened: via TCP directly, or SOCKS or a HTTP proxy, so that needs to happen at the time the channel is opened. Consequently, this can't be changed at any time the channel object exists.

There are several APIs how channels can come into being. If I could see - maybe even run (install a XPI, not a full browser build) - the code, that would help.

Does that code exist at all yet? From what I understood, you need that to segregate all requests made for one site from all requests for another site. Even if both contact google-analytics.com, the two should be separate, by means of a different SOCKS username. That would mean you'd need to intercept requests that the browser makes, not only modify those that you initiate. Does that intercept code exist yet? If you had only one global proxy so far, you might not have needed it, or maybe you have it already for other reasons. Such a hookup would be a task by itself.

comment:3 Changed 7 years ago by benb

Cc: ben.bucksch.news@… added

comment:4 Changed 7 years ago by benb

Status: newneeds_information

comment:5 Changed 7 years ago by benb

Owner: changed from mikeperry to benb
Status: needs_informationassigned

comment:6 Changed 7 years ago by benb

Status: assignedneeds_information

comment:7 Changed 7 years ago by benb

holla. I'm not finding any traces of any username or password in the code. I had assumed that Mozilla currently has one global username and password for the SOCKS proxy. (I don't use SOCKS myself, and never have in the past.) But it seems there's no username/password code at all:
http://mxr.mozilla.org/comm-central/source/mozilla/netwerk/socket/nsSOCKSIOLayer.cpp#459
So, I'll have to add that, too.

comment:8 Changed 7 years ago by ben

Data flow:
My first approach would be to extend nsIProxyInfo with username and hostname, then to let nsISocketProvider.newSocket()/addToSocket() accept nsIProxyInfo instead of individual proxy values. This allows to pass the username from nsSocketTransport2.cpp to nsISOCKSIOLayer.cpp, which implements SOCKS and can send the username to the proxy.

Before nsSocketTransport: Luckily, nsISocketTransportService.createTransport() already accepts nsIProxyInfo. createTransport is called directly by the protocol implementations including http. nsHttpChannel implements nsIProxiesChannel, which allows access to the nsIProxyInfo.

So, you should be able to:
var httpChannel = .....QueryInterface(Ci.nsIHttpChannel);
assert(httpChannel instanceof Ci.nsIHttpChannel);
assert(httpChannel instanceof Ci.nsIProxiedChannel);
var proxy = httpChannel.proxyInfo;
proxy.username = "abc";
proxy.password = "def";

I have yet to verify *when* you would have to do that, and whether the APIs allows you to do that at all before the proxy connection is physically opened.

Changed 7 years ago by ben

proxyInfo API change

Changed 7 years ago by ben

API change, v2. Slightly polished.

comment:9 Changed 7 years ago by ben

I haven't really tested this much, this is just to show the approach to allow some feedback.

comment:10 Changed 7 years ago by ben

Status: needs_informationneeds_review

comment:11 Changed 7 years ago by ben

Owner: changed from benb to ben
Status: needs_reviewaccepted

Mike tells me by email "Overall, it looks like it's heading in the right direction."

comment:12 Changed 7 years ago by benb

Cc: com.tor.trac@… removed

Changed 7 years ago by ben

Implement SOCKS username and password, with pref

comment:13 Changed 7 years ago by ben

The above implements the SOCKS V4 and V5 protocols to send the username and (for V5 only) password.

There are also new prefs "network.proxy.socks_username" and "network.proxy.socks_password" to set them to static values to be used for all connections. This is not the usecase needed here for Tor, but I assume it's useful for other users, and for testing.

Speaking of testing, I don't know a SOCKS server that supports auth. I have tested with OpenSSH "ssh -D 1234 me@proxy", and that works. If I set SOCKS V5 and do not specify a username, the connection works and the remote site sees the IP address of the proxy. If I set SOCKS V5 and set the above username pref, and restart the browser (!), the connection fails entirely, presumably because OpenSSH doesn't support the auth method "username/password". If I set SOCKS V4 and set the above username pref, and restart the browser, the connection works, presumably because OpenSSH ignores the username sent in SOCKS V4 protocol. The two protocols differ a lot when it comes to authentication.

TODO:

  • Test: with a SOCKS V4 and a SOCKS V5 proxy that supports usernames.
  • Per-channel: Find out when exactly to set the username on the channel, and test that.

comment:14 Changed 7 years ago by ben

Protocol overview, esp. re auth method: http://en.wikipedia.org/wiki/SOCKS#SOCKS4

Changed 7 years ago by ben

Implement SOCKS username and password, with pref - separate patch, for easier review

comment:15 Changed 7 years ago by ben

  • Test: with a SOCKS V4 and a SOCKS V5 proxy that supports usernames.

Took me a while (2h) to get the test environment installed and working.
Tried to install dante-server, failed to start with linker error, broken deb package.
Compiled ssocksd <http://ssocks.sourceforge.net> from source, and tested that with SOCKS V5 username and password, with curl and Firefox.
Found a bug in the code, there's something very strange with the combined write/read buffer mData / mDataLength, it wasn't reset properly. I fixed one instance, but there is another problem, because it doesn't read the username response, my function is never called. Need to debug.

comment:16 Changed 7 years ago by mikeperry

How about pointing this at a real tor 0.2.3.x client? It should accept username + password and isolate by default.

As for testing on the actual channel, Torbutton has an on-modify-request observer in the torbutton_http_observer object in torbutton.js. I suppose you could hack it in there:
https://gitweb.torproject.org/torbutton.git/blob/master:/src/chrome/content/torbutton.js#l3878

comment:17 Changed 7 years ago by mikeperry

For a pre-built TBB with a tor 0.2.3.x client, you should be able to use the alphas and drop in your own browser. Just get the latest version here:
https://www.torproject.org/dist/torbrowser/linux/ (appears to be 2.3.24-alpha-1).

comment:18 Changed 7 years ago by mikeperry

benb: By the way, we probably will actually use the Torbutton observer change to deploy this. Just clarifying because I know I told you initially that we'd prefer to make such a change entirely in Firefox to eliminate our need for the observer, but something that works is better than nothing here.

comment:19 Changed 7 years ago by ben

I finally managed to get torbutton to work with my self-compiled Firefox, and my patch seems to work just fine.

In the process, I updated Firefox to current trunk, which caused lots of conflicts in the patches. I'll attach a new patch.

Changed 7 years ago by ben

Updated to current Firefox trunk

Changed 7 years ago by ben

Same state, before Firefox update

comment:20 Changed 7 years ago by ben

Given the conflicts, I think it's best to upstream this.
A bugzilla bug already exists: https://bugzilla.mozilla.org/show_bug.cgi?id=122752

Changed 7 years ago by ben

Test: Torbutton change to set username per connection

comment:21 Changed 7 years ago by ben

according to wireshark, I am sending the username, and the server sends success, but then nothing. the browser doesn't finish loading. wireshark can't decypher the response. I don't know how to debug this further. I think I'll have to write a unit test with a SOCKS server. I assume the bug is in my code.
I'll also try some printf() debugging of that SOCKS state machine.

comment:22 Changed 7 years ago by mikeperry

One approach might be to look at how Tor 0.2.3.x handles SOCKS usernames from a working app with Wireshark. I believe Pidgin sends them in a way that Tor 0.2.3.x is happy with.

comment:23 Changed 7 years ago by mikeperry

In Pidgin, you can set the SOCKS username and password under the account-specific "Proxy" settings. Pidgin versions for which this is known to work properly have a "Tor/Privacy (SOCKS)" proxy type dropdown option.

comment:24 Changed 7 years ago by mikeperry

Turns out there is a bug with Tor's SOCKS handshake that causes username+password not to be sent if you're actually trying to negotiate an auth method. See #8117. Shouldn't cause total connection failure though...

comment:25 Changed 6 years ago by mikeperry

Keywords: tbb-linkability added

comment:26 Changed 6 years ago by ben

Seems like #8117 was fixed now.

comment:27 Changed 6 years ago by ben

  • Updated the patch to current tor browser based on Firefox 24 ESR.
  • Found a SOCKS server that's trivial to install, configure and run: JSocks <http://jsocks.sourceforge.net/>
  • Tested JSocks:
    • Normal Firefox, without SOCKS username: works
    • Tor browser with my patch, without SOCKS username, with SOCKS server hostname: fails (DNS lookups intentionally broken?)
    • Tor browser with my patch, without SOCKS username, with SOCKS server as IP address: works
    • Tor browser with my patch, with SOCKS username, SOCKS v5: FAIL
    • Tor browser with my patch, with SOCKS username, SOCKS v4: Auth succeed, request goes out, then is aborted by SOCKS server.
  • Note for all tests: After changing proxy settings, restart browser. Also, disable cache (in normal Firefox).
Last edited 6 years ago by ben (previous) (diff)

comment:28 Changed 6 years ago by ben

  • Tested JSocks:
    • Tor browser with my patch, with SOCKS username, SOCKS v5: FAIL
    • Tor browser with my patch, with SOCKS username, SOCKS v4: Auth succeed, request goes out, then is aborted by SOCKS server.

sigh That is a bug in the SOCKS server, too. It doesn't implement username/password properly. It has a configure file option to have usernames and passwords, and actually has a username/password implementation, but nonetheless uses auth method none (and checking against Ident) instead of username/password, and fails when the client sends username/password method. Who knows what the developer was thinking...

In Wireshark, what I send looks correct. Compare comment:21.

Last edited 6 years ago by ben (previous) (diff)

comment:29 Changed 6 years ago by ben

Furthermore, SOCKS v4 is broken for me in the original Tor browser (downloaded from website, without my patch) with Tor. SOCKS v5 works.
No wonder I'm having problems to test this.


I found one bug in my patch, in the state machine for SOCKS v5, and fixed that.


There's one more bug.

Note to self: It's stopping at this point:

        case SOCKS5_WRITE_USERNAME_REQUEST:
            if (WriteToSocket(fd) != PR_SUCCESS)
                return PR_FAILURE;
            WantRead(2);
            mState = SOCKS5_READ_USERNAME_RESPONSE;
            return PR_SUCCESS;
        case SOCKS5_READ_USERNAME_RESPONSE:
            if (ReadFromSocket(fd) != PR_SUCCESS)    <==== failing here
                return PR_FAILURE;
            return ReadV5UsernameResponse();

I do see the response from the SOCKS server in Wireshark. Just the Mozilla code doesn't read the bytes. I have to figure out why. The |WantRead(2);| before is an obvious culprit, but we should be getting 2 bytes, and I do see 2 bytes in Wireshark coming back.

However, in ReadFromSocket(), PR_Read() returns -1 with PR_WOULD_BLOCK_ERROR. Normally, that shouldn't be fatal, but it seems that's where everything stops.

http://mxr.mozilla.org/comm-central/source/mozilla/netwerk/socket/nsSOCKSIOLayer.cpp

Last edited 6 years ago by ben (previous) (diff)

comment:30 Changed 6 years ago by ben

Status: acceptedneeds_information

Changed 6 years ago by ben

Updated to Firefox 24; 1 bug fixed

comment:31 Changed 6 years ago by ben

(xref #10848 HTTP proxy)

comment:32 Changed 6 years ago by ben

I wrote a test extension, to verify that the API is available at all, and that we can access and change the information at all, after the channel has been created. It may be too late to change the proxy info at that point. Also, proxies connections might be cached. Both could potentially be a show-stopper.

Seems that is indeed a problem, even earlier than I thought.

I wrote in comment:8

So, you should be able to:
var httpChannel = .....QueryInterface(Ci.nsIHttpChannel);
assert(httpChannel instanceof Ci.nsIHttpChannel);
assert(httpChannel instanceof Ci.nsIProxiedChannel);
var proxy = httpChannel.proxyInfo;

Rather:

var req = new XMLHttpRequest();
  // or Cc["@mozilla.org/xmlextras/xmlhttprequest;1"]
  // .createInstance(Ci.nsIXMLHttpRequest);
req.open("GET", "http://www.torproject.org");
var channel = req.channel;
assert(channel instanceof Ci.nsIHttpChannel);
assert(channel instanceof Ci.nsIProxiedChannel);
var channel = channel.QueryInterface(Ci.nsIProxiedChannel);
assert(channel, "nsIProxiedChannel not found");
assert(channel.proxyInfo, "proxyInfo not found on nsIProxiedChannel");
var proxy = channel.proxyInfo;

But proxyInfo is null, I get "proxyInfo not found on nsIProxiedChannel".

I get this both in a stock Firefox with an HTTP proxy, and in Tor browser with my changes. I didn't add the proxyInfo attribute, so this is unrelated to my changes. So, I'll have to look more into the data flow of the channel and proxy, to see when a proxyInfo is available.

Changed 6 years ago by ben

Attachment: ext.xpi added

Text extension, using the nsIChannel API to change proxy username/password

comment:33 Changed 5 years ago by arthuredelstein

Cc: arthuredelstein@… added

comment:34 Changed 5 years ago by arthuredelstein

New patches, including a modified version of Ben's patch, above (socks-username-6733-proxyInfo-9.diff) have been posted to #3455.

comment:35 Changed 5 years ago by mikeperry

Resolution: fixed
Status: needs_informationclosed

These are merged for 4.5-alpha-1.

Note: See TracTickets for help on using tickets.