Opened 6 years ago

Closed 6 years ago

#6940 closed task (fixed)

analyze thunderbird HTTP proxy behaviour

Reported by: tagnaq Owned by: sukhbir
Priority: Medium Milestone:
Component: Applications/TorBirdy Version:
Severity: Keywords:
Cc: ioerror, sukhbir.in@… Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I think the HTTP proxy is only used for HTTP related things - the 
question is - what happens when it has no HTTP proxy? Do HTTP
related things simply leak out?

Child Tickets

Change History (41)

comment:1 Changed 6 years ago by tagnaq

Owner: changed from ioerror to sukhbir
Status: newassigned

The text above is from Jake. Sukhbir is checking on this.

I see these testcases:

  • user has no HTTP proxy set at all
  • user has a HTTP proxy set but there is no proxy running

(- there is a http proxy but not sending his request to tor)

comment:2 Changed 6 years ago by ioerror

Cc: ioerror added

comment:3 Changed 6 years ago by sukhbir

Cc: sukhbir.in@… added

Hi,

[TorBirdy + TBTracer]

  • user has no HTTP proxy set at all

With TorBirdy, I disabled the HTTP/SSL proxies. Thunderbird was not
leaking anything -- everything was being routed through Tor.

Addons - worked.
Updates - worked.

I could see mostly HTTPS connections, however, I do see some plain
HTTP GET requests being made. Say this:

[HTTP] [Sunday, September 23, 2012 6:09:57 PM] GET /contribute/
HTTP/1.1 Host: www.mozilla.org Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate
Connection: keep-alive
[HTTP] [Sunday, September 23, 2012 6:09:59 PM] HTTP/1.1 302 Found
Server: Apache X-Backend-Server: bedrock4.webapp.scl3.mozilla.com
Content-Type: text/html; charset=iso-8859-1 Date: Sun, 23 Sep 2012
18:06:12 GMT

This happens (infrequently) when the addons page (and with specific
addons) is being accessed. So any thoughts about this?

  • user has a HTTP proxy set but there is no proxy running

(the earlier approach)

Addons - not working.
Updates - not working.

I have pushed the new changes (no HTTP/SSL proxies).

comment:4 Changed 6 years ago by sukhbir

(I have not tested Enigmail/HKP traffic yet)

comment:5 Changed 6 years ago by sukhbir

Ok, so I just tested with Enigmail.

HTTP proxy (fail closed): Key fetching/ searching doesn't work.
No HTTP proxy: Enigmail leaks.

It seems from #2846 that we can't get gpg to use a SOCKS proxy, so there's that.

comment:6 Changed 6 years ago by ioerror

I say that we file a bug with Enigmail and try to get SOCKS support into GPG.

comment:7 Changed 6 years ago by ioerror

It looks like gpgkeys_hkpms does support SOCKS.

comment:8 Changed 6 years ago by ioerror

I propose that we call gpg with usewithtor in Enigmail as a temporary work around for those with torsocks/usewithtor - I wonder how to make Enigmail do this?

comment:9 Changed 6 years ago by ioerror

Otherwise, I guess we can try to integrate Moxie's crazy HTTP proxy hack: https://github.com/moxie0/Convergence/blob/master/client/components/LocalProxy.js

comment:10 in reply to:  8 ; Changed 6 years ago by sukhbir

Replying to ioerror:

I propose that we call gpg with usewithtor in Enigmail as a temporary work around for those with torsocks/usewithtor - I wonder how to make Enigmail do this?

We can't do this through Enigmail.

comment:11 Changed 6 years ago by ioerror

I've also mailed the gnupg-devel mailing list and it should show up in the archive soon: http://lists.gnupg.org/pipermail/gnupg-devel/2012-September/date.html

comment:12 in reply to:  10 ; Changed 6 years ago by ioerror

Replying to sukhbir:

Replying to ioerror:

I propose that we call gpg with usewithtor in Enigmail as a temporary work around for those with torsocks/usewithtor - I wonder how to make Enigmail do this?

We can't do this through Enigmail.

Are you sure? Enigmail creates the process - why can't it have a command prefix?

I understand that _currently_ we can't do it - what if we extend Enigmail to be Tor aware?

comment:13 in reply to:  12 ; Changed 6 years ago by sukhbir

Replying to ioerror:

Replying to sukhbir:

We can't do this through Enigmail.

Are you sure? Enigmail creates the process - why can't it have a command prefix?

I understand that _currently_ we can't do it - what if we extend Enigmail to be Tor aware?

Yes, I meant currently :) If we were to extend Enigmail, sure, it is possible. But then it depends on which route do we want to take -- get this done through the gpg folks or through Enigmail or with Moxie's solution. Let's see...

comment:14 in reply to:  13 Changed 6 years ago by ioerror

Replying to sukhbir:

Replying to ioerror:

Replying to sukhbir:

We can't do this through Enigmail.

Are you sure? Enigmail creates the process - why can't it have a command prefix?

I understand that _currently_ we can't do it - what if we extend Enigmail to be Tor aware?

Yes, I meant currently :) If we were to extend Enigmail, sure, it is possible. But then it depends on which route do we want to take -- get this done through the gpg folks or through Enigmail or with Moxie's solution. Let's see...

I think it would be easy to ask for a command line prefix as a feature - unrelated to anything else - it would even be a good idea.

comment:15 Changed 6 years ago by ioerror

At the moment, I think we should set the proxy in Enigmail and let it fail as it has always been failing. Hopefully we can find a solution such as using Moxie's code later. This should allow us to unset the HTTP proxy for the rest of Thunderbird and things should be OK, more functional, even.

comment:16 in reply to:  15 ; Changed 6 years ago by sukhbir

Replying to ioerror:

At the moment, I think we should set the proxy in Enigmail and let it fail as it has always been failing.

Done.

Hopefully we can find a solution such as using Moxie's code later. This should allow us to unset the HTTP proxy for the rest of Thunderbird and things should be OK, more functional, even.

Yup.

comment:17 in reply to:  16 Changed 6 years ago by ioerror

Replying to sukhbir:

Replying to ioerror:

At the moment, I think we should set the proxy in Enigmail and let it fail as it has always been failing.

Done.

Hopefully we can find a solution such as using Moxie's code later. This should allow us to unset the HTTP proxy for the rest of Thunderbird and things should be OK, more functional, even.

Yup.

Moxie is a hero and thinks that the local JavaScript proxy hack is actually possible.

comment:18 Changed 6 years ago by ioerror

I've opened a ticket about the HTTP proxy in javascript: #6958

comment:19 in reply to:  18 ; Changed 6 years ago by mikeperry

Replying to ioerror:

I've opened a ticket about the HTTP proxy in javascript: #6958

I agree that Moxie needs mad props for bending XPCOM to his will with that JS HTTP proxy thing. However, I think a direct SOCKS patch for GPG (#2846) has less vulnerability surface and should be simpler code in total.

The reason I think that the direct SOCKS patch reduces the vulnerability surface is nuanced. It probably doesn't matter unless we actually disable jsctypes in our own builds of Thunderbird (#6152).

So for now, my recommendation is:

  1. Disable GPG network activity by setting the proxy to garbage for now.
  2. Try Moxie's code from #6958. If you can get it to work out of the box within an hour, use it for now
  3. If Moxie's code takes more than an hour to get it to work, you should try to provide a patch for SOCKS support to GPG (#2846). It's possible someone may have already written one already, somewhere...

comment:20 in reply to:  19 Changed 6 years ago by ioerror

Replying to mikeperry:

Replying to ioerror:

I've opened a ticket about the HTTP proxy in javascript: #6958

I agree that Moxie needs mad props for bending XPCOM to his will with that JS HTTP proxy thing. However, I think a direct SOCKS patch for GPG (#2846) has less vulnerability surface and should be simpler code in total.

We don't ship GPG as part of TorBirdy - we have to go with what we have and that is an older gpg, without a currently non-existent SOCKS patch or SOCKS support.

The reason I think that the direct SOCKS patch reduces the vulnerability surface is nuanced. It probably doesn't matter unless we actually disable jsctypes in our own builds of Thunderbird (#6152).

So for now, my recommendation is:

  1. Disable GPG network activity by setting the proxy to garbage for now.

That will break Enigmail for everyone. We're setting it to what should be a Torified HTTP proxy and if it fails, it fails. We won't break it for everyone.

  1. Try Moxie's code from #6958. If you can get it to work out of the box within an hour, use it for now

That is the goal.

  1. If Moxie's code takes more than an hour to get it to work, you should try to provide a patch for SOCKS support to GPG (#2846). It's possible someone may have already written one already, somewhere...

Such a patch is worthwhile but it will not help us until it is tested, merged, and then deployed to everyone. Thus, we'll still be waiting for a long long time. We should do both but first, we'll make a local HTTP proxy somehow.

comment:21 in reply to:  5 Changed 6 years ago by tagnaq

Replying to sukhbir:

Ok, so I just tested with Enigmail.

HTTP proxy (fail closed): Key fetching/ searching doesn't work.
No HTTP proxy: Enigmail leaks.

OK, this is expected behaviour.

comment:22 in reply to:  6 Changed 6 years ago by tagnaq

Replying to ioerror:

I say that we file a bug with Enigmail and try to get SOCKS support into GPG.

Which bug are you about to report?

comment:23 in reply to:  11 Changed 6 years ago by tagnaq

Replying to ioerror:

I've also mailed the gnupg-devel mailing list and it should show up in the archive soon: http://lists.gnupg.org/pipermail/gnupg-devel/2012-September/date.html

http://lists.gnupg.org/pipermail/gnupg-devel/2012-September/026923.html

Looks promising unfortunately an OS specific thing..
Did anyone try it?

comment:24 Changed 6 years ago by tagnaq

gpg --keyserver-options http-proxy=socks5://127.0.0.1:9050 --search jacob@appelbaum.net
[...]
gpgkeys: HTTP search error 7: Unable to receive initial SOCKS5 response.
[...]

Tor SocksPort is running on 9050.
curl version is > 7.21.7.

Is it working for you? (I should probably subscribe to gnupg-devel)

The output with debug

gpg --keyserver-options http-proxy=socks5://127.0.0.1:9050,debug,verbose --search jacob@appelbaum.net

was not much more helpful.

comment:25 Changed 6 years ago by tagnaq

Oh sorry 9050 was not a SocksPort ;)

comment:26 Changed 6 years ago by ioerror

I tried with a SOCKS port of 9050 (Tor):

gpg --keyserver-options http-proxy=socks5://127.0.0.1:9050,debug,verbose --search jacob@appelbaum.net
gpg: searching for "jacob@appelbaum.net" from hkp server pool.sks-keyservers.net
gpgkeys: curl version = GnuPG curl-shim
gpgkeys: search type is 0, and key is "jacob@appelbaum.net"
* HTTP proxy is "socks5://127.0.0.1:9050"
* HTTP URL is "http://pool.sks-keyservers.net:11371/pks/lookup?op=index&options=mr&search=jacob%40appelbaum%2Enet"
* HTTP auth is "null"
* HTTP method is GET
?: invalid HTTP proxy (socks5://127.0.0.1:9050): unsupported URI
gpgkeys: HTTP search error 7: couldn't connect: Success
gpg: key "jacob@appelbaum.net" not found on keyserver
gpg: keyserver internal error
gpg: keyserver search failed: keyserver error

I'm not clear that it is supposed to work that way - I think rather we'd need to write a patch to add a SOCKS5 proxy option. Is that incorrect?

comment:27 Changed 6 years ago by tagnaq

No, there is no patch needed.

The following works fine for me:

gpg --keyserver-options http-proxy=socks5://127.0.0.1:<Insert your SocksPort here>,debug,verbose --search jacob@appelbaum.net

Make sure you are using a recent curl version as David pointed out.

comment:28 Changed 6 years ago by ioerror

gpg2 on Ubuntu seems to support things a little better but it still doesn't work:

gpg2 --keyserver-options http-proxy=socks5://127.0.0.1:9050,debug,verbose --search jacob@appelbaum.net
gpg: searching for "jacob@appelbaum.net" from hkp server pool.sks-keyservers.net
gpgkeys: curl version = libcurl/7.21.3 GnuTLS/2.8.6 zlib/1.2.3.4 libidn/1.18
gpgkeys: search type is 0, and key is "jacob@appelbaum.net"
* About to connect() to proxy 127.0.0.1 port 9050 (#0)
*   Trying 127.0.0.1... * connected
* Connected to 127.0.0.1 (127.0.0.1) port 9050 (#0)
> GET http://pool.sks-keyservers.net:11371/pks/lookup?op=index&options=mr&search=jacob%40appelbaum.net HTTP/1.1
Host: pool.sks-keyservers.net:11371
Accept: */*
Proxy-Connection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache

* HTTP 1.0, assume close after body
< HTTP/1.0 501 Tor is not an HTTP Proxy
< Content-Type: text/html; charset=iso-8859-1
< 
* Closing connection #0
gpg: key "jacob@appelbaum.net" not found on keyserver

comment:29 in reply to:  27 Changed 6 years ago by ioerror

Replying to tagnaq:

No, there is no patch needed.

The following works fine for me:

gpg --keyserver-options http-proxy=socks5://127.0.0.1:<Insert your SocksPort here>,debug,verbose --search jacob@appelbaum.net

Make sure you are using a recent curl version as David pointed out.

There are a few things at play here - a major one is that gpg must be actually using curl (!) and when it does, it must be the right version. Sadly, curl appears to see the right proxy with gpg2 but then it doesn't do the right thing at all.

comment:30 in reply to:  28 Changed 6 years ago by tagnaq

Replying to ioerror:

gpgkeys: curl version = libcurl/7.21.3 GnuTLS/2.8.6 zlib/1.2.3.4 libidn/1.18

Your curl version is < 7.21.7 and therefore probably not supporting the socks5:// proxy yet.

comment:31 Changed 6 years ago by rransom

Make sure you use curl's socks5-hostname proxy type, not socks5 (which leaks DNS queries, at least in curl 7.22.0).

comment:32 Changed 6 years ago by ioerror

Ok - well, I guess we have a partial solution here.

comment:33 Changed 6 years ago by ioerror

So if anything, I think TorBirdy should set:

--keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050,debug,verbose

Or

--keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050,debug,verbose

The full command would look like this:

gpg --keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050 --search jacob@appelbaum.net

comment:34 Changed 6 years ago by ioerror

I've committed a change to how we call gpg in [master cc60822]

In theory we now have working Thunderbird without an HTTP proxy and with a new enough gpg, we're using Tor directly; if gpg is older, I believe it will simply fail.

The only testing left is to see how it fails. Oh and Windows.

comment:35 Changed 6 years ago by ioerror

ChangeLog changes are in [master b3e3919]

comment:36 Changed 6 years ago by tagnaq

<side note>I think this ticket was done with Sukhbirs anwers (results). Now we are basically a duplicate of #2846 - lets continue there? </ side note>

Ok - well, I guess we have a partial solution here.

What can we about Windows user?

Replying to ioerror:

So if anything, I think TorBirdy should set:

--keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050,debug,verbose

Or

--keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050,debug,verbose

I don't see any difference in the first and second options? Do you think we need debug and verbose too?
I suppose we shouldn't use them statically there.

comment:38 Changed 6 years ago by ioerror

Resolution: fixed
Status: assignedclosed

I'm going to call this done but I think we need a different ticket for "Windows and TorBirdy, how does it work?"

comment:39 Changed 6 years ago by tagnaq

Resolution: fixed
Status: closedreopened

We should analyze if we leak in cases where gpg/curl has no socks5 support.

Do we know it already?

comment:40 in reply to:  39 Changed 6 years ago by ioerror

Replying to tagnaq:

We should analyze if we leak in cases where gpg/curl has no socks5 support.

Ah sure - I was thinking that could go into a new ticket.

Do we know it already?

Yes, I have some bad news about it.

comment:41 Changed 6 years ago by ioerror

Resolution: fixed
Status: reopenedclosed

I've put the details into #2846 - lets close this bug.

Note: See TracTickets for help on using tickets.