add http proxy support to Tor

Many applications, such as wget, apt-get, gpg, etc. do not speak socks, are unlikely to speak socks anytime soon, but support http.

Privoxy or polipo are of no help. They provides only one http port, with the one big drawback: all http connections will be presses through the same SocksPort (identity correlation).

torsocks is of no big help either. I think it has been designed, when identity correlation wasn't a big topic. By default torsocks uses /etc/torsocks.conf and also presses all applications started with usewithtor into the same SocksPort (identity correlation again). To me it also looks like torsocks is practically unmaintained, there is a critical bug open, IPv6 can leak real IP, no progress for a very long time.

As a solution I propose a HttpPort directive for torrc. And stream isolate the same way, Tor stream isolates SocksPort. (Different HttpPorts get isolated, also isolate by http username or http password etc.)

I don't know how much effort it were to add a http proxy. You have already some very basic http support "Tor is not a http proxy". Can you translate the http proxy request and internally redirect it to your existing socks code?

,, Imho "#2846 (closed) Patch GPG to support SOCKS proxies" does not make sense. Why patch individual applications, when you can support all of them by adding http proxy support to Tor... Another Related Ticket: #1667 (moved).

changed milestone to %Tor: unspecified

added component::core tor/tor http-proxy milestone::Tor: unspecified owner::arma points::10 priority::high resolution::duplicate severity::normal status::closed tor-client type::enhancement usability labels

Trac:
Owner: N/A to arma
Status: new to assigned

You're welcome to assign it to me, but that won't make it done any faster.

We wouldn't mind having an http proxy inside Tor, if we could have a simple one that is secure and maintained by somebody else (e.g. as a library that we link in).

We don't currently have any plans to write our own. They suck to write and maintain.

Trac:
Priority: major to normal
Milestone: N/A to Tor: unspecified

It sounds like it would be much, much easier to maintain torsocks and fix it up than to implement an http(s) proxy properly.

Trac:
Milestone: Tor: unspecified to Tor: very long term

Not that I'm disagreeing, but remember that torsocks isn't a Windows solution.

Replying to arma:

You're welcome to assign it to me, but that won't make it done any faster.

We wouldn't mind having an http proxy inside Tor, if we could have a simple one that is secure and maintained by somebody else (e.g. as a library that we link in).

adrelanos, would you compile a list of potential libraries and add it to this ticket?

Whatever happened to Shim?

If I had to take a guess, I'd say that the answer is Tor + libevent + shim and so, I'd guess that it is stuck in bitrot; did it ever work or work very well?

I just cloned git://github.com/nmathewson/shim.git and it works reasonable well:

I ran shim like so:

shim -l 127.0.0.1 -p 8119 -v socks4a://127.0.0.1:9050

I ran gpg (as an example program that needs http proxy suppport) like so:

gpg --keyserver-options http-proxy=http://127.0.0.1:8119,debug,verbose --search jacob@appelbaum.net

shim reported the connection:

conn: socks server set to 127.0.0.1:9050
proxy: listening on 127.0.0.1:8119
proxy: new client connection from 127.0.0.1:44932
proxy: closing idle client connection.

gpg was happy and downloaded the proper key as expected. Sadly, gpg leaked the DNS request.

nickm or arma:

Does it seem reasonable that for 2.4.x we might just launch something like shim, if the system has it? Similar to how we handle tor-fw-helper?

Or do you guys want to see an actual http proxy port opened by Tor the way that we do with SOCKS?

seems to be problematic maintenance-wise, I guess. How would we get a list of things like shim, and see if their version is suitable etc? I think either we ship it, or we don't do anything automatically

Replying to tagnaq:

adrelanos, would you compile a list of potential libraries and add it to this ticket?

I am not sure there are any. There are however "add outgoing http proxy support to my application" ones, but I haven't found any "open http listener".

If I understand right, privoxy and polipo wrote their own http server.

Replying to Sebastian:

seems to be problematic maintenance-wise, I guess. How would we get a list of things like shim, and see if their version is suitable etc? I think either we ship it, or we don't do anything automatically

I think that nick will murder me for saying this but... I think we could ship shim as we do with tor-fw-helper. However for our bundles we can simple ensure that Tor launches it and that it works.

shim "works" today, what is the problem with using it, I wonder?

polipo also works today, but we don't want to ship it. The biggest reason against including it in TBB is that it isn't necessary for any function of TBB, and an additional maintenance burden, and also extra code for the bundle. I do understand that some people want an http proxy, but they are in such a dramatic minority that I do not think it is worthwhile to consider including it in TBB. Now, this isn't to say that you can't provide ways to easily package shim and make it available as an additional download or something like that, so that people who need it can easily get it

Replying to Sebastian:

polipo also works today, but we don't want to ship it.

I think that largely that we don't want to ship it because it had a lot of bugs that we found during an audit. It was also long abandoned and who knows the status now? I'm not sure.

The biggest reason against including it in TBB is that it isn't necessary for any function of TBB, and an additional maintenance burden, and also extra code for the bundle.

I'd want this in the Vidalia bundle and probably not in TBB. In an ideal world, we'd have a way to keep TBB entirely isolated and if users want a system Tor they'd install the Vidalia bundle or some system packages.

I do understand that some people want an http proxy, but they are in such a dramatic minority that I do not think it is worthwhile to consider including it in TBB.

I think the key here is that it is really a pain to use a SOCKS proxy and even more of a pain for users to chain it to some random HTTP proxy. If we have a minimal shim (heh) and it's also managed by Tor, I think we'd find people using it. I know that I would use it. I know that TorBirdy would like to use it because of the gpg issues (#6940 (closed) and #2846 (closed)) among others. Users of TorBirdy at the moment must install the Vidalia bundle - as it stands that is already quite a lot of setup.

Now, this isn't to say that you can't provide ways to easily package shim and make it available as an additional download or something like that, so that people who need it can easily get it

I'm glad that shim appears to work and I'd want to ensure it is actually safe, usable, etc before we'd ship it. I would however point out that Tor's SOCKS port requires no additional setup - it is just there and boy does that confuse users. I can imagine that for TorBirdy, we might actually stuff a Tor and a shim into it if we're really having issues. I would however really like to avoid putting binary components into TorBirdy and rather I'd like to add a useful helper to the Vidalia bundle. Though I wouldn't mind if like tor-fw-helper, shim was managed by tor - it would make me likely to use that same config on many different platforms, even when I don't have Vidalia or X windows.

"Audit shim and bring it up-to-date" is a reasonable thing to do. Somebody would need to take on the responsibility of being shim maintainer. I don't know that shipping shim by default would make senese.

Replying to nickm:

"Audit shim and bring it up-to-date" is a reasonable thing to do.

I'm not sure what it needs - it compiles without warnings (yay) and it seems to function just as it should. It looks "finished" in as much as any C program. :)

It does need compiler hardening and all that stuff added, of course.

Somebody would need to take on the responsibility of being shim maintainer. I don't know that shipping shim by default would make senese.

The open question for me is - "what would it take to make an HTTP proxy port a Tor configuration line as we have with SOCKSPort?"

It seems to me that the easy way is to have a shim binary component and just run it. That isn't the cleanest way, I guess. A feature for 0.2.4.x, perhaps?

It seems to me that the best way is to have Tor do the entire thing in a thread or internally in some other way. A feature for 0.2.4.x or 0.2.5.x, I guess.

nickm - Which way seems reasonable? If the proxy code was inside of Tor - would that be reasonable? Or do you outright reject the idea? :)

If we can't put it in Tor, I guess we're sorta stuck for the applications that require an HTTP proxy - they either all need to become SOCKS aware (not going to happen) or they need to ship an HTTP proxy (possible but painful for all non-native code applications like TorBirdy).

I'm sorta stuck here because I think JonDos has done this correctly - they have a hybrid SOCKS/HTTP proxy port. Everything just works for them without any trouble. It seems to be a pretty good design.

Replying to ioerror:

It seems to me that the easy way is to have a shim binary component and just run it. That isn't the cleanest way, I guess. A feature for 0.2.4.x, perhaps?

Would that support multiple HttpPorts and stream isolation?

Would each HttpPort require one internally opened SocksPort to forward to? (Not implicating that this is bad.)

It seems to me that the best way is to have Tor do the entire thing in a thread or internally in some other way. A feature for 0.2.4.x or 0.2.5.x, I guess.

How much more effort would that be from your perspective?

I'm not sure an http proxy inside Tor necessarily strikes me as the best idea, but a good general purpose mechanism to control stream isolation from outside the Tor daemon itself and a separate HTTP->SOCKS5 proxy designed to make use of that does.