Opened 5 years ago

Closed 6 months ago

#6060 closed enhancement (duplicate)

add http proxy support to Tor

Reported by: proper Owned by: arma
Priority: High Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: tor-client usability http-proxy
Cc: proper, intrigeri Actual Points:
Parent ID: Points: 10
Reviewer: Sponsor:

Description

Many applications, such as wget, apt-get, gpg, etc. do not speak socks, are unlikely to speak socks anytime soon, but support http.

Privoxy or polipo are of no help. They provides only one http port, with the one big drawback: all http connections will be presses through the same SocksPort (identity correlation).

torsocks is of no big help either. I think it has been designed, when identity correlation wasn't a big topic. By default torsocks uses /etc/torsocks.conf and also presses all applications started with usewithtor <app> into the same SocksPort (identity correlation again). To me it also looks like torsocks is practically unmaintained, there is a critical bug open, IPv6 can leak real IP, no progress for a very long time.

As a solution I propose a HttpPort directive for torrc. And stream isolate the same way, Tor stream isolates SocksPort. (Different HttpPorts get isolated, also isolate by http username or http password etc.)

I don't know how much effort it were to add a http proxy. You have already some very basic http support "Tor is not a http proxy". Can you translate the http proxy request and internally redirect it to your existing socks code?


Imho "#2846 Patch GPG to support SOCKS proxies" does not make sense. Why patch individual applications, when you can support all of them by adding http proxy support to Tor... Another Related Ticket: #1667.

Child Tickets

Change History (43)

comment:1 Changed 5 years ago by proper

Owner: set to arma
Status: newassigned

comment:2 Changed 5 years ago by arma

Milestone: Tor: unspecified
Priority: majornormal

You're welcome to assign it to me, but that won't make it done any faster.

We wouldn't mind having an http proxy inside Tor, if we could have a simple one that is secure and maintained by somebody else (e.g. as a library that we link in).

We don't currently have any plans to write our own. They suck to write and maintain.

comment:3 Changed 5 years ago by nickm

Milestone: Tor: unspecifiedTor: very long term

It sounds like it would be much, much easier to maintain torsocks and fix it up than to implement an http(s) proxy properly.

comment:4 Changed 5 years ago by arma

Not that I'm disagreeing, but remember that torsocks isn't a Windows solution.

comment:5 in reply to:  2 Changed 5 years ago by tagnaq

Replying to arma:

You're welcome to assign it to me, but that won't make it done any faster.

We wouldn't mind having an http proxy inside Tor, if we could have a simple one that is secure and maintained by somebody else (e.g. as a library that we link in).

adrelanos, would you compile a list of potential libraries and add it to this ticket?

comment:6 Changed 5 years ago by ioerror

Whatever happened to Shim?

comment:7 Changed 5 years ago by ioerror

If I had to take a guess, I'd say that the answer is Tor + libevent + shim and so, I'd guess that it is stuck in bitrot; did it ever work or work very well?

comment:8 Changed 5 years ago by ioerror

I just cloned git://github.com/nmathewson/shim.git and it works reasonable well:

I ran shim like so:

shim -l 127.0.0.1 -p 8119 -v socks4a://127.0.0.1:9050

I ran gpg (as an example program that needs http proxy suppport) like so:

gpg --keyserver-options http-proxy=http://127.0.0.1:8119,debug,verbose --search jacob@appelbaum.net

shim reported the connection:

conn: socks server set to 127.0.0.1:9050
proxy: listening on 127.0.0.1:8119
proxy: new client connection from 127.0.0.1:44932
proxy: closing idle client connection.

gpg was happy and downloaded the proper key as expected. Sadly, gpg leaked the DNS request.

comment:9 Changed 5 years ago by ioerror

nickm or arma:

Does it seem reasonable that for 2.4.x we might just launch something like shim, if the system has it? Similar to how we handle tor-fw-helper?

Or do you guys want to see an actual http proxy port opened by Tor the way that we do with SOCKS?

comment:10 Changed 5 years ago by Sebastian

seems to be problematic maintenance-wise, I guess. How would we get a list of things like shim, and see if their version is suitable etc? I think either we ship it, or we don't do anything automatically

comment:11 Changed 5 years ago by proper

Replying to tagnaq:

adrelanos, would you compile a list of potential libraries and add it to this ticket?

I am not sure there are any. There are however "add outgoing http proxy support to my application" ones, but I haven't found any "open http listener".

If I understand right, privoxy and polipo wrote their own http server.

comment:12 in reply to:  10 Changed 5 years ago by ioerror

Replying to Sebastian:

seems to be problematic maintenance-wise, I guess. How would we get a list of things like shim, and see if their version is suitable etc? I think either we ship it, or we don't do anything automatically

I think that nick will murder me for saying this but... I think we could ship shim as we do with tor-fw-helper. However for our *bundles* we can simple ensure that Tor launches it and that it works.

shim "works" today, what is the problem with using it, I wonder?

comment:13 Changed 5 years ago by Sebastian

polipo also works today, but we don't want to ship it. The biggest reason against including it in TBB is that it isn't necessary for any function of TBB, and an additional maintenance burden, and also extra code for the bundle. I do understand that some people want an http proxy, but they are in such a dramatic minority that I do not think it is worthwhile to consider including it in TBB. Now, this isn't to say that you can't provide ways to easily package shim and make it available as an additional download or something like that, so that people who need it can easily get it

comment:14 in reply to:  13 Changed 5 years ago by ioerror

Replying to Sebastian:

polipo also works today, but we don't want to ship it.

I think that largely that we don't want to ship it because it had a lot of bugs that we found during an audit. It was also long abandoned and who knows the status now? I'm not sure.

The biggest reason against including it in TBB is that it isn't necessary for any function of TBB, and an additional maintenance burden, and also extra code for the bundle.

I'd want this in the Vidalia bundle and probably not in TBB. In an ideal world, we'd have a way to keep TBB entirely isolated and if users want a system Tor they'd install the Vidalia bundle or some system packages.

I do understand that some people want an http proxy, but they are in such a dramatic minority that I do not think it is worthwhile to consider including it in TBB.

I think the key here is that it is really a pain to use a SOCKS proxy and even more of a pain for users to chain it to some random HTTP proxy. If we have a minimal shim (heh) and it's also managed by Tor, I think we'd find people using it. I know that I would use it. I know that TorBirdy would like to use it because of the gpg issues (#6940 and #2846) among others. Users of TorBirdy at the moment must install the Vidalia bundle - as it stands that is already quite a lot of setup.

Now, this isn't to say that you can't provide ways to easily package shim and make it available as an additional download or something like that, so that people who need it can easily get it

I'm glad that shim appears to work and I'd want to ensure it is actually safe, usable, etc before we'd ship it. I would however point out that Tor's SOCKS port requires no additional setup - it is just there and boy does that confuse users. I can imagine that for TorBirdy, we might actually stuff a Tor and a shim into it if we're really having issues. I would however really like to avoid putting binary components into TorBirdy and rather I'd like to add a useful helper to the Vidalia bundle. Though I wouldn't mind if like tor-fw-helper, shim was managed by tor - it would make me likely to use that same config on many different platforms, even when I don't have Vidalia or X windows.

comment:15 Changed 5 years ago by nickm

"Audit shim and bring it up-to-date" is a reasonable thing to do. Somebody would need to take on the responsibility of being shim maintainer. I don't know that shipping shim by default would make senese.

comment:16 in reply to:  15 ; Changed 5 years ago by ioerror

Replying to nickm:

"Audit shim and bring it up-to-date" is a reasonable thing to do.

I'm not sure what it needs - it compiles without warnings (yay) and it seems to function just as it should. It looks "finished" in as much as any C program. :)

It does need compiler hardening and all that stuff added, of course.

Somebody would need to take on the responsibility of being shim maintainer. I don't know that shipping shim by default would make senese.

The open question for me is - "what would it take to make an HTTP proxy port a Tor configuration line as we have with SOCKSPort?"

It seems to me that the _easy_ way is to have a shim binary component and just run it. That isn't the cleanest way, I guess. A feature for 0.2.4.x, perhaps?

It seems to me that the _best_ way is to have Tor do the entire thing in a thread or internally in some other way. A feature for 0.2.4.x or 0.2.5.x, I guess.

nickm - Which way seems reasonable? If the proxy code was inside of Tor - would that be reasonable? Or do you outright reject the idea? :)

If we can't put it in Tor, I guess we're sorta stuck for the applications that require an HTTP proxy - they either all need to become SOCKS aware (not going to happen) or they need to ship an HTTP proxy (possible but painful for all non-native code applications like TorBirdy).

I'm sorta stuck here because I think JonDos has done this correctly - they have a hybrid SOCKS/HTTP proxy port. Everything just works for them without any trouble. It seems to be a pretty good design.

comment:17 in reply to:  16 ; Changed 5 years ago by proper

Replying to ioerror:

It seems to me that the _easy_ way is to have a shim binary component and just run it. That isn't the cleanest way, I guess. A feature for 0.2.4.x, perhaps?

Would that support multiple HttpPorts and stream isolation?

Would each HttpPort require one internally opened SocksPort to forward to? (Not implicating that this is bad.)

It seems to me that the _best_ way is to have Tor do the entire thing in a thread or internally in some other way. A feature for 0.2.4.x or 0.2.5.x, I guess.

How much more effort would that be from your perspective?

comment:18 Changed 5 years ago by andrea

I'm not sure an http proxy *inside Tor* necessarily strikes me as the best idea, but a good general purpose mechanism to control stream isolation from outside the Tor daemon itself and a separate HTTP->SOCKS5 proxy designed to make use of that does.

comment:19 in reply to:  17 ; Changed 5 years ago by ioerror

Replying to proper:

Replying to ioerror:

It seems to me that the _easy_ way is to have a shim binary component and just run it. That isn't the cleanest way, I guess. A feature for 0.2.4.x, perhaps?

Would that support multiple HttpPorts and stream isolation?

I'd imagine it would be just like SocksPort.

Would each HttpPort require one internally opened SocksPort to forward to? (Not implicating that this is bad.)

God, I hope not.

It seems to me that the _best_ way is to have Tor do the entire thing in a thread or internally in some other way. A feature for 0.2.4.x or 0.2.5.x, I guess.

How much more effort would that be from your perspective?

Honestly, it depends on what nick has to say - I'm fairly certain that an easy thing is to spawn a thread for each HTTPPort and connect it internally to the SocksPort parsing code. Even easier would be to chain it to a SocksPort that Tor already has configured (ewww).

comment:20 in reply to:  18 ; Changed 5 years ago by rransom

Replying to andrea:

I'm not sure an http proxy *inside Tor* necessarily strikes me as the best idea,

It's a horrible idea.

but a good general purpose mechanism to control stream isolation from outside the Tor daemon itself and a separate HTTP->SOCKS5 proxy designed to make use of that does.

The stream isolation code supports isolation based on SOCKS username and password. That should be sufficient.

comment:21 in reply to:  19 Changed 5 years ago by ioerror

Replying to andrea:

I'm not sure an http proxy *inside Tor* necessarily strikes me as the best idea, but a good general purpose mechanism to control stream isolation from outside the Tor daemon itself and a separate HTTP->SOCKS5 proxy designed to make use of that does.

I generally agree from every perspective excepting usability. As it stands now our users have a spattering of different http proxy servers or _no_ HTTP proxy server. The former results in crazy mis-configurations and security issues, the latter results in leaks (!) and worse.

If Tor launches it as we do with tor-fw-helper, I think it isn't really "inside Tor" in a meaningful sense. I think a thread is generally "inside Tor" and in this case, I'm torn - if it is a thread, users will just get an HTTP proxy without any fuss, once we solve the fussy part for them. If it is a program we exec(), I am certain it won't be as straight forward for them to actually use it.

comment:22 in reply to:  20 ; Changed 5 years ago by ioerror

Replying to rransom:

Replying to andrea:

I'm not sure an http proxy *inside Tor* necessarily strikes me as the best idea,

It's a horrible idea.

Why is that? Do you also feel like the Socks proxy inside of Tor is a horrible idea?

but a good general purpose mechanism to control stream isolation from outside the Tor daemon itself and a separate HTTP->SOCKS5 proxy designed to make use of that does.

The stream isolation code supports isolation based on SOCKS username and password. That should be sufficient.

Indeed, if we take that route, I agree.

comment:23 Changed 5 years ago by nickm

Replying to ioerror:

Replying to nickm:

"Audit shim and bring it up-to-date" is a reasonable thing to do.

I'm not sure what it needs - it compiles without warnings (yay) and it seems to function just as it should. It looks "finished" in as much as any C program. :)

It does need compiler hardening and all that stuff added, of course.

What it needs IMO is auditing for security and standards compliance.

Somebody would need to take on the responsibility of being shim maintainer. I don't know that shipping shim by default would make senese.

The open question for me is - "what would it take to make an HTTP proxy port a Tor configuration line as we have with SOCKSPort?"

For me, that's not a goal. Tor is not an all-singing all-dancing all-purpose application launcher, nor do I want to push more code into the main Tor process. I'd like us to move in the direction of moving functionality out of Tor.

It seems to me that the _easy_ way is to have a shim binary component and just run it. That isn't the cleanest way, I guess. A feature for 0.2.4.x, perhaps?

It seems to me that the _best_ way is to have Tor do the entire thing in a thread or internally in some other way. A feature for 0.2.4.x or 0.2.5.x, I guess.

nickm - Which way seems reasonable? If the proxy code was inside of Tor - would that be reasonable? Or do you outright reject the idea? :)

Strongly reject. Tor is not inetd; Tor is not systemd. "It could be in tor" and "applications could use it" are not a justification for putting it in Tor.

We *do* need a better way to handle the multiple pieces of the Tor experience, but "do everything in Tor" is exactly the wrong answer.

comment:24 in reply to:  22 ; Changed 5 years ago by rransom

Replying to ioerror:

Replying to rransom:

Replying to andrea:

I'm not sure an http proxy *inside Tor* necessarily strikes me as the best idea,

It's a horrible idea.

Why is that? Do you also feel like the Socks proxy inside of Tor is a horrible idea?

HTTP is far more complex than SOCKS.

comment:25 in reply to:  24 Changed 5 years ago by ioerror

Replying to rransom:

Replying to ioerror:

Replying to rransom:

Replying to andrea:

I'm not sure an http proxy *inside Tor* necessarily strikes me as the best idea,

It's a horrible idea.

Why is that? Do you also feel like the Socks proxy inside of Tor is a horrible idea?

HTTP is far more complex than SOCKS.

Shim's HTTP proxy code is more complex than a SOCKS proxy and on that, I think we agree. The question is - if we have that kind of complexity is there a benefit?

Onion routing is in itself quite complex but for the goal of anonymity, we're taking a more complex route to make the goal a reality. Complexity is not a really compelling argument against the interface - bugs exist in all tools - the compelling argument for me would be that it wouldn't help anyone to be more anonymous or to use Tor more safely than they are today. That argument though wouldn't fly because the reality appears to be the exact opposite. Users *require* an HTTP interface to Tor and they're either getting it unsafely or they're not getting it at all. That seems less than ideal.

comment:26 in reply to:  23 Changed 5 years ago by ioerror

Replying to nickm:

Replying to ioerror:

Replying to nickm:

"Audit shim and bring it up-to-date" is a reasonable thing to do.

I'm not sure what it needs - it compiles without warnings (yay) and it seems to function just as it should. It looks "finished" in as much as any C program. :)

It does need compiler hardening and all that stuff added, of course.

What it needs IMO is auditing for security and standards compliance.

Ok - I think both of those are reasonable. What is the absolute canonical git repo? Is it yours? If so, I'd be willing to perform both of those audits. I would like some guidance of which standards matter to us and what specific security issues that concern us.

I assume we want to consider the security issues of a program written in C; {buffer,heap,int} overflows, format strings, {double,use after}free(), etc.

I also assume we'd want to ensure that the HTTP/1.1 standard is followed as much as is possible as far as shim's clients are concerned. I don't know of any standard that regulates HTTP->SOCKS4a proxies, so I think we're flying by the seat of our pants there. :)

Any requirements of such a thing - regardless of where we put it - I'm open to considering and trying to resolve.

Somebody would need to take on the responsibility of being shim maintainer. I don't know that shipping shim by default would make senese.

The open question for me is - "what would it take to make an HTTP proxy port a Tor configuration line as we have with SOCKSPort?"

For me, that's not a goal. Tor is not an all-singing all-dancing all-purpose application launcher, nor do I want to push more code into the main Tor process. I'd like us to move in the direction of moving functionality out of Tor.

Ok, I think our goals aren't so different here. I don't want a full HTTP proxy with caching - I want the most minimal thing that will help reduce harm for our users. I think there is a balance to be struck and that is what happened with DNSPort - it is a minimal thing that at least gives some of the features that our users need. It has been extremely handy, even if imperfect or limited; it isn't standards compliant but holy cow, it is useful!

I am hoping to solve this with a clean design in #6948 - so I totally hear you. I'm also in favor of that as a reality, sooner rather than later. If we can solve #6948, I would say we could break out each of these things quite nicely and move more and more code out of Tor proper. I am totally a fan of that while also being concerned that we may not succeed anytime soon.

It seems to me that the _easy_ way is to have a shim binary component and just run it. That isn't the cleanest way, I guess. A feature for 0.2.4.x, perhaps?

It seems to me that the _best_ way is to have Tor do the entire thing in a thread or internally in some other way. A feature for 0.2.4.x or 0.2.5.x, I guess.

nickm - Which way seems reasonable? If the proxy code was inside of Tor - would that be reasonable? Or do you outright reject the idea? :)

Strongly reject. Tor is not inetd; Tor is not systemd. "It could be in tor" and "applications could use it" are not a justification for putting it in Tor.

Ok. That settles it, I guess. If both options are rejected, even as a thread that doesn't loop to a SocksPort, I'll continue with designs in #6948.

We *do* need a better way to handle the multiple pieces of the Tor experience, but "do everything in Tor" is exactly the wrong answer.

Perhaps some hybrid choices exist that might satisfy us both? I'm open to finding any kind of middle ground that might reduce harm to users (such as those that suffer from the gpg bug #2846) - if that means I need to write a tiny launcher like in #6948, Ok!

I think the clear take away is that Tor will probably not have an HTTPPort like we a SocksPort or a DNSPort. I accept that reality and I will work on #6948 unless there is a hybrid choice that you propose. If there is no such hybrid notion that would be reasonable to get into 2.4.x or later, I'd ask that we close the ticket and add an FAQ item.

comment:27 Changed 5 years ago by proper

Before giving up, please wait a few minutes for my proposal.

comment:28 Changed 5 years ago by nickm

Replying to ioerror:

Replying to nickm:

Replying to ioerror:

Replying to nickm:

"Audit shim and bring it up-to-date" is a reasonable thing to do.

I'm not sure what it needs - it compiles without warnings (yay) and it seems to function just as it should. It looks "finished" in as much as any C program. :)

It does need compiler hardening and all that stuff added, of course.

What it needs IMO is auditing for security and standards compliance.

Ok - I think both of those are reasonable. What is the absolute canonical git repo? Is it yours?

There is none; mine is the closest there is.

If so, I'd be willing to perform both of those audits. I would like some guidance of which standards matter to us and what specific security issues that concern us.

Well, if it's going to claim to be an HTTP proxy, it should implement some version of HTTP. Probably 1.1?

I don't know which specific security issues most affect http proxies.

[...]

Any requirements of such a thing - regardless of where we put it - I'm open to considering and trying to resolve.

Somebody would need to take on the responsibility of being shim maintainer. I don't know that shipping shim by default would make senese.

The open question for me is - "what would it take to make an HTTP proxy port a Tor configuration line as we have with SOCKSPort?"

For me, that's not a goal. Tor is not an all-singing all-dancing all-purpose application launcher, nor do I want to push more code into the main Tor process. I'd like us to move in the direction of moving functionality out of Tor.

Ok, I think our goals aren't so different here. I don't want a full HTTP proxy with caching - I want the most minimal thing that will help reduce harm for our users. I think there is a balance to be struck and that is what happened with DNSPort - it is a minimal thing that at least gives some of the features that our users need. It has been extremely handy, even if imperfect or limited; it isn't standards compliant but holy cow, it is useful!

I am hoping to solve this with a clean design in #6948 - so I totally hear you. I'm also in favor of that as a reality, sooner rather than later. If we can solve #6948, I would say we could break out each of these things quite nicely and move more and more code out of Tor proper. I am totally a fan of that while also being concerned that we may not succeed anytime soon.

Is *that* what #6948 was about? I have no idea what a zygote is, or why shared memory mutexes were something we needed, so I kinda assumed it was an 'implementation technique' ticket, not a 'better architecture' ticket.

[...]

Ok. That settles it, I guess. If both options are rejected, even as a thread that doesn't loop to a SocksPort, I'll continue with designs in #6948.

Well, nothing is settled -- I can be wrong, and I hope I can change my mind if I am.

But I don't think I'm wrong here. HTTP is a very complex protocol, and I think about 100% of what you'd need to do with an HTTP proxy can be done out-of-process from Tor, as I understand it.

If the only argument against a separate proxy is "but then you would have to run two programs", I don't think it's a great one, since having more processes is the direction we should IMO be moving towards, for better security and modularity.

comment:29 Changed 5 years ago by proper

Since Nick unfortunately rejected (or not?) being the http proxy directly inside Tor AND managing Shim with HttpPort... Here is my proposal...

Who says torc may only be phrased by Tor? Users can still add HttPort's to torrc. Tor is free to ignore them.

The tor-http package (or other name) contains Shim and ALSO phrases torrc. tor-http forwards the traffic to SocksPort 9050 (or different, configured in torrc). Each HttpPort will start a Shim instance. Each Shim instance will use a different SOCKS username for stream isolation.

comment:30 Changed 5 years ago by rransom

Resolution: wontfix
Status: assignedclosed

comment:31 in reply to:  28 Changed 5 years ago by ioerror

Resolution: wontfix
Status: closedreopened

Replying to nickm:

Replying to ioerror:

Replying to nickm:

Replying to ioerror:

Replying to nickm:

"Audit shim and bring it up-to-date" is a reasonable thing to do.

I'm not sure what it needs - it compiles without warnings (yay) and it seems to function just as it should. It looks "finished" in as much as any C program. :)

It does need compiler hardening and all that stuff added, of course.

What it needs IMO is auditing for security and standards compliance.

Ok - I think both of those are reasonable. What is the absolute canonical git repo? Is it yours?

There is none; mine is the closest there is.

Ok, I've forked it and I'll take it under my wing.

If so, I'd be willing to perform both of those audits. I would like some guidance of which standards matter to us and what specific security issues that concern us.

Well, if it's going to claim to be an HTTP proxy, it should implement some version of HTTP. Probably 1.1?

Sure, I think that is reasonable - I'm not sure that it makes sense to support _all_ of HTTP 1.1 if for example there is a subset that is commonly accepted for a non-caching proxy to implement, I'd shoot for that goal.

I don't know which specific security issues most affect http proxies.

Ok.

[...]

Any requirements of such a thing - regardless of where we put it - I'm open to considering and trying to resolve.

Somebody would need to take on the responsibility of being shim maintainer. I don't know that shipping shim by default would make senese.

The open question for me is - "what would it take to make an HTTP proxy port a Tor configuration line as we have with SOCKSPort?"

For me, that's not a goal. Tor is not an all-singing all-dancing all-purpose application launcher, nor do I want to push more code into the main Tor process. I'd like us to move in the direction of moving functionality out of Tor.

Ok, I think our goals aren't so different here. I don't want a full HTTP proxy with caching - I want the most minimal thing that will help reduce harm for our users. I think there is a balance to be struck and that is what happened with DNSPort - it is a minimal thing that at least gives some of the features that our users need. It has been extremely handy, even if imperfect or limited; it isn't standards compliant but holy cow, it is useful!

I am hoping to solve this with a clean design in #6948 - so I totally hear you. I'm also in favor of that as a reality, sooner rather than later. If we can solve #6948, I would say we could break out each of these things quite nicely and move more and more code out of Tor proper. I am totally a fan of that while also being concerned that we may not succeed anytime soon.

Is *that* what #6948 was about? I have no idea what a zygote is, or why shared memory mutexes were something we needed, so I kinda assumed it was an 'implementation technique' ticket, not a 'better architecture' ticket.

Ah - there has been a long standing discussion with Mike about this topic. I think at this point I owe a large set of tickets to trac that outline the design I've drawn up. It has been in the queue, so I'll get it done sometime soon.

[...]

Ok. That settles it, I guess. If both options are rejected, even as a thread that doesn't loop to a SocksPort, I'll continue with designs in #6948.

Well, nothing is settled -- I can be wrong, and I hope I can change my mind if I am.

I understand. I think that the key is the energy to be spent with changing your mind is a lot more than if you found the idea compelling. So in that sense, I feel like it is easier to simply spent that energy on this zygote design.

It seems rather straight forward to me to provide the most minimal HTTP interface possible so as to reduce harm; that didn't convince you and largely because I offered shim as a thing to provide it.

Shim is already a program, so that means it sounds like Tor is a launcher or some such thing. I understand why you wouldn't want Tor to just include another full program. Though I also think that could be fine, I don't think it is important enough to discuss further. I do think a thread that opens an HTTP port is that important but again, as it is a pretty serious uphill battle, I feel demotivated.

But I don't think I'm wrong here. HTTP is a very complex protocol, and I think about 100% of what you'd need to do with an HTTP proxy can be done out-of-process from Tor, as I understand it.

I think a port that provides support for CONNECT is probably not too complicated and that is why I mentioned the DNSPort - we don't have to do everything but if we do something, we can do a lot of good, where otherwise, people are screwed or software just breaks badly.

If the only argument against a separate proxy is "but then you would have to run two programs", I don't think it's a great one, since having more processes is the direction we should IMO be moving towards, for better security and modularity.

The main argument is one of harm reduction where today, TorBirdy will need to solve this problem, likely by becoming an HTTP proxy. We actually found some code from Moxie that will do the trick (and make us GPL, ironically). If Tor provided that, we'd just depend on Tor and we'd call it a day. That is how a JonDos user is kept safe.

As an aside and part of why I'm working on the zygote is that TorBirdy will also likely have to include a Tor and be a Tor controller. That also means that if the user ever later installs Tor, they'll have two Tor clients on a given machine. We'll try to be smart but boy, we really do need to find a better solution. In theory, we can have TorBirdy users run privoxy. In practice, other than Tails users and very advanced users, no one does it - so most users are in harms way, especially if they use GnuPG with TorBirdy (and SOCKS) or any other HTTP proxy only app.

So the general issue for me is one of overall integration and the specific is one of proving a kind of interface, slightly different from what we already provide, because it would reduce harm. How we reduce that harm only matters in that it _must_ be zero config and not fail open.

comment:32 in reply to:  29 Changed 5 years ago by ioerror

Replying to proper:

Since Nick unfortunately rejected (or not?) being the http proxy directly inside Tor AND managing Shim with HttpPort... Here is my proposal...

Well, we should probably back off from mentioning Shim. We don't need shim per se. I believe we'd just need support for CONNECT and a few other things. I believe RFC2616 and RFC3143 are relevant here.

Who says torc may only be phrased by Tor? Users can still add HttPort's to torrc. Tor is free to ignore them.

Indeed - we can do this today with comments. eg:

#shimListenPort 8080

That is however totally horrible. We should not take that path for the love of anonymity.

The tor-http package (or other name) contains Shim and ALSO phrases torrc. tor-http forwards the traffic to SocksPort 9050 (or different, configured in torrc). Each HttpPort will start a Shim instance. Each Shim instance will use a different SOCKS username for stream isolation.

We could do such a thing if this really is a dead issue. I think we should not consider that unless we're really hard up. I'd much prefer the tackle this with a zygote approach.

comment:33 Changed 5 years ago by proper

What if I can expand my proposal, what if we can use shimListenPort 8080 without #?

If all you dislike is the # in front of shimListenPort, I think Nick could be fine with ignoring that line in Tor?

*Some* support on the Tor side would be needed. That and adding ## comments to torrc about the tor-http package and shimListenPort comment should be possible?

comment:34 Changed 5 years ago by T(A)ILS developers

Cc: tails@… added

comment:35 Changed 5 years ago by nickm

I'm afraid I don't understand why having a single configuration file here would actually be any easier than having a separate configuration file. It's a clever idea, but what good is it? It's not like you only have room on your HD for a small number of files, or like microsoft charges you $10 for every configuration file you have.

comment:36 Changed 5 years ago by nickm

Keywords: tor-client added

comment:37 Changed 5 years ago by nickm

Component: Tor ClientTor

comment:38 in reply to:  35 Changed 5 years ago by proper

Replying to nickm:

I'm afraid I don't understand why having a single configuration file here would actually be any easier than having a separate configuration file. It's a clever idea, but what good is it?

It's nicer to have everything related to Tor in one configuration file. When you read through the configuration file you learn about anything and don't need to look for additional hacks. One can already add SocksPort, DnsPort and TransPort to torrc. Adding the last missing piece there, HttpPort, seems natural to me. I prefer to have some officially supported Tor package including HttpPort over all the hackish TorifyHOWTO workarounds.

comment:39 Changed 5 years ago by fk

It has been brought to my attention that somebody is wrong on the Internet and I'm here to help.

Privoxy has been able to leverage Tor's stream separation by using separate socks ports for years. It's commonly done by request tagging, I rely on it every day to separate Firefox, fetch, curl and various other clients I use and it is working (for me) as expected:
http://config.privoxy.org/user-manual/actions-file.html#CLIENT-HEADER-TAGGER

While there is currently no IsolateSOCKSAuth support, that's mainly because nobody cared enough to provide patches as stream separation is already available through other means:
http://sourceforge.net/tracker/?func=detail&aid=3541363&group_id=11118&atid=361118

Even if this wasn't the case, I don't think the paraphrased argument "Privoxy and Polipo suck, therefore Tor needs to act as HTTP proxy" is a particular good one.

If the Tor project really wants to reverse the current policy and ship a HTTP proxy with Tor again, I think it would be worth trying to communicate with the various HTTP proxy upstreams about what the requirements are, to see if they can be satisfied already or maybe in a future release after adding the missing pieces (potentially by accepting the required patches).

It's water under the bridge, but in my opinion the Tor project has a pretty poor track record when it comes to working with HTTP proxy upstreams and especially the recent number it did on Juliusz probably didn't help to instill goodwill in general.

As far as native HTTP proxy support in Tor is concerned, I agree with Jake that a HttpPort implementation that only supports CONNECT requests would be comparably trivial to implement, but like to point out that most HTTP clients have no support for tunneling http:// requests through HTTP proxies by using CONNECT. And those that do probably already support socks5 anyway.

One of the obvious risks of implementing support for HTTP methods other than CONNECT without constantly monitoring and parsing the arriving data (which requires implementing a significant amount of RFC2616) would be requests on a reused connection ending up at the wrong server.

Nowadays some clients aggressively pipeline requests right away instead of waiting for the first response as "suggested" in the standard and in this case the old "trick" of changing the Connection header in the first arriving request to "Connection: close" and trusting without properly verifying that client and server respect it no longer cuts it.

Full disclosure: it's "fk" as in "Fabian Keil" and I'm obviously biased.

comment:40 Changed 5 years ago by proper

It has been brought to my attention that somebody is wrong on the Internet and I'm here to help.

:)

If the Tor project really wants to reverse the current policy and ship a HTTP proxy with Tor again, I think it would be worth trying to communicate with the various HTTP proxy upstreams about what the requirements are, to see if they can be satisfied already or maybe in a future release after adding the missing pieces (potentially by accepting the required patches).

Sounds good.

http://config.privoxy.org/user-manual/actions-file.html#CLIENT-HEADER-TAGGER

As discussed here it is my opinion that this method is too difficult and unreliable.

If someone else interested in the http to Tor socks (with stream isolation), I'd like to see your opinion.

http://sourceforge.net/tracker/?func=detail&aid=3541363&group_id=11118&atid=361118

This one would be awesome. Would that make a fine http solution or what is required?

comment:41 Changed 3 years ago by intrigeri

Cc: intrigeri added; tails@… removed

comment:42 Changed 6 months ago by nickm

Milestone: Tor: very long termTor: unspecified

Batch modify: these tickets seem to be things that wouldn't actually be a big redesign or a big amount of work, so they belong "Unspecified", not "Very Long Term"

comment:43 Changed 6 months ago by nickm

Keywords: usability http-proxy added
Points: 10
Priority: MediumHigh
Resolution: duplicate
Severity: Normal
Status: reopenedclosed

Update: I think that this ticket has a lot of confusion above: what makes sense IMO is not supporting any kind of http proxy/rewriting/cacheing complexity, but instead supporting "HTTP CONNECT" as an alternative to SOCKS. I'm opening 22407 as tracking HTTP CONNECT only, and closing this as duplicate.

Note: See TracTickets for help on using tickets.