Opened 8 months ago

Closed 7 months ago

#33519 closed defect (fixed)

Support multiple simultaneous SOCKS connections

Reported by: dcf Owned by:
Priority: Medium Milestone:
Component: Circumvention/Snowflake Version:
Severity: Normal Keywords: turbotunnel
Cc: arma, cohosh, phw, arlolra, dcf Actual Points:
Parent ID: #19001 Points:
Reviewer: Sponsor:

Description

The Snowflake client accepts multiple simultaneous SOCKS connections from tor, but it only tries to collect one proxy at a time, and each proxy can service only one SOCKS connection (this is true in the turbotunnel branch as well). One of the SOCKS connections gets the only available proxy, while the others starve.

I can think of a few ways to approach this.

  1. Dynamically adjust the max parameter according to how many SOCKS connections there are currently. If there's one SOCKS connection, we need only one proxy. If there's another SOCKS connection, raise the limit to allow the proxy-collecting thread to pick up another one, and lower the limit again if the number of SOCKS connections drops back down.
  2. Start up a separate proxy-collecting thread for each SOCKS connection, as suggested at comment:12:ticket:21314. Each SOCKS connection will make its own broker requests and collect its own proxies, not interacting with those of any other SOCKS connection. A downside of this is that the number of Snowflake proxies you are contacting leaks the number of SOCKS connections you have ongoing. (Which can also be seen as a benefit in that if there are zero SOCKS connections, you don't even bother to contact the broker.)
  3. Make it possible for multiple SOCKS connections to share the same proxy. Continue using a global proxy-collecting thread, and make there be a single shared RedialPacketConn instead of a separate one for each SOCKS connection. As things work now, this would require tagging every packet with the ClientID, instead of sending the ClientID once and letting it be the same implicitly for all packets that follow.
  4. Make it possible for multiple SOCKS connections to share the same proxy, and use a single KCP/QUIC connection for all SOCKS connections. Separate SOCKS connections go into separate streams within the KCP/QUIC connection. In other words, rather than doing both sess = kcp.NewConn2/quic.Dial and sess.OpenStream in the SOCKS handler, we do sess = kcp.NewConn2/quic.Dial in main and then sess.OpenStream in the SOCKS handler. This way we could continue tagging the ClientID just once, because the program would only ever work with one ClientID at a time. However this way would make it harder to do the "stop using the network when not being used" of #21314, because that single KCP/QUIC connection would try to keep itself alive all the time and would contact the broker every time it needed a new proxy. Perhaps we could make it so that if there are zero streams, we close the KCP/QUIC connection, and lazily create a new one if and when we get another SOCKS connection.
status quo 1 2 3 4
proxy-collecting threads one global one global one per SOCKS one global one global
proxy limit per thread 1 # of SOCKS 1 1 1
proxies shared between SOCKSes? dedicated dedicated dedicated shared shared
PacketConns one per SOCKS one per SOCKS one per SOCKS one global one global
KCP/QUIC connections one per SOCKS one per SOCKS one per SOCKS one per SOCKS one global
KCP/QUIC streams one per SOCKS one per SOCKS one per SOCKS one per SOCKS one per SOCKS
ClientID on every packet? no no no yes no

Child Tickets

Change History (8)

comment:1 Changed 8 months ago by dcf

Summary: Support multiple simultaneous SOCKS proxiesSupport multiple simultaneous SOCKS connections

comment:2 Changed 7 months ago by arma

I'm a big fan of fixing this one. I just hit it again in my Tor Browser + snowflake (making my Tor Browser unable to connect to anything), and I'm going to move back to normal Tor Browser until it's fixed and I have a new container of dogfood to try. :)

I am cool with whichever architecture y'all pick. If you need more chefs, I encourage you to ask ahf for advice.

comment:3 Changed 7 months ago by dcf

Here's a candidate patch, for the QUIC branch at least.
https://gitweb.torproject.org/user/dcf/snowflake.git/commit/?h=turbotunnel-quic&id=c07666c1408bcf1204f3318aa67c7623e16188db
It's basically option (3) from the ticket description. It sets up one PacketConn in main, and starts a new QUIC connection over it for each new SOCKS connection. It turns out that it doesn't even require the overhead of attaching a ClientID to every packet. We can just prefix each WebRTC connection with the ClientID as before, and the QUIC connection ID takes care of disambiguating the multiple virtual connections.

The easiest way to test it is to start two tor clients, one that manages a snowflake-client, and one that uses the same snowflake-client as the first one. Edit client/snowflake.go and make it listen on a static port:

ln, err := pt.ListenSocks("tcp", "127.0.0.1:5555")

Create files torrc.1 and torrc.2:

UseBridges 1
DataDirectory datadir.1
SOCKSPort 8001
ClientTransportPlugin snowflake exec client/client -url http://127.0.0.1:8000/ -ice stun:stun.l.google.com:19302 -log snowflake.log -max 1
Bridge snowflake 0.0.3.0:1
UseBridges 1
DataDirectory datadir.2
SOCKSPort 8002
ClientTransportPlugin snowflake socks5 127.0.0.1:5555
Bridge snowflake 0.0.3.0:1

Fire everything up:

broker/broker --disable-tls --addr 127.0.0.1:8000
proxy-go/proxy-go -broker http://127.0.0.1:8000/
tor -f torrc.1
tor -f torrc.2

If you run this test before the changes I'm talking about, the second tor will be starved of a proxy. If you run it with the changes, both tors will share one proxy.


Unfortunately the same idea doesn't carry over directly into the KCP branch. There are at least two impediments:

  • https://github.com/xtaci/kcp-go/issues/165
    kcp-go assumes that you will use a PacketConn for at most one KCP connection: it closes the underlying PacketConn when the KCP connection is closed, so you can't reuse for more connections. This one is easy to work around.
  • https://github.com/xtaci/kcp-go/issues/166
    The kcp-go server only supports one KCP connection per client address (which would ordinarily be an IP:port but in our case is a 64-bit ClientID). This one requires a change to kcp-go to fix.

The alternative for KCP is option (4): use one global PacketConn and one global KCP connection. Each new SOCKS connection gets a new smux stream on the same KCP connection. I was reluctant to do this in the QUIC branch because the quic-go connection type (quic.Session) is a stateful, failure-prone entity. It has its own timeout and can fail (conceptually) at any time, and if it's a global object, what then? The analogous type in kcp-go, kcp.UDPSession, is simpler, but I think we will cannot guarantee a single one will survive for the lifetime of the process. So we'd have to introduce an abstraction to manage the global shared kcp.UDPSession and restart it if it dies.

Last edited 7 months ago by dcf (previous) (diff)

comment:4 Changed 7 months ago by dcf

Conversation on the kcp-go issue tracker convinced me that what I was doing in comment:3 is the wrong approach. Two sessions sharing the same PacketConn will each be reading packets intended for the other, resulting in effectively 50% packet loss. (I think, unless quic-go has some special code to handle this case.) Retransmissions will probably make the connection work, but obviously performance will be bad.

Instead, I have new commits that implement option 4 for both the KCP and QUIC branch.

turbotunnel-kcp
turbotunnel-quic

The commits introduce a sessionManager object that creates a PacketConn and session on demand, and demand, and thereafter reuses the same PacketConn and session. If the session ever dies (which should be an unusual case), it tears down the PacketConn and allows a new PacketConn and session to be created on demand the next time they are needed. quic-go provides a nice channel that we can read to find out when the session dies; with kcp-go we poll the IsClosed method periodically.

You'll be able to test the commits using the same procedure as in comment:3.

I've started Tor Browser builds based on 9.5a8.

comment:5 Changed 7 months ago by cohosh

Parent ID: #19001

comment:6 Changed 7 months ago by dcf

They are built from snowflake-turbotunnel-kcp and snowflake-turbotunnel-quic respectively.

This time, I removed the elaborate date-based update logic and just set app.update.auto=false. Updates will be downloaded but not automatically installed. Just keep clicking "Not Yet" to stay on the turbotunnel build.

comment:7 in reply to:  6 ; Changed 7 months ago by arma

Replying to dcf:

Good stuff. I've been using this build for the past few days, and it's still working. I tried to trip it up a little bit and it recovered after a while. So far so good. I'll plan to challenge it more, but first:

A debugging aid that would be helpful for me, which I realized while staring at all these lines:

2020/03/23 08:12:40 Traffic Bytes (in|out): 40 | 0 -- (1 OnMessages, 0 Sends)
2020/03/23 08:12:42 Traffic Bytes (in|out): 0 | 575 -- (0 OnMessages, 1 Sends)

If these log lines could tell me which outgoing connection (aka which incoming socks connection) these messages were for (even just saying a SocksId number or something), then when I'm watching to see how it recovers, it will be easier to tell when snowflake is handling messages for the *old* connection, vs when it is using the new connection.

I think of this because when I failed my outgoing internet for a while and then brought it back, and then Tor did the

Mar 20 09:19:49.331 [notice] Our circuit 3155147085 (id: 27) failed to get a response from the first hop (0.0.3.0:1). I'm going to try to rotate to a better connection.

reaction, I got the impression that a lot of snowflake's alleged sending and receiving had to do with the old connection for a good while after Tor had given up on that conn. (I was loading a cnn.com page, and snowflake sent and received several megabytes and my page still hadn't loaded yet. I began to question whether the bytes I was getting were even for the page I was loading.)

It is also definitely possible that I am misunderstanding what's going on, and the quic messages at that level are e.g. the aggregate for all currently handled connections. If that's the case maybe I want a breakdown of how many of them are for which connection.

Thanks!

comment:8 in reply to:  7 Changed 7 months ago by dcf

Resolution: fixed
Status: newclosed

Replying to arma:

If these log lines could tell me which outgoing connection (aka which incoming socks connection) these messages were for (even just saying a SocksId number or something), then when I'm watching to see how it recovers, it will be easier to tell when snowflake is handling messages for the *old* connection, vs when it is using the new connection.

Thanks for testing. This suggestion isn't so easy currently, because the bandwidth logger is per snowflake, not per SOCKS connection (per KCP/QUIC stream). But for now I can add stream numbers to the log and log when each stream begins and ends. KCP uses odd integers for stream numbers and QUIC uses even.

---- Handler: begin stream 3 ---
---- Handler: begin stream 5 ---
---- Handler: closed stream 5 ---
---- Handler: closed stream 3 ---
---- Handler: begin stream 0 ---
---- Handler: begin stream 4 ---
---- Handler: closed stream 4 ---
---- Handler: closed stream 0 ---

https://gitweb.torproject.org/user/dcf/snowflake.git/commit/?h=turbotunnel-kcp&id=7e665838032bbd185046997dcb04d3441f6ac3fa
https://gitweb.torproject.org/user/dcf/snowflake.git/commit/?h=turbotunnel-quic&id=4ed4cfcc10110dae79e63e202788a3777553f9d7

Note: See TracTickets for help on using tickets.