Opened 3 years ago

Last modified 4 weeks ago

#16824 needs_revision defect

Emit a warning message about side channel leaks when using relays as clients

Reported by: starlight Owned by:
Priority: High Milestone: Tor: 0.3.6.x-final
Component: Core Tor/Tor Version: Tor: 0.2.6.10
Severity: Normal Keywords: mike-can, tor-client, tor-relay, tor-log, sidechannel, easy, 035-triaged-in-20180711
Cc: arma, dmr Actual Points:
Parent ID: Points: 1
Reviewer: mikeperry Sponsor:

Description

Analysis presented in bug #16585 demonstrates client circuit formation processing perturbs relay cell forwarding in a manner that may be susceptible to traffic confirmation analysis.

With the present code structure it is recommended that simultaneous client and relay operation be aggressively discouraged with a new torrc configuration parameter to inhibit it--default value set to prevent. In addition log warnings should be generated when both client and relay functions are allowed to operate concurrently.

Correct support of simultaneous client and relay function requires segregation of the client function to a separate thread running on a different processor core than the relay function.

Correcting the current client implementation's deficit of transaction granularity is unlikely to eliminate the potential for a sophisticated advisory to detect perturbation of cell forwarding by client circuit creation activity.

Child Tickets

Change History (55)

comment:1 Changed 3 years ago by starlight

Version: Tor: 0.2.6.10

comment:2 Changed 3 years ago by nickm

Cc: arma added
Milestone: Tor: 0.2.7.x-final

Adding a warning at least would be easy for 0.2.7; we should look into that. Roger, thoughts?

comment:3 Changed 3 years ago by s7r

@starlight do you have any suggestions for a fix? Shipping Tor into separate packages as mentioned on the mail list(tor-core; tor-relay and tor-client) is not a good option. A warning message is trivial to add but a patch is even better.

Quoting you:
Correct support of simultaneous client and relay function requires segregation of the client function to a separate thread running on a different processor core than the relay function.

If we would just add a separate worker to the same daemon (same torrc file) to handle the client function, wouldn't that be a fix? Don't know in what measure this is possible, but the more we automate it the better.

comment:4 Changed 3 years ago by starlight

Indeed, a separate client-worker thread that passes
cells to and from the relay worker thread appears an
ideal solution. My other recommendations are premised
on the assumption that this would require significant
work and mitigating confirmation-analysis exposure for
users in the near term is important.

If the above is implemented, a warning should be
logged if fewer than two physical processors are
available to the tor process since running both
client and relay threads on a single core would
cause client activity to impact relay forwarding.
Two real physical cores should be the minimum,
as two SMT threads (aka Hyperthreads) on a single
physical core would not truly isolate the relay
and client.

During implementation, some attention should be
paid as to whether the client/relay interaction
is in any way detectible through statistical
analysis of aggregate relay traffic.

comment:5 Changed 3 years ago by nickm

Keywords: PostFreeze027 added

I'd merge patches for these for 0.2.7 if they come in on time. In some cases, that will require figuring out an as-yet-unsolved bugs.

comment:6 Changed 3 years ago by mikeperry

I'm not sure that a warning here is going to make a lot of sense, or be very understandable. Certainly circuit formation and client usage is even more visible when there is not relay traffic involved, right? You just watch the packets. They're all client usage in that case.. Unless there is something I'm missing?

comment:7 in reply to:  6 Changed 3 years ago by starlight

Replying to mikeperry:

circuit formation and client usage is even more visible when there is not relay traffic involved

Agreed, no doubt w/r/t pure-client operation.

I have two thoughts:

1) in general, if two configurations or implementations
exist, x1 and x2 where x2 is more anonymous than x1,
one should prefer x2

2) some consider it a reasonable idea to configure a client
and relay in the same daemon instance with the belief
that this would obfuscate local client traffic to some
degree; but with the implementation as it presently
stands such an idea is false and should be denigrated

comment:8 Changed 3 years ago by s7r

@mikeperry is right. Regardless if the same Tor instance runs a client only or a client and a relay, an active observer can distinguish the client traffic mixed with the relayed traffic using the timing method described by you. But, running a relay and a client (even on the same instance) provides you security against other attacks, which are not active, so this x2 option (be a client and a relay, even on the same Tor instance) is at least equally anonymous, if not (as I see it is) more anonymous.

Adding a warning here will confuse users. How can we explain this well enough in a single log line? Saying that it's not safe would be false, saying it is less anonymous would be false, saying the client traffic is compromised would be false (the traffic is not compromised, just worst case distinguished from the relayed traffic, running the client in a separate instance would not mitigate this).

comment:9 Changed 3 years ago by mikeperry

While serious, the side channel in #16585 doesn't fully differentiate all client and relay traffic. It only lets you know that a client circuit setup is happening, not if or when an existing circuit is being used for client traffic. As far as I can tell, it also doesn't directly disclose the volume of client traffic relative to relay traffic, either. Nor does it expose when client circuits are actually closed.

This means that significantly less information is available to an adversary who is monitoring a relay (that is also used as a client) than is available to an adversary who is watching a machine that has a separate relay tor instance and a client tor instance, even with the side channel.

I do believe that #16585 is serious and should be fixed, especially since it seems like it could also be a vector for other side channels as well, and potentially even in client-only scenarios. However, shouting confusing, nuanced, and/or partially correct information at users in our loglines isn't the right stopgap in the meantime, IMO.

comment:10 Changed 3 years ago by starlight

I accept the argument. Don't know the code and acknowledge Mike's analysis. Many users are not technical and so best to not alarm or confuse anyone unnecessarily, especially if issue will be mitigated in the not-too-distant future. For anyone technical enough to be concerned, much information can be found in #16585. Have no objection to this ticket being closed "wontfix" or "not a bug".

comment:11 Changed 3 years ago by someone_else

This may also allow some new and cheap variation of the the relay overload attacks and enable attacks on the relay traffic, not just the client traffic.

comment:12 Changed 3 years ago by nickm

Priority: normalmajor

comment:13 Changed 3 years ago by nickm

Keywords: 027-backport added
Milestone: Tor: 0.2.7.x-finalTor: 0.2.8.x-final

comment:14 Changed 3 years ago by mikeperry

Severity: Normal

Ok, so I've thought more about this, and if #16585 remains unsolved/unmitigated, a potential workaround for technical users is to use their relay as a bridge for a second tor client process that runs locally. This will prevent the side channel from happening.

Unfortunately, to avoid exposing yourself to additional attacks, you probably want to hack that second tor client to use 4 hops (so your local relay-as-bridge, and 3 hops after that). You also want a second layer guard to be used by your second Tor client after your relay-as-bridge, so that Guard discovery attacks find that guard and not the relay that you run.

Since this is two paragraphs of text, unless we create a torrc options to do the above, we're still probably blocked here. But I do think this is at least a way forward. Unfortunately, the Guard-after-bridge piece is a very involved change because of how that code works right now. Simply changing to 4 hops for that second Tor client is trivial, though.

comment:15 Changed 3 years ago by starlight

What Guard discovery attacks exist that succeed where the adversary does not have access to the path between the client and the Guard? In most cases the 4-hop client relay would reside inside a private network (or even in the TCP stack of a single server) where accessing client-guard traffic is far more difficult than on the open Internet.

comment:16 Changed 3 years ago by mikeperry

First off, guard discovery attacks are far more an issue for a hidden service than a normal client. In general, guard discovery works by causing the tor instance to create lots of new circuits, and waiting until a malicious middle node is chosen. That malicious middle then gets to learn the guard node next to it. This is also called the 'predecessor attack' when guards are not involved.

I think a normal client probably could get away with just using 3 hops as-is in this local-relay-as-bridge configuration, and not modify their second tor client instance at all, because the traffic volumes are should be much smaller and not directly under adversary control. Also, for many client application protocols, there's not an easy way to cause the client to keep building circuits.

The reason I think it's a bad idea to run a hidden service in this configuration without an actual guard after the local-relay-as-bridge is because I suspect that the combination of guard discovery and the ability for the adversary to generate a lot of traffic to a hidden service means that it will be obvious to someone who is monitoring your guard that it is both the relay and the client: They can perform guard discovery, find your relay, and then send a bunch of traffic at the hidden service and not see that traffic volume leave the guard to any currently connected remote clients. If instead, your hidden service is using an additional guard after your relay, you benefit from the additional relay cover traffic between those two nodes. I could explain this more formally, but it basically all boils down to the base rate of background traffic that the adversary needs to consider in each situation.

However, I could see an argument that removing the side channel that client activity is happening at all on a given relay is more important than these second-order guard discovery concerns. After all, the adversary does need two attacks to determine the existence of a client on a relay for the 'local bridge' configuration, where as they only need passive monitoring to determine if relays are running a hidden service in the standard configuration (via bug #16585 - nice find btw).

Therefore, I could personally be convinced that a simple log message telling users that they may want to run a second tor instance and use the relay as a local bridge is a good enough stopgap. I don't know how Nick and Roger feel, though, and they are both highly distracted by ED responsibilities.

Last edited 3 years ago by mikeperry (previous) (diff)

comment:17 Changed 3 years ago by mikeperry

Summary: coexistence of client and relay processing on same thread poses traffic confirmation riskEmit a warning message about side channel leaks when using relays as clients

comment:18 Changed 3 years ago by mikeperry

starlight - do you mind if I clean up your formatting in your earlier comments of this bug and #16585? I think the poor formatting is making people tune out before they realize the true severity of this bug and #16585.

comment:19 Changed 3 years ago by starlight

Please go ahead and edit as you see fit.

A log warning satisfies my concerns.

comment:20 in reply to:  19 Changed 3 years ago by mikeperry

Replying to starlight:

Please go ahead and edit as you see fit.

A log warning satisfies my concerns.

Ok. So there's a specific action item here, the log message I am thinking about would read something like:
"You appear to be attempting to use Tor for both relay and client functionality at the same time. Unfortunately, a side channel issue (Bug #16585) means that this fact will be visible in network traffic patterns. If this is a problem for you, we recommend running a second local Tor client instance and configuring it to use this relay as a local bridge. Your bridge line for this second instance would be: Bridge 127.0.0.1:port $IDENTITY_FP"

If the user doesn't have their dirport set, we'll also need to tell them to do that until #12538 is solved, or they won't be able to use their tor relay as a bridge.

comment:21 Changed 3 years ago by starlight

Message text is good IMO.

Is there any particular reason why it's better to configure the client-only relay in bridge mode rather than configure the local-public-relay as normal Guard?

comment:22 Changed 3 years ago by mikeperry

Keywords: mike-can added

comment:23 in reply to:  21 Changed 3 years ago by mikeperry

Replying to starlight:

Message text is good IMO.

Is there any particular reason why it's better to configure the client-only relay in bridge mode rather than configure the local-public-relay as normal Guard?

Because if you set the relay a Bridge line, Tor's bootstrapping code uses it as the sole source of directory information, and never tries to touch the directory authorities. If you set it as Guard, however, your second Tor instance will still bootstrap directly from the dirauths. Concealing the additional directory authority hits are crucial, since the goal is to conceal client activity on the relay itself.

comment:24 Changed 3 years ago by nickm

Milestone: Tor: 0.2.8.x-finalTor: 0.2.9.x-final

It is impossible that we will fix all 226 currently open 028 tickets before 028 releases. Time to move some out. This is my second pass through the "new" and tickets, looking for things to move to 0.2.9.

comment:25 Changed 3 years ago by nickm

Points: small

comment:26 Changed 2 years ago by isabela

Points: small1

comment:27 Changed 2 years ago by isabela

Keywords: isaremoved added
Milestone: Tor: 0.2.9.x-finalTor: 0.2.???

comment:28 Changed 2 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:29 Changed 22 months ago by nickm

Keywords: tor-03-unspecified-201612 added
Milestone: Tor: 0.3.???Tor: unspecified

Finally admitting that 0.3.??? was a euphemism for Tor: unspecified all along.

comment:30 Changed 17 months ago by nickm

Keywords: tor-03-unspecified-201612 removed

Remove an old triaging keyword.

comment:31 Changed 17 months ago by nickm

Keywords: isaremoved removed

comment:32 Changed 17 months ago by nickm

Keywords: 027-backport removed

These are not ripe for an 027 backport

comment:33 Changed 17 months ago by nickm

Keywords: PostFreeze027 removed

comment:34 Changed 16 months ago by nickm

Keywords: tor-client tor-relay sidechannel logging easy added

comment:35 Changed 5 months ago by n-critser

Status: newneeds_review

Based on previous comments above and in IRC, the scope of this change was limited to adding a warning message for users who are running both in Relay and Client modes. The exact wording of the new message is as follows :
"You are currently running as a server, and a client.

If you do not actually want to run Tor as a client,
you can add SocksPort 0 to your configuration, to turn off
the default client behavior."

PR # https://github.com/torproject/tor/pull/114

comment:36 Changed 5 months ago by starlight

The warning is too mild. Clear indication of traffic analysis vulnerability should be present.

comment:37 Changed 5 months ago by nickm

Milestone: Tor: unspecifiedTor: 0.3.5.x-final

comment:38 Changed 5 months ago by n-critser

In that case I can add the beginning section of the text recommended by mikeperry:
"You appear to be attempting to use Tor for both relay and client functionality at the same time. Unfortunately, a side channel issue (Bug #16585) means that this fact will be visible in network traffic patterns. "

comment:39 Changed 5 months ago by starlight

Yes, that does it nicely. Thx.

comment:40 Changed 5 months ago by n-critser

Warning has be updated to :

log_warn(LD_CONFIG, "You appear to be attempting to use Tor for both "

"relay and client functionality at the same time. "
"Unfortunately, a side channel issue (Bug #16585) means that "
"this fact will be visible in network traffic patterns. "
"If you do not actually want to run Tor as a client, "
"you can add SocksPort 0 to your configuration, to turn off "
"the default client behavior.");

comment:41 Changed 5 months ago by arma

Two thoughts:

(A) We're only doing this log message if they *use* the socksport, right? Otherwise every relay will get this log message by default, unless they change their SocksPort line, even if they *aren't* "attempting to use Tor for both relay and client functionality"?

(B) I'm still unclear on what attack we're talking about here. In the earlier bug, #16585, there was some bug where relay cells would get unnecessarily starved when the client goes wild trying to build circuits that all fail? So in that case maybe it's the relay bandwidth *graphs* that give things away? But if there weren't graphs, you might still be able to send a constant rate of traffic through the relay and see when it slows down? But even if it's not a relay at all, and even if we fixed the starvation bug, you might be able to send ping packets toward the suspect's IP address, and see a slow-down when there's competing client activity?
https://www.freehaven.net/anonbib/#remote-traffic-pets12
So it seems weird to me to have a warning message on one particular situation -- and a situation that seems more of a bug we can actually fix -- but not in all the other situations that can be bad too.

comment:42 Changed 5 months ago by starlight

Seemed to me a warning would arrive once client activity commenced on a traffic forwarding relay. Had not considered how it would be implemented, whether SocksPort!=0 and ORPort!=NULL would trigger it. Perhaps the message should emit on the first socks connection when ORPort is configured? Or perhaps SockPort=0 should default when ORPort is set and the message arrive when both are asserted?

To quote my earlier self:

2) some consider it a reasonable idea to configure a client
and relay in the same daemon instance with the belief
that this would obfuscate local client traffic to some
degree; but with the implementation as it presently
stands such an idea is false and should be denigrated

The idea of the warning is to alert users to potential risk, in consideration of the time and effort that will likely pass before the risk is alleviated. Already quite some time has passed.

Mike Perry suggested a warning as an alternative to my original idea that such configurations be discouraged via a new parameter, his reasoning in comment:16 above.

Last edited 5 months ago by starlight (previous) (diff)

comment:43 in reply to:  42 Changed 5 months ago by starlight

replied when edit intended -- a peril of white-on-black color schemes in a black-on-white world

Last edited 5 months ago by starlight (previous) (diff)

comment:44 Changed 5 months ago by n-critser

Arma,Starlight,
Unless we have specific examples of traffic analysis in this case, which it seems like we do not. Wouldn't it be better here to warn the user that running both Client and Relay mode simultaneously can result in starvation of the Relay. And then focus on fixing the starvation problem.

Since this issue has been around for over 3 years a warning message could serve to both mitigate further users from starving their Relays, while also being a constant reminder to ourselves and the community at large that we need to fix this issue.

comment:45 Changed 5 months ago by starlight

As long as the message includes the (Bug #16585) reference I'm ok with it.

comment:46 Changed 5 months ago by starlight

Possibly logic from #26062 can be used to decide when to issue a warning.

comment:47 Changed 5 months ago by asn

Reviewer: mikeperry

comment:48 Changed 5 months ago by mikeperry

Status: needs_reviewneeds_revision

Initial thoughts: I don't like issuing a warning about merely having the SocksPort set. It is set by default. I also don't think telling relay operators to set it to 0 solves any problems here. In fact, all it does is make those people who deliberately set it more obvious.

I don't really care about the case where the relay stalls briefly merely because it happens to build some testing/predicted circuits for a little while. In the grand scheme of things that cause Tor's performance to be poor, this side channel by itself is very very low on the list (though it is a result of our single-threaded architecture, which *is* a huge performance problem).

However, I do care about the case where people are actively using their relay as a client. This indicates that they are trying to get some benefit from having relay traffic mixed with client traffic, and in the process, they are making it clear to external observers that their relay is doing client things (due to stalls of all traffic during client circuit construction from #16585).

I still think that what those people actually want to do is use their Tor relay as a local bridge, and pin a guard node after it. Unfortunately, having a guard after a bridge is not something we support yet for general use (iirc, isis was working on a bridgeguards patch to prevent bridge enumeration by a censor, but I don't know if that got finished). We can have a guard/guards after this bridge for hidden services via the HSLayer2Nodes -- in that cause your layer2 guards become like entry guards, and your layer3 guards are functionally layer2 guards. But then, you would be a client that looks like they are using layer2 vanguards but not layer3 vanguars, which may be odd or rare. I have to think more about this to decide if that actually matters/is detectable. And then, even in this case, it is only something that relay operators who are running hidden services can do.

Anyway as-is this is needs revision. We should discuss what we actually think people who want do this should be doing, and why, rather than just yelling at everybody who uses the default SocksPort in their relay.

comment:49 Changed 5 months ago by mikeperry

Additional point: people who set up a hidden service on their relay's Tor daemon can set their SocksPort to 0 and yet still succumb to the side channel issues of #16585. So we definitely can't even base this on SocksPort usage.

comment:50 Changed 4 months ago by n-critser

mikeperry,
What do you think the best way to come to a consensus on this would be? Seems like we have 2 options available for deciding if a user is both a relay and a client:

  1. configuration
  2. execution of code

If using only the SOCKSPORT catches too many false positives then, I could check some of the other existing values to ensure that the user is definitely configured to be both.
Otherwise, I would need to find some other solid indicator functions for both and base the warning on those.
Which do you prefer?

comment:51 Changed 3 months ago by nickm

Keywords: 035-triaged-in-20180711 added

comment:52 Changed 3 months ago by dmr

Cc: dmr added
Keywords: tor-log added; logging removed

tor-log is a relatively new keyword; using instead of logging.

comment:53 Changed 2 months ago by mikeperry

Wait don't we already do this? #12908

Why is that warn not being emitted?

Last edited 2 months ago by mikeperry (previous) (diff)

comment:54 in reply to:  53 Changed 2 months ago by arma

Replying to mikeperry:

Wait don't we already do this? #12908

Why is that warn not being emitted?

That ticket is about relay + onion service, not relay + client.

comment:55 Changed 4 weeks ago by nickm

Milestone: Tor: 0.3.5.x-finalTor: 0.3.6.x-final

Deferring various feature-y things to 0.3.6. If one of these is actually happening in 0.3.5, please let me know!

Note: See TracTickets for help on using tickets.