Opened 10 months ago

Last modified 7 months ago

#24973 needs_information defect

Tor should be more gentle when launching dozens of circuits at once

Reported by: asn Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version: Tor: unspecified
Severity: Normal Keywords: tor-dos, tor-hs, performance, 034-triage-20180328, 034-removed-20180502
Cc: alecmuffett Actual Points:
Parent ID: #14006 Points: 3
Reviewer: Sponsor:

Description

When starting up Tor as an HS, Tor will create dozens of circs in quick succession (preemptive circs, HSDir circs, intro point circs, CBT circs).

Consider a Tor that starts up with 10 HSes configured(many of those Tors exist); it will easily attempt to make hundreds of circuits within seconds.

It might make sense to have some sort of circuit rate limiting during startup to avoid flooding our guard with so many circuit creations. I notice that Tor will whine about the guard failing circuits during startup very often, and perhaps this initial circuit flood is what's causing it?

We already have MaxClientCircuitsPending but not sure if that's enough. We also used to delay bootup of multiple onion services, but not sure we do this anymore (can't see it in hs_config_service_all()).

Child Tickets

Change History (12)

comment:1 in reply to:  description ; Changed 10 months ago by arma

Replying to asn:

We already have MaxClientCircuitsPending but not sure if that's enough.

Also, MaxClientCircuitsPending only counts CIRCUIT_PURPOSE_C_GENERAL circuits, and Mike's new feature in #23101 has stopped using C_GENERAL circuits for posting or fetching hsdescs.

Also also, my upcoming plans in #24902 will start sending destroys back if you send me too many create cells in too short a time period.

comment:2 in reply to:  description Changed 10 months ago by arma

Replying to asn:

Consider a Tor that starts up with 10 HSes configured(many of those Tors exist); it will easily attempt to make hundreds of circuits within seconds.

You can see one of those Tors in action (probably with more than 10 HSes, but then some of them might be client-side onion fetches too) at #24897:

Jan 20 16:33:30.000 [notice] We'd like to launch a circuit to handle a connection, but we already have 128 general-purpose client circuits pending. Waiting until some finish. [1808313 similar message(s) suppressed in last 600 seconds]
Last edited 10 months ago by arma (previous) (diff)

comment:3 in reply to:  1 ; Changed 10 months ago by arma

Replying to arma:

Also also, my upcoming plans in #24902 will start sending destroys back if you send me too many create cells in too short a time period.

Would it be totally crazy for clients to take a look at the dos_cc_circuit_max_count consensus param (or whatever we end up naming it) from #24902, and try to hold themselves under it when they have some control over their circuit load, like in this case?

comment:4 Changed 10 months ago by alecmuffett

Cc: alecmuffett added

comment:5 in reply to:  3 Changed 10 months ago by dgoulet

Status: newneeds_information

Replying to arma:

Would it be totally crazy for clients to take a look at the dos_cc_circuit_max_count consensus param (or whatever we end up naming it) from #24902, and try to hold themselves under it when they have some control over their circuit load, like in this case?

In general, I do think we want a mechanism in a tor *client* to never go above a circuit creation rate and sounds like a good logic to use the DoS mitigation rate as the upper limit (which can be a moving target over time). That makes client use a rate that they know they won't get blacklisted for that in theory. Although all this makes sense if our DoS mitigation starts being applied with a single TCP connection (DoSCircuitCreationMinConnections 1).

This also has a side effect of effectively having a way to control the circuit creation rate of the entire network (for normal tor clients). Fun power to have (?).

However, that might not play super well with busy hidden service opening rendezvous circuits at a much higher rate than a normal client would do. And also, I wouldn't probably do this differently with HS because it seems to me an easy way to provide a way to neutralize a service, keep opening circuits to reach that limit and the circuit effectively becomes unresponsive because it can't launch more RP circuits. Else, it would queue the requests and go FIFO with those while it respects the circuit rate.

At first, I would go with circuit rate limiting tor clients (excluding HS). Is this something we want/can have in 033?

comment:6 Changed 10 months ago by teor

I think we need to limit clients and HSs, otherwise there isn't much point in doing this patch.

comment:7 Changed 10 months ago by dgoulet

I honestly don't see an easy way for 033 to have this properly for both client and HS as I think they should not follow the same limitation else we'll just make services one step closer to "not fun to use".

I would maybe prefer having a way to queue request while respecting the circuit creation rate but on HS side I see another DoS opportunity there.

For client, this is much more simpler but for HS we are talking about more thinking and engineering in my opinion which is not something I see easily done for 033...

comment:8 Changed 10 months ago by teor

Ok, let's think about the service side in 0.3.4 and later.

comment:9 Changed 9 months ago by dgoulet

Parent ID: #14006

This is now very related to #14006 that is for client we do have MaxClientCircuitsPending so at worst they do a burst of 32 circuits.

For HS, this can go much bigger with 5+ configured service. I suspect that if we can find a solution in #14006, this is resolved. Not flagging it as duplicate because maybe that circuit limit rate idea is still something we want for an overall client.

comment:10 Changed 9 months ago by asn

Milestone: Tor: 0.3.3.x-finalTor: 0.3.4.x-final

Pushing this to 034 since we lack a plan. Let's work on #14006 in 033 if we can.

comment:11 Changed 8 months ago by nickm

Keywords: 034-triage-20180328 added

comment:12 Changed 7 months ago by asn

Keywords: 034-removed-20180502 added
Milestone: Tor: 0.3.4.x-finalTor: unspecified
Note: See TracTickets for help on using tickets.