Opened 6 years ago

Closed 5 years ago

Last modified 12 months ago

#13414 closed defect (wontfix)

Increase Authorities' AuthDirMaxServersPerAddr to 4 or 8 to use more processors

Reported by: teor Owned by:
Priority: Medium Milestone:
Component: Core Tor/Tor Version: Tor: unspecified
Severity: Normal Keywords: tor-auth tor-router
Cc: mikeperry, isis, moritz, nickm, arma Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Due to the increase in logical processors per machine, a recent conversation on the tor-dev mailing list suggested increasing the Tor Authorities' AuthDirMaxServersPerAddr,[0] (Mortiz Bartl) either to unlimited[1] (isis) or 4 or 8 [2][3] (mikeperry, teor).

I suggest we initially increase the consensus parameter to 8, quadrupling CPU-bound throughput, and then, if successful, change the default in code in a major release.

Details:

The increase in logical and physical processors per machine has outstripped tor's ability to parallelise its workload, artificially limiting the network throughput.[4] (AFO-Admin)

  • Scarcity of IPv4 addresses, particularly in some regions
  • Multiple relays sharing IPv4 addresses due to VPSs and/or NAT - see #13234

Long-term:

Long-term work that will resolve this issue:

  • Parallelise more of tor's compute workload [5]
  • Optimise Cryptography, either through algorithm choice or code refactoring
  • Implement/Test/Deploy/Activate IPv6 ORPorts

Potential Concerns:

This could make Sybil attacks slightly easier, but we already mitigate against Syblils on the same IP using the /24 filter. isis wasn't concerned about extra Sybils from this change.[1]

This change may slightly increase the size of the consensus. However, there are multiple upcoming plans to reduce consensus size, including:

  • Consensus Diffs
  • Reducing Consensus Size by Excluding the Slowest Relays

[0]: https://lists.torproject.org/pipermail/tor-dev/2014-September/007570.html
[1]: https://lists.torproject.org/pipermail/tor-dev/2014-September/007573.html
[2]: https://lists.torproject.org/pipermail/tor-dev/2014-September/007574.html
[3]: https://lists.torproject.org/pipermail/tor-dev/2014-October/007575.html
[4]: https://lists.torproject.org/pipermail/tor-dev/2014-September/007565.html
[5]: https://lists.torproject.org/pipermail/tor-dev/2014-September/007558.html

Child Tickets

Change History (13)

comment:1 Changed 6 years ago by teor

Keywords: tor-auth added; tor-authorities removed
Milestone: Tor: 0.2.6.x-final
Version: Tor: unspecified

Set timeframe to 0.2.6.x-final per nickm on #13234
Can someone help me cc nickm on this?

comment:2 in reply to:  1 Changed 6 years ago by isis

Cc: nickm added

Replying to teor:

Can someone help me cc nickm on this?

Done. :)

For what it's worth, I am concerned for Sybil Attacks... it's just that I'm inclined to think that preventing IPs on large pipes from using all their bandwidth is more likely to limit good relay operators who would like to help the network, rather than limit determined attackers.

That said, I agree with mikeperry and teor that limiting it to some smallish number like 4, 8, or even 16 would be better than leaving it completely unrestricted, due to potential DoS implications of the latter.

Last edited 6 years ago by isis (previous) (diff)

comment:3 Changed 6 years ago by teor

Sorry, isis, I should have quoted you on Sybils directly :-)

Here are some factors for consideration, including some quick calculations. Most favour a parameter of 4-8.

Limits based on DoS

Yes, there are DoS implications for an attacker who could control 1000s of IPs: doubling the size of the consensus, for example.

So, if we use "double the consensus" as a rough guide to undesirability, and conservatively assume 6000 routers, an attacker would need to control:

  • 16* 375 IPv4 addresses - 1.5 x /24
  • 8* 750 IPv4 addresses - 3 x /24
  • 4*1500 IPv4 addresses - 6 x /24
  • 2*3000 IPv4 addresses - 12 x /24

Based on that quick and dirty calculation, I'd limit AuthDirMaxServersPerAddr to 8 router instances per IP, to make a consensus DoS slightly harder than a class C block. Even AFO-Admin would have needed at most 14 on their server to saturate their pipe. (And they could get a second IP for that, if needed. Or activate AES-NI and similar optimisations to bring the CPU load down.)

Limits based on Hardware

As an example, Intel's Xeon series ranges from 1-10 cores, with 2 & 4 cores being the most common. Hyper-threading increases this to 4-8 threads/logical processors for many of the later models.[0]

Given that we already have limited parallelism with tor worker processes, I think 8 relays per IP would work well for the majority of server hardware.

Limits based on OS resources

Using 8 instances to connect to around 5000 routers each could see us using 40,000 connections/files. As discussed in #9708, this starts to stretch the ulimit / sysctl kern.* parameters on Linux & *BSD/OS X systems.

By the way, we currently set MAX_CPUWORKERS and MAX_DETECTABLE_CPUS to 16 in tor, so we should perhaps adjust that as well if we go to 16 or more instances per IP. (Up or down?)

I could imagine a server with 8 tor instances and 8 logical CPUs using 8 tor + 8*8 cpuworkers = 72 processes just for tor. Above this point, all that process scheduling starts to become counter-productive.

By contrast, the 4 + 4*4 case yields 20 processes, which is a little low, even for a 4-processor system, particularly when cpuworkers don't do much.

So I'd favour 8 for these reasons as well. (And 8-processor+ users can learn to configure NumCPUs if they have issues.)

[0]: https://en.wikipedia.org/wiki/Xeon#Overview

comment:4 Changed 6 years ago by teor

Notes from a discussion on #tor-dev, to be turned into an email to tor-relays

[14:30] <teor> A bit of context: multithreading, optimisation, and / or IPv6 will eventually solve this issue
[14:31] <qwerty1> yeah
[14:31] <teor> But in the meantime, many routers are limited by IPv4 scarcity, limited tor parallelism, and increasing cores per server
[14:31] <teor> Some are also sharing IPv4 addresses inadvertently through NAT and/or VPS
...
[14:35] <teor> My quick DoS calculations are: to equal the size of the Tor network, currently an attacker needs to control 3000 IPv4 addresses = 12 x /24
...
[14:36] <teor> And 8 per IP is 750 IPv4 addresses - 3 x /24
[14:37] <teor> Any larger than 8, and an attacker with a single /24 can have a significant impact on the network. This is clearly undesirable.
[14:38] <teor> Also, as far as OS resources go, any more than 8 starts to hit file and process scheduling hard
...
[14:39] <teor> And hardware: 4-8 is the most common number of logical cores for Intel's Xeon range. Others are similar.
[14:39] <teor> Although AFO-Admin's router may need up to 14 tor processes to use all their bandwidth. But that's just 1 more IP at 8 per IP.
[14:40] <teor> TL;DR: I like 8

comment:5 Changed 6 years ago by nickm

FWIW, I'm not so sure about this one. I worry about the sybil implications.

comment:6 Changed 6 years ago by teor

Are you suggesting no change, Nick?
Or an incremental change from, say, 2 to 3?

comment:7 Changed 6 years ago by nickm

I don't actually know; I bet Roger would have some good guidance about how to decide.

comment:8 Changed 6 years ago by teor

Cc: arma added

Roger, would increasing AuthDirMaxServersPerAddr to 3 keep us safe (enough) from sybils, while increasing the amount of bandwidth available on the fastest relays?

comment:9 Changed 6 years ago by nickm

Milestone: Tor: 0.2.6.x-finalTor: 0.2.???

I'm thinking that with the sybils last month, we might not want to make it *easier* to spin up 3k nodes? Maybe we'll get other answers here though. I don't think this gets resoled this month though.

comment:10 Changed 5 years ago by teor

Resolution: wontfix
Severity: Normal
Status: newclosed

I don't think this is going to be useful, it appears that multithreading will happen first.

Please feel free to reopen if there's progress.

comment:11 Changed 4 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:12 Changed 4 years ago by nickm

Milestone: Tor: 0.3.???

Milestone deleted

comment:13 Changed 12 months ago by cypherpunks

Still no multithreading with multisocket many core servers limited by AuthDirMaxServersPerAddr 2 tor. Hard to saturate 10GE \20GE

Note: See TracTickets for help on using tickets.