Opened 4 years ago

Last modified 16 months ago

#15251 new task

Make tor support starting with 10.000 Tor Hidden Service

Reported by: naif Owned by:
Priority: Low Milestone: Tor: unspecified
Component: Core Tor/Tor Version: Tor: unspecified
Severity: Normal Keywords: tor-hs, scalability, tor-dos
Cc: yawning, meejah, asn, arma Actual Points:
Parent ID: Points: 10
Reviewer: Sponsor:

Description

This ticket is to make Tor support loading and managing 10.000 Tor Hidden services .

Today when you pre-create 10.000 Tor Hidden Service and start Tor, very weird things happen and, basically, it does not work.

This ticket is to analyze and address all the issues related in supporting a Tor instance that load, expose and publish 10.000 Tor Hidden Service descriptors.

This ticket, for testing/experimentation, could make use of functionality described at #6411 .

This ticket is required for implementation of OnionFlare service within Tor2web project https://github.com/globaleaks/Tor2web/issues/228 .

Child Tickets

TicketStatusOwnerSummaryComponent
#17254needs_revisionTvdWScalable HSes by splitting intro/rendezvousCore Tor/Tor
#22210newcircuit_is_acceptable is slow due to IP and fingerprint parsingCore Tor/Tor

Change History (17)

comment:1 Changed 4 years ago by yawning

Cc: arma added
Keywords: tor-hs dos scalability added
Milestone: Tor: 0.2.???
Type: defecttask
Version: Tor: unspecified

CCing arma since he understands this issue better than I do.

So I asked arma about this recently under the guise of spinning up an extremely large number of HSes all serving nyancat. This was mostly hypothetical but it highlights the need to characterize the various performance/DoS issues surrounding being able to rapidly spin up a ridiculous amount of hidden services.

As far as I understand it there's several issues:

  • The local tor instance will explode somewhere around the 100 HS mark and start chewing up 100% CPU.
  • Even if that were fixed, the guard will probably die eventually.

I'm of the opinion that testing along these lines should be done on a private tor network till all the horrible things that may/will happen to the network are well understood, and the 100 per tor client instance may be a good thing till the larger network implications are understood.

comment:2 Changed 4 years ago by naif

@yawning i may suggest to test with real tor network with less than 30-40 TorHS, that very unlikely will cause any harm to Tor network, but will quickly enable debugging of the most common problems.

A testing script using txtorcon support for TorHS loading over TorCP, capable of specifying the amount of TorHS to load and the delay between each TorHS creation over TorCP, would enable quick testing

comment:3 Changed 3 years ago by arma

Severity: Normal

many many onion services will a) use a lot of CPU for establishing each hop of the circuit, and b) use a lot of bandwidth for the same.

If you have enough onion services, then the create and created cells by themselves will fill up your bandwidth, and since all the various "give up on that circuit and try another one" timeouts are fixed, after a certain point you'll constantly be giving up on partially-made circuits and starting over, never finishing anything.

comment:4 Changed 3 years ago by yawning

#13737 should help considerably here assuming the user has sufficiently beefy hardware.

As a future enhancement to that branch, we could attempt to detect outgoing circuit build failures and attempt to degrade more gracefully by dropping rdv requests (under the principle that some service to users is better than no service to users), but I'm not sure how effective such a measure will be given that at some point the HS's Guard will be overloaded...

comment:5 Changed 3 years ago by mikeperry

Keywords: tor-dos added; dos removed

Canonicalize dos tag to tor-dos

comment:6 Changed 3 years ago by nickm

Points: 10

comment:7 Changed 2 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:8 Changed 2 years ago by nickm

Keywords: tor-03-unspecified-201612 added
Milestone: Tor: 0.3.???Tor: unspecified

Finally admitting that 0.3.??? was a euphemism for Tor: unspecified all along.

comment:9 Changed 22 months ago by nickm

Keywords: tor-03-unspecified-201612 removed

Remove an old triaging keyword.

comment:10 Changed 22 months ago by nickm

Priority: MediumLow

comment:11 Changed 16 months ago by naif

GlobaLeaks project do integrate Tor with dynamic setup of Tor Onion Services and by Q1/2018 will release a project that require setup of about 9600 Tor Onion Services, so we will probably work on it somehow.

Each GlobaLeaks instance will have it's own HTTPS certificate enrolled and maintained automatically with LetsEncrypt and it's own Onion Service.

This project will be setting up a virtual GlobaLeaks for each italian public agencies in an automated way, as a massive-scale anti-corruption project.

If someone is going or willing to support Tor debugging to achieve that goal, we'll be more than happy.

comment:12 Changed 16 months ago by teor

You will probably need to fix the performance issues in #22210.
You might also need something like OnionBalance, or more likely, rendezvous handoff from #17254.

comment:13 in reply to:  11 ; Changed 16 months ago by asn

Replying to naif:

GlobaLeaks project do integrate Tor with dynamic setup of Tor Onion Services and by Q1/2018 will release a project that require setup of about 9600 Tor Onion Services, so we will probably work on it somehow.

It's possible that 10k onion services on a single host will probably wreck your guard(s) because of the amount of introduction/HSDir circuits, even if mots of them are low/zero traffic.

Each GlobaLeaks instance will have it's own HTTPS certificate enrolled and maintained automatically with LetsEncrypt and it's own Onion Service.

I recently heard that LetsEncrypt can't make onion certs because they are DV, and not EV.

If someone is going or willing to support Tor debugging to achieve that goal, we'll be more than happy.

I'm interested in helping you with this, so that you achieve your goals without damaging the network. Perhaps we can do an IRC meeting or something?

comment:14 in reply to:  13 ; Changed 16 months ago by naif

Replying to asn:

It's possible that 10k onion services on a single host will probably wreck your guard(s) because of the amount of introduction/HSDir circuits, even if mots of them are low/zero traffic.

That's a thing, wondering if the Facebook-alike optimisation (where it does not require Server Location Anonimity) can help with that setup, or if it would require some different kind of of crypto-circuits-related aspects?

Each GlobaLeaks instance will have it's own HTTPS certificate enrolled and maintained automatically with LetsEncrypt and it's own Onion Service.

I recently heard that LetsEncrypt can't make onion certs because they are DV, and not EV.

Sorry, I meant having 2 channels: HTTPS (with letsencrypt on public IP) and Onion on Tor.

Those will be used for such a national anticorruption platform, and for each single public agencies we will need to have the couple of an HTTPS URL and Onion URL.

So those are independent.

If someone is going or willing to support Tor debugging to achieve that goal, we'll be more than happy.

I'm interested in helping you with this, so that you achieve your goals without damaging the network. Perhaps we can do an IRC meeting or something?

Perfect, if you can join #globaleaks channel there's evilaliv3 that's doing today some preliminary testing with 500 onion address, inserting 10 onion address every 5 minutes, then trying to see what happen as we create a "network blackout" of few minutes.

comment:15 in reply to:  14 Changed 16 months ago by teor

Replying to naif:

Replying to asn:

It's possible that 10k onion services on a single host will probably wreck your guard(s) because of the amount of introduction/HSDir circuits, even if mots of them are low/zero traffic.

That's a thing, wondering if the Facebook-alike optimisation (where it does not require Server Location Anonimity) can help with that setup, or if it would require some different kind of of crypto-circuits-related aspects?

Single onion services don't have guards. They do HSDir posts over a 3-hop path (but with no set guard) to avoid denial of service, but do intro and rendezvous over a single-hop path.

You should set the 2 options needed to turn single onion services on. Check the man page for their exact spellings. You'll probably also need "SOCKSPort 0".


If someone is going or willing to support Tor debugging to achieve that goal, we'll be more than happy.

I'm interested in helping you with this, so that you achieve your goals without damaging the network. Perhaps we can do an IRC meeting or something?

Perfect, if you can join #globaleaks channel there's evilaliv3 that's doing today some preliminary testing with 500 onion address, inserting 10 onion address every 5 minutes, then trying to see what happen as we create a "network blackout" of few minutes.

Try restarting the tor instance as well. #22210 happens on startup.

comment:16 Changed 16 months ago by naif

@teor do you think that Tor2webMode 1 (that require compile-time flags) will make connections going to HSDir to become 1-hop only?
Ref: https://trac.torproject.org/projects/tor/ticket/2553

That way the single-onion will also make outgoing "single-hop" connections for the connections to HSDirs ?

comment:17 in reply to:  16 Changed 16 months ago by teor

Replying to naif:

@teor do you think that Tor2webMode 1 (that require compile-time flags) will make connections going to HSDir to become 1-hop only?
Ref: https://trac.torproject.org/projects/tor/ticket/2553

That way the single-onion will also make outgoing "single-hop" connections for the connections to HSDirs ?

Don't do this. Connecting to HSDirs over a 1-hop path allows HSDirs to selectively deny service to clients and onion services based on their IP address. This is why single onion services connect over a 3-hop path.

If you have this many onion services on a tor instance, it will need to be connected to most relays anyway. If you use a single onion service, it won't use fixed guards, so it will spread the HSDir circuit load over the entire network.

Using single-hop paths for HSDirs is a bug in Tor2web that we plan to fix in #20104.
Also, we plan on removing Tor2web in a few years time when we remove v2 onion services. Tor2web isn't well tested or supported.

Also, have you considered using subdomains on a few onion services, rather than trying to set up 10,000?
Or do you need the authentication to each individual entity?

Note: See TracTickets for help on using tickets.