Opened 4 years ago

Closed 3 years ago

Last modified 3 years ago

#17037 closed defect (fixed)

Too many introductions makes hidden service unreachable

Reported by: alberto Owned by:
Priority: Medium Milestone:
Component: Core Tor/Tor Version: Tor: 0.2.7.2-alpha
Severity: Normal Keywords: tor-hs
Cc: asn, dgoulet Actual Points:
Parent ID: #15463 Points:
Reviewer: Sponsor:

Description

After few time- Hidden Service unavailable.
In tor logs- very many records like(every type of records count- 20-30 in one second):


Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.
Sep 11 02:14:15.000 [warn] Couldn't relaunch rendezvous circuit to '[scrubbed]'.


Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>
Sep 11 02:14:15.000 [warn] Giving up launching first hop of circuit to rendezvous point [scrubbed] for service <NAME_HIDDEN_SERVICE>


Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.
Sep 11 02:21:27.000 [warn] connection_edge_process_relay_cell (at origin) failed.


Sep 11 01:51:17.000 [warn] rend_service_receive_introduction(): Bug: Internal error: Got an INTRODUCE2 cell on an intro circ (for service "<HIDDEN_SERVICE_NAME>")
with no corresponding rend_intro_point_t. (on Tor 0.2.7.2-alpha-dev fa89eb60e952b964)
Sep 11 01:51:17.000 [warn] rend_service_receive_introduction(): Bug: Internal error: Got an INTRODUCE2 cell on an intro circ (for service "<HIDDEN_SERVICE_NAME>")
with no corresponding rend_intro_point_t. (on Tor 0.2.7.2-alpha-dev fa89eb60e952b964)
Sep 11 01:51:17.000 [warn] rend_service_receive_introduction(): Bug: Internal error: Got an INTRODUCE2 cell on an intro circ (for service "<HIDDEN_SERVICE_NAME>")
with no corresponding rend_intro_point_t. (on Tor 0.2.7.2-alpha-dev fa89eb60e952b964)
Sep 11 01:51:17.000 [warn] rend_service_receive_introduction(): Bug: Internal error: Got an INTRODUCE2 cell on an intro circ (for service "<HIDDEN_SERVICE_NAME>")
with no corresponding rend_intro_point_t. (on Tor 0.2.7.2-alpha-dev fa89eb60e952b964)
Sep 11 01:51:17.000 [warn] rend_service_receive_introduction(): Bug: Internal error: Got an INTRODUCE2 cell on an intro circ (for service "<HIDDEN_SERVICE_NAME>")
with no corresponding rend_intro_point_t. (on Tor 0.2.7.2-alpha-dev fa89eb60e952b964)


After restart TOR - many restarting problems (But no problems with network, and early- never see errors like this)


Sep 11 00:44:32.000 [notice] Bootstrapped 0%: Starting
Sep 11 00:44:33.000 [notice] Bootstrapped 80%: Connecting to the Tor network
Sep 11 00:44:33.000 [notice] Bootstrapped 85%: Finishing handshake with first hop
Sep 11 00:44:56.000 [warn] Problem bootstrapping. Stuck at 85%: Finishing handshake with first hop. (DONE; DONE; count 10; recommendation warn; host 35E85BAE6
4BBFE1C921A3167644D8FB7E766FC25 at 85.114.132.36:9001)
Sep 11 00:44:56.000 [warn] 13 connections have failed:
Sep 11 00:44:56.000 [warn] 5 connections died in state handshaking (TLS) with SSL state SSLv3 read finished A in HANDSHAKE
Sep 11 00:44:56.000 [warn] 5 connections died in state handshaking (TLS) with SSL state SSLv2/v3 read server hello A in HANDSHAKE
Sep 11 00:44:56.000 [warn] 3 connections died in state connect()ing with SSL state (No SSL object)


Too in logs many 'retrying new circuit' records.


Sep 11 00:57:46.000 [notice] We tried for 15 seconds to connect to '[scrubbed]' using exit $60683F31D32D1BAC2FA10770CFBCE2A16E6467FF~31173torproxy03 at 185.65
.135.228. Retrying on a new circuit.
Sep 11 00:57:46.000 [notice] We tried for 15 seconds to connect to '[scrubbed]' using exit $60683F31D32D1BAC2FA10770CFBCE2A16E6467FF~31173torproxy03 at 185.65
.135.228. Retrying on a new circuit.
Sep 11 00:57:47.000 [notice] We tried for 15 seconds to connect to '[scrubbed]' using exit $60683F31D32D1BAC2FA10770CFBCE2A16E6467FF~31173torproxy03 at 185.65
.135.228. Retrying on a new circuit.


Can't make debug logs, 'cause attack start at random time, and if wait this- size of debug log file- very big(don't have terabytes of disk spaces)

Problem exist at 0.2.7.2-alpha, too exist at 0.2.6 branches.

If move hidden service to another server and OS(and another hosting company)- problem absolutely same.
Hidden service work, after few time - 10-20-30-60 minutes- many records in logs like I write earlier at top of ticket. Hidden service not available.
TOR process give near 10% of CPU (in normal work - 2-3%)

Child Tickets

Change History (8)

comment:1 Changed 4 years ago by yawning

Cc: asn dgoulet added
Keywords: tor-hs added
Parent ID: #15463
Priority: criticalnormal

Looks like an overloaded guard at first glance. I don't particularly view this as a critical issue since there is no alternative to this solution than "too bad, hope the Guard isn't overloaded later", since rotating to a different Guard will do really bad things to anonymity.

Do rdv. circuits honor MaxClientCircuitsPending? If not, should there be a separate var?

CCing the HS people who may have an actual solution to the issue.

comment:2 Changed 4 years ago by alberto

HS work with MaxClientCircuitsPending value is 256. This is advice from one of TOR-developers since 1st DoS 6-8 month ago.

comment:3 Changed 4 years ago by teor

Milestone: Tor: 0.2.???

comment:4 Changed 3 years ago by arma

Severity: Normal
Status: newneeds_information

Is this bug still happening with the stable Tor 0.2.7.x releases?

We fixed e.g. #16702 which might have helped cause this one.

comment:5 Changed 3 years ago by dgoulet

Resolution: fixed
Status: needs_informationclosed

Indeed, #16702 should fixed this issue thus available in version _after_ 0.2.7.2. I haven't seen this error in all the HS I run (0.2.8.0+) so if you see it again, please re-open the ticket.

comment:6 Changed 3 years ago by arma

Summary: Strange errors. Seems like new type of attack to hidden service.Too many introductions makes hidden service unreachable

(I'm retitling the ticket since I was just showing it to somebody and the old title wasn't very helpful.)

comment:7 Changed 3 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:8 Changed 3 years ago by nickm

Milestone: Tor: 0.3.???

Milestone deleted

Note: See TracTickets for help on using tickets.