2019 Q1: Denial of service on v2 and v3 onion service
Dear tor team, We have setup a discussion board, on the tor network. And there is someone that is exploiting within our servers, by taking it down it every time and the forums will respond with "Server not found". We are pretty sure this problem is on the side of the TOR browser, is there anything we could do to sort this? With many thanks for taking time into reading this.
Trac:
Username: pidgin
Activity
changed milestone to %Tor: 0.4.3.x-final
Trac:
Parent Ticket: #29999 (moved)
Child Ticket(s): #29610 (moved), #29637 (moved)added 041-deferred-20190530 041-longterm 042-deferred-20190918 component::core tor/tor milestone::Tor: 0.4.3.x-final network-team-roadmap-2019-Q1Q2 owner::pidgin parent::29999 points::10 priority::immediate reporter::pidgin resolution::fixed security severity::normal sponsor::27-must status::closed tor-dos tor-hs type::defect labels
Setting Trac component (someone please reset it if it is incorrect).
I don't understand the comment "We are pretty sure this problem is on the side of the TOR browser" since this sounds to me like a tor server-side issue. It is also unclear if this is a bug report or a support request.
Also, if you need immediate help please ask on IRC: https://www.torproject.org/about/contact.html.en#irc
Trac:
Component: - Select a component to Core Tor/Tor-
Replying to mcs:
Setting Trac component (someone please reset it if it is incorrect).
I don't understand the comment "We are pretty sure this problem is on the side of the TOR browser" since this sounds to me like a tor server-side issue. It is also unclear if this is a bug report or a support request.
Also, if you need immediate help please ask on IRC: https://www.torproject.org/about/contact.html.en#irc
My apologizes if it's not on TOR side, i am not here to discuss whether it is or not. Just trying to sort this problem out, so people won't have to deal with the problem in the future. And thank you for linking me the irc "I could not use the IRC through TOR, so i will leave this ticket open".
Trac:
Username: pidgin 
Closed #29610 (moved) as a duplicate here. What is causing you to conclude that this is an attack against Tor, instead of (say) an attack against some other component? Right now, there's not enough information in your report(s) to tell.
-
the service behind onion HiddenService is fine, it is serving requests. before the DDOS there have not been "Server Not Found". Actually it was the hackers third iteration. First step from hacker was brute force DDOS which made tor cpu load 100%. countermeasure: vanguards and using ExcludeNodes (torrc) Second iteration from hacker was to use random nodes, about 1000+, to do tor cpu load 100%. countermeasure: vanguards / onionbalance. now tor browser gives "server not found", countermeasure not found yet
Trac:
Username: pidgin #29610 (moved) and #29637 (moved) are duplicates of this ticket.
#29637 (moved) contains the same information from the user, and this response from me:
The attack probably isn't on the Tor client. It's more likely it's on the service, HSDir or Intro Points. It may be on the service's Guard or Rendezvous Point, but they're harder to find and target.
Have you tried v3 onion services? They are much more resistant to attacks. There is no OnionBalance for v3 yet, but the v3 crypto is more efficient, so you might not need it.
Are you sure that your service is still running ok? Your service might be serving a small amount of traffic, and blocked on network requests. So its CPU would be low, and most clients would not be able to connect.
Trac:
Summary: Need urgent help! to Denial of service on v2 onion service-
Replying to teor:
Did you add a v3 service to the same tor instance? Or did you start a separate tor instance with a v3 service?
it is a separate tor instance.
Trac:
Username: pidgin This looks like a duplicate of #25461 (moved).
We'll need to fix #25461 (moved), or confirm that you're not using an affected version, before we can isolate this bug,
How do you know there is an attacker, rather than just lots of clients using your service?
Trac:
Parent: N/A to #25461 (moved)-
Replying to asn:
We might need a log file (preferably info/debug level) to get deeper into this. Please make sure you don't reveal any sensitive info (e.g. guard names) through the log file.
Otherwise, do you see any warn logs, or any weird stuff on the info/debug severity that you can point out?
here is the log from the onion v3 service, a debug/info log follows soon :
-scrubbed-:42:47.000 [warn] No valid circuit build time data out of 1000 times, 3 modes, have_timeout=1, 2332.000000ms -scrubbed-:42:47.000 [warn] No valid circuit build time data out of 1000 times, 3 modes, have_timeout=1, 2332.000000ms -scrubbed-:42:47.000 [warn] No valid circuit build time data out of 1000 times, 3 modes, have_timeout=1, 2332.000000ms -scrubbed-:42:47.000 [warn] No valid circuit build time data out of 1000 times, 3 modes, have_timeout=1, 2332.000000ms -scrubbed-:42:47.000 [warn] No valid circuit build time data out of 1000 times, 3 modes, have_timeout=1, 2332.000000ms -scrubbed-:42:47.000 [warn] No valid circuit build time data out of 1000 times, 3 modes, have_timeout=1, 2332.000000ms -scrubbed-:42:47.000 [warn] No valid circuit build time data out of 1000 times, 3 modes, have_timeout=1, 2332.000000ms -scrubbed-:42:47.000 [warn] No valid circuit build time data out of 1000 times, 3 modes, have_timeout=1, 2332.000000ms -scrubbed-:42:48.000 [warn] circuit_build_times_update_alpha(): Bug: Could not determine largest build time (0). Xm is 3125ms and we've abandoned 999 out of 1000 circuits. (on Tor 0.4.0.1-alpha 81f1b89efc94723f) -scrubbed-:42:48.000 [warn] circuit_build_times_update_alpha(): Bug: Could not determine largest build time (0). Xm is 3125ms and we've abandoned 998 out of 1000 circuits. (on Tor 0.4.0.1-alpha 81f1b89efc94723f) -scrubbed-:42:50.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. -scrubbed-:42:50.000 [warn] Giving up on launching a rendezvous circuit to [scrubbed] for hidden service [scrubbed] -scrubbed-:42:52.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60s after 18 timeouts and 1000 buildtimes. -scrubbed-:45:30.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. -scrubbed-:45:30.000 [warn] Giving up on launching a rendezvous circuit to [scrubbed] for hidden service [scrubbed] -scrubbed-:46:40.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. -scrubbed-:46:40.000 [warn] Giving up on launching a rendezvous circuit to [scrubbed] for hidden service [scrubbed] -scrubbed-:47:25.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. -scrubbed-:47:25.000 [warn] Giving up on launching a rendezvous circuit to [scrubbed] for hidden service [scrubbed] -scrubbed-:47:27.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60s after 18 timeouts and 131 buildtimes. -scrubbed-:47:27.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 120s after 18 timeouts and 0 buildtimes. -scrubbed-:48:47.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. -scrubbed-:48:47.000 [warn] Failed to launch rendezvous circuit to [scrubbed] -scrubbed-:50:34.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. -scrubbed-:50:34.000 [warn] Giving up on launching a rendezvous circuit to [scrubbed] for hidden service [scrubbed]
a debug/info log follows soon
Trac:
Username: pidgin -
Here is a debug log for a tor process under 100% cpu with v2 onion service. some info scrubbed. created 4MB debug logfile per second .
-scrubbed-:27.000 [debug] relay_send_command_from_edge_(): delivering 14 cell forward. -scrubbed-:27.000 [debug] relay_send_command_from_edge_(): Sending a RELAY_EARLY cell; 5 remaining. -scrubbed-:27.000 [debug] relay_encrypt_cell_outbound(): encrypting a layer of the relay cell. -scrubbed-:27.000 [debug] relay_encrypt_cell_outbound(): encrypting a layer of the relay cell. -scrubbed-:27.000 [debug] append_cell_to_circuit_queue(): Made a circuit active. -scrubbed-:27.000 [debug] connection_or_process_cells_from_inbuf(): 11: starting, inbuf_datalen 1028 (0 pending in tls object). -scrubbed-:27.000 [debug] channel_process_cell(): Processing incoming cell_t 0x7ffcaedf5950 for channel 0x563a770113e0 (global ID 2) -scrubbed-:27.000 [debug] circuit_get_by_circid_channel_impl(): circuit_get_by_circid_channel_impl() returning circuit 0x563a779c4eb0 for circ_id 2380366636, channel ID 2 (0x563a770113e0) -scrubbed-:27.000 [debug] circuit_receive_relay_cell(): Sending to origin. -scrubbed-:27.000 [debug] connection_edge_process_relay_cell(): Now seen 10207 relay cells here (command 15, stream 0). -scrubbed-:27.000 [debug] connection_edge_process_relay_cell(): Got an extended cell! Yay. -scrubbed-:27.000 [info] circuit_finish_handshake(): Finished building circuit hop: -scrubbed-:27.000 [info] internal circ (length 5, last hop $-scrubbed-): $-scrubbed-(open) $scrubbed(open) $scrubbed(open) $Cscrubbed(closed) $-scrubbed-(closed) -scrubbed-:27.000 [debug] btc_cevent_rcvr(): CIRC gid=1541 evtype=2 reason=0 onehop=0 -scrubbed-:27.000 [debug] circuit_build_times_add_time(): Adding circuit build time 22290 -scrubbed-:27.000 [debug] circuit_send_intermediate_onion_skin(): starting to send subsequent skin. -scrubbed-:27.000 [info] circuit_send_intermediate_onion_skin(): Sending extend relay cell. -scrubbed-:27.000 [debug] relay_send_command_from_edge_(): delivering 14 cell forward. -scrubbed-:27.000 [debug] relay_send_command_from_edge_(): Sending a RELAY_EARLY cell; 5 remaining. -scrubbed-:27.000 [debug] relay_encrypt_cell_outbound(): encrypting a layer of the relay cell. -scrubbed-:27.000 [debug] relay_encrypt_cell_outbound(): encrypting a layer of the relay cell. -scrubbed-:27.000 [debug] relay_encrypt_cell_outbound(): encrypting a layer of the relay cell. -scrubbed-:27.000 [debug] append_cell_to_circuit_queue(): Made a circuit active. -scrubbed-:27.000 [debug] connection_or_process_cells_from_inbuf(): 11: starting, inbuf_datalen 514 (0 pending in tls object). -scrubbed-:27.000 [debug] channel_process_cell(): Processing incoming cell_t 0x7ffcaedf5950 for channel 0x563a770113e0 (global ID 2) -scrubbed-:27.000 [debug] circuit_get_by_circid_channel_impl(): circuit_get_by_circid_channel_impl() returning circuit 0x563a77524ad0 for circ_id 3378515571, channel ID 2 (0x563a770113e0) -scrubbed-:27.000 [debug] circuit_receive_relay_cell(): Sending to origin. -scrubbed-:27.000 [debug] connection_edge_process_relay_cell(): Now seen 10208 relay cells here (command 35, stream 0). -scrubbed-:27.000 [info] rend_service_receive_introduction(): Received INTRODUCE2 cell for service "uuuuuuuuuuuuuuuuuuu" on circ 3378515571. -scrubbed-:27.000 [debug] extend_info_from_node(): using YYYYYYYYYYYYYYY for scrubbed -scrubbed-:27.000 [info] extend_info_from_node(): Including Ed25519 ID for $scrubbedscrubbed -scrubbed-:27.000 [info] rep_hist_note_used_internal(): New port prediction added. Will continue predictive circ building for 3003 more seconds. -scrubbed-:27.000 [debug] circuit_find_to_cannibalize(): Hunting for a circ to cannibalize: purpose 17, uptime 0, capacity 1, internal 1 -scrubbed-:27.000 [debug] new_route_len(): Chosen route length 5 (6571 direct and 6571 indirect routers suitable). -scrubbed-:27.000 [info] onion_pick_cpath_exit(): Using requested exit node '$scrubbedscrubbed' -scrubbed-:27.000 [debug] onion_extend_cpath(): Path is 0 long; we want 5 -scrubbed-:27.000 [info] select_primary_guard_for_circuit(): Selected primary guard rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrriiiiiiiiiiiiiiiiiii ($-scrubbed-) for circuit. -scrubbed-:27.000 [debug] extend_info_from_node(): using XXXXXXXXXXXXXXzzzzzzzzzzzzzzzzzzzzzz for rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrriiiiiiiiiiiiiiiiiii -scrubbed-:27.000 [info] extend_info_from_node(): Including Ed25519 ID for $-scrubbed-~rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrriiiiiiiiiiiiiiiiiii at XXXXXXXXXXXXXX -scrubbed-:27.000 [debug] onion_extend_cpath(): Chose router $-scrubbed-~rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrriiiiiiiiiiiiiiiiiii at XXXXXXXXXXXXXX for hop #1 (exit is scrubbed)
-scrubbed-:27.000 [debug] onion_extend_cpath(): Path is 1 long; we want 5 -scrubbed-:27.000 [debug] choose_good_middle_server(): Contemplating intermediate hop #2 (closed): random choice. -scrubbed-:27.000 [debug] choose_good_middle_server(): Picking a sticky node (cur_len = 1) -scrubbed-:27.000 [debug] extend_info_from_node(): using rrrrrrrrrrrrrrrrrrrrrrrrzzzzzzzzzzzzzzzzzzzzzz for araglaucogularis -scrubbed-:27.000 [info] extend_info_from_node(): Including Ed25519 ID for $scrubbed
scrubbed -scrubbed-:27.000 [debug] onion_extend_cpath(): Chose router $scrubbedscrubbed for hop #2 (closed) (exit is scrubbed) -scrubbed-:27.000 [debug] onion_extend_cpath(): Path is 2 long; we want 5 -scrubbed-:27.000 [debug] choose_good_middle_server(): Contemplating intermediate hop #3 (closed): random choice. -scrubbed-:27.000 [debug] choose_good_middle_server(): Picking a sticky node (cur_len = 2) -scrubbed-:27.000 [debug] extend_info_from_node(): using scrubbed:kkkkkkkkkkkkkkkkkkkkkkkkkk for rrrrrrrrrrrrrrrrrrrrrrrrrrrr -scrubbed-:27.000 [info] extend_info_from_node(): Including Ed25519 ID for $xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxrrrrrrrrrrrrrrrrrrrrrrrrrrrr at scrubbed -scrubbed-:27.000 [debug] onion_extend_cpath(): Chose router $xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxrrrrrrrrrrrrrrrrrrrrrrrrrrrr at scrubbed for hop #3 (closed) (exit is scrubbed) -scrubbed-:27.000 [debug] onion_extend_cpath(): Path is 3 long; we want 5 -scrubbed-:27.000 [debug] choose_good_middle_server(): Contemplating intermediate hop #4 (closed): random choice. -scrubbed-:27.000 [debug] router_choose_random_node(): We found 5695 running nodes. -scrubbed-:27.000 [debug] router_choose_random_node(): We removed 0 excludednodes, leaving 5695 nodes. -scrubbed-:27.000 [debug] router_choose_random_node(): We removed 4 excludedsmartlist, leaving 5691 nodes. -scrubbed-:27.000 [debug] compute_weighted_bandwidths(): Generated weighted bandwidths for rule weight as middle node based on weights Wg=0.400800 Wm=1.000000 We=0.000000 Wd=0.000000 with total bw 25389038256.000000 -scrubbed-:27.000 [debug] extend_info_from_node(): using fffffffffffffffffffffffffffffff:kkkkkkkkkkkkkkkkkkkkkkkkkk for wwwwwwwwwwwwwwwwwwwwww -scrubbed-:27.000 [info] extend_info_from_node(): Including Ed25519 ID for $gggggggggggggggggggggggggggggggggggwwwwwwwwwwwwwwwwwwwwww at fffffffffffffffffffffffffffffff -scrubbed-:27.000 [debug] onion_extend_cpath(): Chose router $gggggggggggggggggggggggggggggggggggwwwwwwwwwwwwwwwwwwwwww at fffffffffffffffffffffffffffffff for hop #4 (closed) (exit is scrubbed) -scrubbed-:27.000 [debug] onion_extend_cpath(): Path is 4 long; we want 5 -scrubbed-:27.000 [debug] onion_extend_cpath(): Chose router $scrubbedscrubbed for hop #5 (closed) (exit is scrubbed) -scrubbed-:27.000 [debug] onion_extend_cpath(): Path is complete: 5 steps long -scrubbed-:27.000 [debug] btc_cevent_rcvr(): CIRC gid=2543 evtype=0 reason=0 onehop=0 -scrubbed-:27.000 [debug] circuit_handle_first_hop(): Looking for firsthop 'XXXXXXXXXXXXXXzzzzzzzzzzzzzzzzzzzzzz' -scrubbed-:27.000 [debug] btc_event_rcvr(): CIRC gid=2543 chan=2 onehop=0 -scrubbed-:27.000 [debug] circuit_handle_first_hop(): Conn open. Delivering first onion skin. -scrubbed-:27.000 [debug] circuit_send_first_onion_skin(): First skin; sending create cell. -scrubbed-:27.000 [debug] circuit_get_by_circid_channel_impl(): circuit_get_by_circid_channel_impl() found nothing for circ_id 3035601791, channel ID 2 (0x563a770113e0) -scrubbed-:27.000 [debug] circuit_deliver_create_cell(): Chosen circID 3035601791. -scrubbed-:27.000 [debug] circuitmux_attach_circuit(): Attaching circuit 3035601791 on channel 2 to cmux 0x563a770079d0 -scrubbed-:27.000 [debug] append_cell_to_circuit_queue(): Made a circuit active. -scrubbed-:27.000 [debug] btc_state_rcvr(): CIRC gid=2543 state=0 onehop=0 -scrubbed-:27.000 [info] circuit_send_first_onion_skin(): First hop: finished sending CREATE cell to '$-scrubbed-~rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrriiiiiiiiiiiiiiiiiii at XXXXXXXXXXXXXX' -scrubbed-:27.000 [info] rend_service_receive_introduction(): Accepted intro; launching circuit to [scrubbed] (cookie 2E7D742C) for service uuuuuuuuuuuuuuuuuuu. -scrubbed-:27.000 [debug] connection_or_process_cells_from_inbuf(): 11: starting, inbuf_datalen 0 (0 pending in tls object). -scrubbed-:27.000 [debug] conn_read_callback(): socket 10 wants to read. -scrubbed-:27.000 [debug] connection_buf_read_from_socket(): 10: starting, inbuf_datalen 0 (0 pending in tls object). at_most 16448. -scrubbed-:27.000 [debug] connection_buf_read_from_socket(): After TLS read of 1028: 1057 read, 0 written -scrubbed-:27.000 [debug] connection_or_process_cells_from_inbuf(): 10: starting, inbuf_datalen 1028 (0 pending in tls object). -scrubbed-:27.000 [debug] channel_process_cell(): Processing incoming cell_t 0x7ffcaedf5950 for channel 0x563a76ffd020 (global ID 1) -scrubbed-:27.000 [debug] circuit_get_by_circid_channel_impl(): circuit_get_by_circid_channel_impl() returning circuit 0x563a773f5cd0 for circ_id 3703046038, channel ID 1 (0x563a76ffd020) -scrubbed-:27.000 [debug] circuit_receive_relay_cell(): Sending to origin. -scrubbed-:27.000 [debug] connection_edge_process_relay_cell(): Now seen 10209 relay cells here (command 15, stream 0). -scrubbed-:27.000 [debug] connection_edge_process_relay_cell(): Got an extended cell! Yay. -scrubbed-:27.000 [info] circuit_finish_handshake(): Finished building circuit hop: -scrubbed-:27.000 [info] internal circ (length 5, last hop wwwwwwwwwwwwwwwwwwwwww): $hhhhhhhhhhhhhhhhhhhhhhhhh(open) eeeeeeeeeeeeeeeeeeeeeeeee(open) nnnnnnnnnnnnnnnnnnnnnnnnnnnnnn(open) vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv(open) lllllllllllllllllllllllllllllllllllllllll(open) -scrubbed-:27.000 [debug] btc_cevent_rcvr(): CIRC gid=1023 evtype=2 reason=0 onehop=0 -scrubbed-:27.000 [info] entry_guards_note_guard_success(): Recorded success for primary confirmed guard ($hhhhhhhhhhhhhhhhhhhhhhhhh) -scrubbed-:27.000 [debug] btc_state_rcvr(): CIRC gid=1023 state=4 onehop=0 -scrubbed-:27.000 [info] circuit_build_no_more_hops(): circuit built! -scrubbed-:27.000 [debug] btc_cevent_rcvr(): CIRC gid=1023 evtype=1 reason=0 onehop=0 -scrubbed-:27.000 [info] rend_service_rendezvous_has_opened(): Done building circuit 3703046038 to rendezvous with cookie A9ADDEC1 for service uuuuuuuuuuuuuuuuuuu96EB1FD02A5(open) nnnnnnnnnnnnnnnnnnnnnnnnnnnnnn(open) vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv(open) lllllllllllllllllllllllllllllllllllllllll(open) -scrubbed-:27.000 [debug] btc_cevent_rcvr(): CIRC gid=1023 evtype=2 reason=0 onehop=0 -scrubbed-:27.000 [info] entry_guards_note_guard_success(): Recorded success for primary confirmed guard ($hhhhhhhhhhhhhhhhhhhhhhhhh) -scrubbed-:27.000 [debug] btc_state_rcvr(): CIRC gid=1023 state=4 onehop=0 -scrubbed-:27.000 [info] circuit_build_no_more_hops(): circuit built! -scrubbed-:27.000 [debug] btc_cevent_rcvr(): CIRC gid=1023 evtype=1 reason=0 onehop=0 -scrubbed-:27.000 [info] rend_service_rendezvous_has_opened(): Done building circuit 3703046038 to rendezvous with cookie A9ADDEC1 for service uuuuuuuuuuuuuuuuuuu -scrubbed-:27.000 [info] internal circ (length 5): $hhhhhhhhhhhhhhhhhhhhhhhhh(open) eeeeeeeeeeeeeeeeeeeeeeeee(open) nnnnnnnnnnnnnnnnnnnnnnnnnnnnnn(open) vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv(open) lllllllllllllllllllllllllllllllllllllllll(open)Trac:
Username: pidgin -
Replying to teor:
This looks like a duplicate of #25461 (moved).
We'll need to fix #25461 (moved), or confirm that you're not using an affected version, before we can isolate this bug,
How do you know there is an attacker, rather than just lots of clients using your service?
the attacker contacted us for extortion. The attacker turned the DDOS off and on, he informed us about times. cpu loads matches times. cpu load is at all instances at 100% at attack. if DDOS not taking place, all tor processes are at 10% max (average far less)
Trac:
Username: pidgin 
Hm, those logs are OK but not useful enough unfortunately.
The only interesting thing is that your circuit build times are big (like 22.9 seconds for a single circuit):
circuit_build_times_add_time(): Adding circuit build time 22290Not much I can get from it. You also seem to have borked the logs in the end since
rend_service_rendezvous_has_opened()appears twice with duplicated stuff around it.I think ideally we would need a log over a greater period of time (like 3-5 mins) to see what the attacker does over time. Perhaps the way to do it is to only gather info logs (I don't think we need debugs), and compress it before attaching it here. Otherwise, send it to us in a personal email.
Ok, so this is not #25461 (moved).
Here's another interesting log:
-scrubbed-:47:25.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit.This looks like an instance of #28962 (moved), where Tor fails guards based on the absolute number of failures of each guard, rather than checking if the guard is (much) worse than the average number of failures.
We might end up fixing #28862 (moved) as part of this ticket, or as part of our IPv6 work.
Trac:
Parent: #25461 (moved) to N/A-
Replying to teor:
Ok, so this is not #25461 (moved).
Here's another interesting log: {{{ -scrubbed-:47:25.000 [warn] Failed to find node for hop #1 of our path. Discarding this circuit. }}}
This looks like an instance of #28962 (moved), where Tor fails guards based on the absolute number of failures of each guard, rather than checking if the guard is (much) worse than the average number of failures.
We might end up fixing #28862 (moved) as part of this ticket, or as part of our IPv6 work. Hi Teor, what is the e-mail i can mail it to ? Also do you have a public pgp i can import? so i can send it in PGP format ?
Trac:
Username: pidgin -
Replying to pidgin:
Any updates?
You could send logs to my email address/PGP: https://www.torproject.org/about/corepeople.html.en#asn Teor is also there: https://www.torproject.org/about/corepeople.html.en#teor
Keep in mind it's the weekend and we are generally all pretty busy! :/
-
Replying to asn:
Replying to pidgin:
Any updates?
You could send logs to my email address/PGP: https://www.torproject.org/about/corepeople.html.en#asn Teor is also there: https://www.torproject.org/about/corepeople.html.en#teor
Keep in mind it's the weekend and we are generally all pretty busy! :/
Understandable, thank you for the information. Send to both e-mails.
Trac:
Username: pidgin -
Hey Pidgin,
I looked at the logs but nothing jumped out at me as super weird. There is obviously lots of activity on the HS, but doesn't seem like something that would 100% CPU your Tor (you are receiving a rendezvous request every two seconds). There is probably something else going on that I couldn't see from that first look.
Another approach would be to try to profile Tor to see which functions are causing the CPU increase: https://gitweb.torproject.org/tor.git/tree/doc/HACKING/HelpfulTools.md#n183 It's likely that this might also not bear any fruit, but it might be worth trying...
Also, take extra care in sanitizing guard nicknames when sending logs (grep "for guard").
Finally, I'm pretty overwhelmed with stuff for March, so I will probably not have much time to look at this until April.
-
Replying to teor:
Hi pidgin,
We need to know more about your setup to help you:
Replying to teor:
The v3 service doesn't have the guard issue, it's getting much further along. Have you tried running it on a separate machine? (Or by itself without v2 running on the same machine?)
there is enough cpu and bandwidth
Trac:
Username: pidgin Replying to pidgin:
Replying to teor:
Hi pidgin,
We need to know more about your setup to help you:
Replying to teor:
The v3 service doesn't have the guard issue, it's getting much further along. Have you tried running it on a separate machine? (Or by itself without v2 running on the same machine?)
there is enough cpu and bandwidth
Tor also uses many other resources that can become exhausted, like sockets, memory, and various kernel data structures.
Can you please test the v3 service on a machine by itself?
-
the v3 onion tor process was at 100% cpu at another machine
last notice log messages:
scrubbed:39:57.000 [notice] Extremely large value for circuit build timeout: 128s. Assuming clock jump. Purpose 14 (Measuring circuit timeout) scrubbed:39:57.000 [notice] Extremely large value for circuit build timeout: 128s. Assuming clock jump. Purpose 14 (Measuring circuit timeout) scrubbed:39:57.000 [notice] Extremely large value for circuit build timeout: 128s. Assuming clock jump. Purpose 14 (Measuring circuit timeout) scrubbed:39:57.000 [notice] Extremely large value for circuit build timeout: 128s. Assuming clock jump. Purpose 14 (Measuring circuit timeout) scrubbed:39:57.000 [notice] Extremely large value for circuit build timeout: 128s. Assuming clock jump. Purpose 14 (Measuring circuit timeout) scrubbed:39:57.000 [notice] Extremely large value for circuit build timeout: 128s. Assuming clock jump. Purpose 14 (Measuring circuit timeout) scrubbed:57:39.000 [warn] Unknown introduction point auth key on circuit 4162469458 for service [scrubbed] scrubbed:57:39.000 [warn] circuit_mark_for_close_(): Bug: Duplicate call to circuit_mark_for_close at src/feature/hs/hs_service.c:3227 (first at src/feature/hs/hs_service.c:2446) (on Tor 0.4.0.1-alpha 81f1b89efc94723f)
with the logmessage "57:39.000 [warn] circuit_mark_for_close_():" the cpu is now at 0% and the service is not accessible
Trac:
Username: pidgin -
if tor developer need direct traffic analyzing then I propose the following:
-
user pidgin sets up onionbalance server to use an main onion that is being attacked
-
tor developer provide onion url to pidgin so pidgin can plug-in to onionbalance mirror only
-
tor developer receive dos attack to mirror onion, then developer can analyze attack exactly
Trac:
Username: 993872315 -
-
Closed #29919 (moved) as a duplicate for this one. More info over there.
Trac:
Summary: Denial of service on v2 and v3 onion service to 2019 Q1: Denial of service on v2 and v3 onion service -
Replying to asn:
Closed #29919 (moved) as a duplicate for this one. More info over there.
Hi asn,
Understandable why you closed my ticket, at a point of desperation and just hoping someone will take real interest in looking into this. Which is why I am able to offer access to a server that is currently being attacked. I believe I saw a chat log of you first discussing complex mode in 2015 for OnionBalance. Do you have any links for how to enable it/configure it? I am going to try it out to see if it is a resolution for this, with the theory of introduction points being attacked.
Thank you.
Trac:
Username: HelpDOS -
Replying to HelpDOS:
Replying to asn:
Closed #29919 (moved) as a duplicate for this one. More info over there.
Hi asn,
Understandable why you closed my ticket, at a point of desperation and just hoping someone will take real interest in looking into this. Which is why I am able to offer access to a server that is currently being attacked. I believe I saw a chat log of you first discussing complex mode in 2015 for OnionBalance. Do you have any links for how to enable it/configure it? I am going to try it out to see if it is a resolution for this, with the theory of introduction points being attacked.
Thank you.
Hey, I just remembered that complex mode was never implemented for onionbalance, because it was harder to implement and we thought there was no real use for it.
I'm not currently interested (or have the time) to get access to a server that is under attack.
I think the most useful thing right now would be to have more logs that display the attack. I want debug or info logs that last for 1-2 hours of the attack and display the whole Tor lifetime (from startup to shutdown). Please sanitize them correctly (make sure that guard names and onion names are not visible).
Same for vanguard logs on debug or info if you use vanguards.
-
Replying to asn:
Replying to HelpDOS:
Replying to asn:
Closed #29919 (moved) as a duplicate for this one. More info over there.
Hi asn,
Understandable why you closed my ticket, at a point of desperation and just hoping someone will take real interest in looking into this. Which is why I am able to offer access to a server that is currently being attacked. I believe I saw a chat log of you first discussing complex mode in 2015 for OnionBalance. Do you have any links for how to enable it/configure it? I am going to try it out to see if it is a resolution for this, with the theory of introduction points being attacked.
Thank you.
Hey, I just remembered that complex mode was never implemented for onionbalance, because it was harder to implement and we thought there was no real use for it.
I'm not currently interested (or have the time) to get access to a server that is under attack.
I think the most useful thing right now would be to have more logs that display the attack. I want debug or info logs that last for 1-2 hours of the attack and display the whole Tor lifetime (from startup to shutdown). Please sanitize them correctly (make sure that guard names and onion names are not visible).
Same for vanguard logs on debug or info if you use vanguards.
I will provide you with any logs I can later today. Could you please send a full list of anything that could help in debugging just to make sure you have everything relevant? Thank you
Trac:
Username: HelpDOS -
Replying to HelpDOS:
Replying to asn:
Replying to HelpDOS:
Replying to asn:
Closed #29919 (moved) as a duplicate for this one. More info over there.
Hi asn,
Understandable why you closed my ticket, at a point of desperation and just hoping someone will take real interest in looking into this. Which is why I am able to offer access to a server that is currently being attacked. I believe I saw a chat log of you first discussing complex mode in 2015 for OnionBalance. Do you have any links for how to enable it/configure it? I am going to try it out to see if it is a resolution for this, with the theory of introduction points being attacked.
Thank you.
Hey, I just remembered that complex mode was never implemented for onionbalance, because it was harder to implement and we thought there was no real use for it.
I'm not currently interested (or have the time) to get access to a server that is under attack.
I think the most useful thing right now would be to have more logs that display the attack. I want debug or info logs that last for 1-2 hours of the attack and display the whole Tor lifetime (from startup to shutdown). Please sanitize them correctly (make sure that guard names and onion names are not visible).
Same for vanguard logs on debug or info if you use vanguards.
I will provide you with any logs I can later today. Could you please send a full list of anything that could help in debugging just to make sure you have everything relevant? Thank you
Hm. It would be great if we could have all debug logs from Tor startup to Tor shutdown. Please scrub the names of your primary guards and your onion address and anything else that might seem pervasive, but please try to not destroy the accuracy of the logs (by double-pasting or removing surrounding lines).
Another thing that might be helpful would be to try with a blank state file so that Tor discards any previous circuit timeouts and performance measurements etc. (you can find the state file in your data directory. please don't delete it, just backup it somewhere else so that you can then restore it).
Also, it would be better to not use vanguards while you are collecting these logs because it's hard to keep track of what they are doing while reading the logs.
Another useful thing would be to try to profile Tor to see which functions are causing the CPU increase: https://gitweb.torproject.org/tor.git/tree/doc/HACKING/HelpfulTools.md#n183
Finally, please let us know of any custom configurations you've done on your onion service.
Cheers.
Have you tried contacting the writer of hackerfactor? He had many good posts about handling attacks against onion services:
https://www.hackerfactor.com/blog/index.php?/archives/762-Attacked-Over-Tor.html https://www.hackerfactor.com/blog/index.php?/archives/763-The-Continuing-Tor-Attack.html https://www.hackerfactor.com/blog/index.php?/archives/777-Stopping-Tor-Attacks.html https://www.hackerfactor.com/blog/index.php?/archives/779-Behind-the-Tor-Attacks.html https://www.hackerfactor.com/blog/index.php?/archives/789-Cyber-Goat.html https://www.hackerfactor.com/blog/index.php?/archives/791-Tor-Attacks-Revisited.html https://www.hackerfactor.com/blog/index.php?/archives/804-A-New-Tor-Attack.html
-
Replying to pidgin:
Problem is still not solved, still the same error. I have provided everything i could to you guys i have no clue what to do else.
Hello. No one has provided the logs asked in comment:39 yet.
-
Replying to pidgin:
Problem is still not solved, still the same error. I have provided everything i could to you guys i have no clue what to do else.
I found a complete solution but it needs more work to be reliable enough. Get in touch with me via your onion.
Will provide the solution here also but it is a novel method of identifying offending connections so they can then be dropped, which would likely not be implemented into core tor.
Trac:
Username: HelpDOS -
Replying to pidgin:
Problem is still not solved, still the same error. I have provided everything i could to you guys i have no clue what to do else.
Via "librarytask"
Trac:
Username: HelpDOS