Opened 4 years ago

Closed 4 years ago

Last modified 3 years ago

#14129 closed defect (fixed)

Assertion failure in dns.c, possibly connected to UDP DoS attack

Reported by: jowr Owned by:
Priority: High Milestone: Tor: 0.2.4.x-final
Component: Core Tor/Tor Version: Tor: 0.2.6.1-alpha
Severity: Keywords: crash, tor-relay, 025-backport, 024-backport, CVE-assigned, tor-dos
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I run a full exit node, and today I was hit by a udp denial of service. Standard botnet garbage, really. But something surprising happened:

Jan 7 10:54:51 testbed Tor[4289]: Circuit handshake stats since last time: 92457/94799 TAP, 209527/209991 NTor.
Jan 7 15:28:03 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 15:28:03 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 15:28:03 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up
Jan 7 15:28:03 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up
Jan 7 15:28:48 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 15:28:48 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 15:28:48 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up
Jan 7 15:28:48 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up
Jan 7 15:29:01 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 15:29:01 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 15:29:01 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up
Jan 7 15:29:01 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up
Jan 7 15:29:23 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 15:29:23 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 15:29:23 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up
Jan 7 15:29:23 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up
Jan 7 15:30:04 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 15:30:04 testbed Tor[4289]: eventdns: All nameservers have failed
Jan 7 16:06:15 testbed Tor[4289]: tor_assertion_failed_(): Bug: src/or/dns.c:1136: connection_dns_remove: Assertion 0 failed; aborting.
Jan 7 16:06:15 testbed Tor[4289]: tor_assertion_failed_(): Bug: src/or/dns.c:1136: connection_dns_remove: Assertion 0 failed; aborting.
Jan 7 16:06:15 testbed Tor[4289]: Bug: Assertion 0 failed in connection_dns_remove at src/or/dns.c:1136. Stack trace:
Jan 7 16:06:15 testbed Tor[4289]: Bug: Assertion 0 failed in connection_dns_remove at src/or/dns.c:1136. Stack trace:
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(log_backtrace+0x29) [0x50dbe9]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(log_backtrace+0x29) [0x50dbe9]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(tor_assertion_failed_+0x7a) [0x51af6a]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(tor_assertion_failed_+0x7a) [0x51af6a]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(connection_dns_remove+0x240) [0x4fa6e0]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(connection_dns_remove+0x240) [0x4fa6e0]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x428ed9]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x428ed9]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x429209]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x429209]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/lib64/libevent-2.0.so.5(event_base_loop+0x40c) [0x3758b7d8f5c]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/lib64/libevent-2.0.so.5(event_base_loop+0x40c) [0x3758b7d8f5c]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(do_main_loop+0x215) [0x42afd5]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(do_main_loop+0x215) [0x42afd5]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(tor_main+0x15d5) [0x42dc05]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(tor_main+0x15d5) [0x42dc05]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /lib64/libc.so.6(libc_start_main+0xf5) [0x3758abecab5]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /lib64/libc.so.6(
libc_start_main+0xf5) [0x3758abecab5]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x427e31]
Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x427e31]

I have been having issues with DNS connectivity due to the high packet (~100k pps) rate, which means tor has been without consistent DNS for about a half hour before this died.

Having tor crash was extremely surprising.

I am running tor on Gentoo, tor version:

# tor --version
Tor version 0.2.6.1-alpha (git-5a601dd2901644a5).

My suspicion is that DNS requests piled up in an internal tor buffer of some sort, and that got maxed out resulting in an oops.

I'll try to keep an eye on this. Let me know if more information is needed for debugging.

Child Tickets

Change History (14)

comment:1 Changed 4 years ago by nickm

Keywords: tor-relay 025-backport added
Milestone: Tor: 0.2.6.x-final
Priority: normalmajor

comment:2 Changed 4 years ago by nickm

The failure is the assertion failure in dns.c. It looks like connection_dns_remove was called on an edge connection that wasn't actually pending according to the DNS code.

I don't see any harm in turning this from an "assert(0)" into a warning message.

I don't see any easy way for a UDP flood to have triggered this assertion, though. Very interesting.

comment:3 Changed 4 years ago by nickm

Summary: UDP DoS attack results in tor crashAssertion failure in dns.c, possibly connected to UDP DoS attack

comment:4 Changed 4 years ago by nickm

(Also, I'd strongly suggest that if you're running an exit node, you should probably set up your own local caching DNS server. Don't use google's DNS: that sends google a list of every hostname your users try to resolve.)

comment:5 Changed 4 years ago by nickm

Status: newneeds_review

I've added a branch "bug14129_diagnostic_025" in my public repository. It doesn't fix the underlying issue, but it replaces the assertion with a log message in hopes of better diagnosing what's up. Needs review. Should we merge this to master?

comment:6 in reply to:  5 Changed 4 years ago by cypherpunks

Replying to nickm:

It doesn't fix the underlying issue

The underlying issue is probably about purge_expired_resolves and already marked for close pending connection. If so then next patch could help:

--- src/or/dns.c.original	2014-10-10 06:06:24.000000000 -0700
+++ src/or/dns.c	2015-01-08 00:17:02.289666401 -0800
@@ -558,6 +558,8 @@
         /* Connections should only be pending if they have no socket. */
         tor_assert(!SOCKET_OK(pend->conn->base_.s));
         pendconn = pend->conn;
+        /* prevent double-remove. */
+        pendconn->base_.state = EXIT_CONN_STATE_RESOLVEFAILED;
         if (!pendconn->base_.marked_for_close) {
           connection_edge_end(pendconn, END_STREAM_REASON_TIMEOUT);
           circuit_detach_stream(circuit_get_by_edge_conn(pendconn), pendconn);

comment:7 Changed 4 years ago by nickm

Thanks, cypherpunks!

comment:8 Changed 4 years ago by nickm

Keywords: 024-backport added

comment:9 Changed 4 years ago by nickm

Branch "bug14129_024" has a simplified version of my patch, plus the patch from cypherpunks. I'm planning to merge to 0.2.5 and later, and mark for possible backport to 0.2.4.

comment:10 Changed 4 years ago by jowr

Nice, thanks guys. Fast turnaround.

And yes, I put a caching DNS server back up. I disabled it for some reason...

++ Cache Statistics ++
[View: default]

38506257 cache hits
3766166 cache misses

The UDP flood didn't directly impact tor itself, but took out DNS as a secondary effect due to either upstream silliness or some value of weirdness within the system itself. I'm more inclined to think its' the former than the latter.

I've moved DNS into a caching nameserver, and am using ipv6 resolvers as primary targets so that IPv4 UDP silliness doesn't cause me any further issues.

comment:11 Changed 4 years ago by nickm

Milestone: Tor: 0.2.6.x-finalTor: 0.2.4.x-final

(No reviews in 3 days, and this still seems pretty important. Merging to 0.2.5 and later; marking for possible 0.2.4 backport.)

comment:12 Changed 4 years ago by nickm

Resolution: fixed
Status: needs_reviewclosed

This is small and caused no problems in 0.2.5 or later and could conceivably be a DoS. Taking this in 0.2.4.

comment:13 Changed 4 years ago by weasel

Keywords: CVE-assigned added

CVE-2015-2689

comment:14 Changed 3 years ago by mikeperry

Keywords: tor-dos added; dos removed

Canonicalize dos tag to tor-dos

Note: See TracTickets for help on using tickets.