Opened 8 years ago

Closed 7 years ago

Last modified 7 years ago

#5320 closed defect (user disappeared)

My relay is crashing with segfault in libevent.

Reported by: b0b3r Owned by:
Priority: Medium Milestone: Tor: 0.2.3.x-final
Component: Core Tor/Tor Version: Tor: 0.2.2.35
Severity: Keywords: tor-relay
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Hello

My relay is crashing with segfault in libevent after a random time:
...
tor[22467]: segfault at 18 ip 00007fa6be6a4a4e sp 00007fff9951bc00 error 4 in libevent-2.0.so.5.1.4[7fa6be675000+42000]
tor[23459]: segfault at 18 ip 00007f7668d28a4e sp 00007fff6fc2bb70 error 4 in libevent-2.0.so.5.1.4[7f7668cf9000+42000]
tor[26090]: segfault at 18 ip 00007fb0b1d22a4e sp 00007fffe901dd50 error 4 in libevent-2.0.so.5.1.4[7fb0b1cf3000+42000]
...

It's running on gentoo/linux:
tor-0.2.2.35
glibc-2.13
zlib-1.2.5
openssl-1.0.0g
libevent-2.0.16 and 2.0.17
Symptoms are the same on the two very stable servers. The first is a 64-bit with libevent-2.0.16, the second 32bit with libevent-2.0.17. Other software is in the same version all compiled with gcc-4.5.3

tail -n 25 tor.log (after crash):
...
Mar 06 12:46:06.290 [debug] append_cell_to_circuit_queue(): Primed a buffer.
Mar 06 12:46:06.290 [debug] connection_or_flush_from_first_active_circuit(): Made a circuit inactive.
Mar 06 12:46:06.290 [debug] connection_or_process_cells_from_inbuf(): 20: starting, inbuf_datalen 0 (0 pending in tls object).
Mar 06 12:46:06.290 [debug] conn_write_callback(): socket 77 wants to write.
Mar 06 12:46:06.290 [debug] flush_chunk_tls(): flushed 512 bytes, 0 ready to flush, 0 remain.
Mar 06 12:46:06.290 [debug] connection_handle_write_impl(): After TLS write of 512: 0 read, 586 written
Mar 06 12:46:06.291 [debug] conn_read_callback(): socket 77 wants to read.
Mar 06 12:46:06.291 [debug] connection_read_to_buf(): 77: starting, inbuf_datalen 0 (0 pending in tls object). at_most 16384.
Mar 06 12:46:06.291 [debug] connection_read_to_buf(): After TLS read of 512: 586 read, 0 written
Mar 06 12:46:06.291 [debug] connection_or_process_cells_from_inbuf(): 77: starting, inbuf_datalen 512 (0 pending in tls object).
Mar 06 12:46:06.291 [debug] circuit_receive_relay_cell(): Passing on unrecognized cell.
Mar 06 12:46:06.291 [debug] append_cell_to_circuit_queue(): Made a circuit active.
Mar 06 12:46:06.291 [debug] append_cell_to_circuit_queue(): Primed a buffer.
Mar 06 12:46:06.291 [debug] connection_or_flush_from_first_active_circuit(): Made a circuit inactive.
Mar 06 12:46:06.291 [debug] connection_or_process_cells_from_inbuf(): 77: starting, inbuf_datalen 0 (0 pending in tls object).
Mar 06 12:46:06.291 [debug] conn_read_callback(): socket 77 wants to read.
Mar 06 12:46:06.291 [debug] connection_read_to_buf(): 77: starting, inbuf_datalen 0 (0 pending in tls object). at_most 16384.
Mar 06 12:46:06.291 [debug] tor_tls_read(): read returned r=-1, err=-2
Mar 06 12:46:06.291 [debug] connection_read_to_buf(): After TLS read of 0: 874 read, 0 written
Mar 06 12:46:06.291 [debug] connection_or_process_cells_from_inbuf(): 77: starting, inbuf_datalen 0 (0 pending in tls object).
Mar 06 12:46:06.291 [debug] conn_write_callback(): socket 20 wants to write.
Mar 06 12:46:06.291 [debug] flush_chunk_tls(): flushed 512 bytes, 0 ready to flush, 0 remain.
Mar 06 12:46:06.291 [debug] connection_handle_write_impl(): After TLS write of 512: 0 read, 586 written
Mar 06 12:46:06.314 [info] eventdns: Nameserver 195.191.233.252:53 is back up
Mar 06 12:46:06.314 [info] eventdns: Removing timeout for request 0x1b0aec0

torrc:
User tor
SocksPort 0
SocksPolicy reject *
Log debug file /var/log/tor/tor.log
DataDirectory /var/lib/tor/data
ControlPort 9051
ORPort 443
ORListenAddress 195.191.233.220:443
OutboundBindAddress 195.191.233.220
Nickname b0b3r
Address 195.191.233.220
ContactInfo b0b3r <b0b3r(at)b0b3r(dot)pl>
DirPort 80
DirListenAddress 195.191.233.220:80
DirPortFrontPage /etc/tor/exit-notice.html
CellStatistics 1
DirReqStatistics 1
EntryStatistics 1
ExitPortStatistics 1
ExtraInfoStatistics 1
HashedControlPassword (blanked)
RelayBandwidthRate 1 MB
RelayBandwidthBurst 3 MB
ExitPolicy accept *:20-23
ExitPolicy accept *:43
ExitPolicy accept *:53
ExitPolicy accept *:79-81
ExitPolicy accept *:88
ExitPolicy accept *:110
ExitPolicy accept *:143
ExitPolicy accept *:220
ExitPolicy accept *:443
ExitPolicy accept *:464-465
ExitPolicy accept *:543-544
ExitPolicy accept *:563
ExitPolicy accept *:587
ExitPolicy accept *:706
ExitPolicy accept *:749
ExitPolicy accept *:873
ExitPolicy accept *:902-904
ExitPolicy accept *:981
ExitPolicy accept *:989-995
ExitPolicy accept *:1194
ExitPolicy accept *:1220
ExitPolicy accept *:1293
ExitPolicy accept *:1500
ExitPolicy accept *:1723
ExitPolicy accept *:1863
ExitPolicy accept *:2082-2083
ExitPolicy accept *:2086-2087
ExitPolicy accept *:2095-2096
ExitPolicy accept *:3128
ExitPolicy accept *:3389
ExitPolicy accept *:3690
ExitPolicy accept *:4321
ExitPolicy accept *:4643
ExitPolicy accept *:5050
ExitPolicy accept *:5190
ExitPolicy accept *:5222-5223
ExitPolicy accept *:5900
ExitPolicy accept *:6666-6667
ExitPolicy accept *:6679
ExitPolicy accept *:6697
ExitPolicy accept *:8000
ExitPolicy accept *:8008
ExitPolicy accept *:8080
ExitPolicy accept *:8087-8088
ExitPolicy accept *:8443
ExitPolicy accept *:8888
ExitPolicy accept *:9418
ExitPolicy accept *:9999-10000
ExitPolicy accept *:19638
ExitPolicy reject *:*

Child Tickets

Attachments (1)

libevent-evdns-crash.diff (1.7 KB) - added by nickm 8 years ago.

Download all attachments as: .zip

Change History (10)

comment:1 Changed 8 years ago by b0b3r

Component: - Select a componentTor Relay

comment:2 Changed 8 years ago by nickm

Any chance of getting a stack trace here? The crash info alone doesn't tell us where the segfault actually happened.

comment:3 Changed 8 years ago by b0b3r

Finally I got one. Looks like a problem with communication with resolvers. I have three nameservers and one search domain in /etc/resolv.conf.

#0 reply_handle (req=0xa2b7f48, flags=<optimized out>, ttl=0, reply=0x0) at evdns.c:903
#1 0xb76e16e6 in reply_parse (length=260, packet=0xbf97edd4 "\016\263\201\200", base=0x9790138) at evdns.c:1181
#2 nameserver_read (ns=0x9790438) at evdns.c:1374
#3 0xb76e1fc7 in nameserver_ready_callback (fd=12, events=2, arg=0x9790438) at evdns.c:1479
#4 0xb76bb1f0 in event_process_active_single_queue (activeq=<optimized out>, base=<optimized out>) at event.c:1340
#5 event_process_active (base=<optimized out>) at event.c:1407
#6 event_base_loop (base=0x950b8a8, flags=0) at event.c:1604
#7 0x08051af9 in do_main_loop () at main.c:1566
#8 0x080530a5 in tor_main (argc=7, argv=0xbf97fb44) at main.c:2228
#9 0x0804e29b in main (argc=7, argv=0xbf97fb44) at tor_main.c:30

I can provide a full dump, but it may contain some sensitive data.

comment:4 Changed 8 years ago by nickm

Oho! That looks a lot like a bug that's fixed in the patches-2.0 branch in the Libevent repository. If you can check it out easily and confirm, that would be great. Let me know if you need a .tar.gz file or something. Otherwise, the likely fix will appear in Libevent 2.0.18-stable.

comment:5 Changed 8 years ago by nickm

Attaching the relevant libevent patch too, in case you want to apply it by hand.

Changed 8 years ago by nickm

Attachment: libevent-evdns-crash.diff added

comment:6 Changed 7 years ago by nickm

Milestone: Tor: 0.2.3.x-final
Status: newneeds_information

Did that fix it for you?

comment:7 Changed 7 years ago by nickm

Resolution: user disappeared
Status: needs_informationclosed

I believe this is fixed by recent libevents. Please reopen if not.

comment:8 Changed 7 years ago by nickm

Keywords: tor-relay added

comment:9 Changed 7 years ago by nickm

Component: Tor RelayTor
Note: See TracTickets for help on using tickets.