Opened 8 years ago

Closed 3 years ago

#5102 closed defect (user disappeared)

segfault in entry_guard_register_connect_status on tor bridge running obfsproxy on openbsd

Reported by: therealditzydoo Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version: Tor: 0.2.3.11-alpha
Severity: Normal Keywords: tor-bridge
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I am running a tor bridge on openbsd (uname -a output is OpenBSD [REDACTED] 5.1 GENERIC.MP#2 i386). It is statically linked and runs in a chroot. Here's the output when it's started not in the chroot:

Feb 12 05:53:04.331 [notice] Tor v0.2.3.11-alpha running on OpenBSD i386.
Feb 12 05:53:04.331 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Feb 12 05:53:04.331 [notice] This version is not a stable Tor release. Expect more bugs than usual.
Feb 12 05:53:04.347 [notice] Configuration file "/usr/local/etc/tor/torrc" not present, using reasonable defaults.
Feb 12 05:53:04.349 [warn] It's a little hard to tell, but you seem to have Libevent 1.4.0-beta header files, whereas you have linked against Libevent 1.4.14b-stable.  This will probably make Tor crash.
Feb 12 05:53:04.349 [notice] Initialized libevent version 1.4.14b-stable using method kqueue. Good.
Feb 12 05:53:04.349 [notice] Opening Socks listener on 127.0.0.1:9050
Feb 12 05:53:04.000 [notice] Parsing GEOIP file /usr/local/share/tor/geoip.
Feb 12 05:53:04.000 [notice] No AES engine found; using AES_* functions.
Feb 12 05:53:04.000 [notice] This OpenSSL has a good implementation of counter mode; using it.
Feb 12 05:53:04.000 [notice] OpenSSL OpenSSL 1.0.0f 4 Jan 2012 looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Feb 12 05:53:04.000 [notice] Reloaded microdescriptor cache.  Found 3404 descriptors.
Feb 12 05:53:05.000 [notice] We now have enough directory information to build circuits.
Feb 12 05:53:05.000 [notice] Bootstrapped 80%: Connecting to the Tor network.
Feb 12 05:53:06.000 [notice] Heartbeat: Tor's uptime is 0:00 hours, with 2 circuits open. I've sent 0 kB and received 0 kB.
Feb 12 05:53:06.000 [notice] Bootstrapped 85%: Finishing handshake with first hop.
Feb 12 05:53:07.000 [notice] Bootstrapped 90%: Establishing a Tor circuit.
Feb 12 05:53:09.000 [notice] Tor has successfully opened a circuit. Looks like client functionality is working.
Feb 12 05:53:09.000 [notice] Bootstrapped 100%: Done.
^CFeb 12 05:56:54.000 [notice] Interrupt: exiting cleanly.

When run in the chroot (with chroot -u _tor -g _tor /home/chrooted/tor /bin/tor -f /etc/tor/torrc-relay), it runs for a bit, then crashes without leaving anything in the logfile. It dumps a core. Here's the output of bt from gdb:

> gdb mytor mycore 
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-openbsd5.0"...
Core was generated by `tor'.
Program terminated with signal 11, Segmentation fault.
#0  0x1c07b57b in entry_guard_register_connect_status ()
(gdb) bt
#0  0x1c07b57b in entry_guard_register_connect_status ()
#1  0x1c0ba387 in connection_or_set_state_open ()
#2  0x1c08bea5 in command_process_netinfo_cell ()
#3  0x1c08988d in command_process_cell ()
#4  0x1c0baa51 in connection_or_process_cells_from_inbuf ()
#5  0x1c0b7578 in connection_or_process_inbuf ()
#6  0x1c0a91db in connection_process_inbuf ()
#7  0x1c0a6e7a in connection_handle_read_impl ()
#8  0x1c0a6f94 in connection_handle_read ()
#9  0x1c001cb0 in conn_read_callback ()
#10 0x1c137b35 in event_base_loop (base=0x83cda000, flags=0) at /usr/src/lib/libevent/event.c:402
#11 0x1c0045e7 in do_main_loop ()
#12 0x1c005cf7 in tor_main ()
#13 0x1c000406 in main ()
(gdb)

Child Tickets

Change History (17)

comment:1 Changed 8 years ago by therealditzydoo

Here is the output after being compiled with -g -O0:

GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-openbsd5.0"...
Core was generated by `tor'.
Program terminated with signal 11, Segmentation fault.
#0  0x1c07b57b in entry_guard_register_connect_status (digest=0x7d0e9b68 "%ֲ$ⲯ�0370s-[", succeeded=1, 
    mark_relay_status=0, now=1329059709) at circuitbuild.c:3909
3909      SMARTLIST_FOREACH(entry_guards, entry_guard_t *, e,
(gdb) bt
#0  0x1c07b57b in entry_guard_register_connect_status (digest=0x7d0e9b68 "%ֲ$ⲯ�0370s-[", succeeded=1, 
    mark_relay_status=0, now=1329059709) at circuitbuild.c:3909
#1  0x1c0ba387 in connection_or_set_state_open (conn=0x7d0e9b00) at connection_or.c:1700
#2  0x1c08bea5 in command_process_netinfo_cell (cell=0xcfbc0620, conn=0x7d0e9b00) at command.c:916
#3  0x1c08988d in command_process_cell (cell=0xcfbc0620, conn=0x7d0e9b00) at command.c:201
#4  0x1c0baa51 in connection_or_process_cells_from_inbuf (conn=0x7d0e9b00) at connection_or.c:1832
#5  0x1c0b7578 in connection_or_process_inbuf (conn=0x7d0e9b00) at connection_or.c:390
#6  0x1c0a91db in connection_process_inbuf (conn=0x7d0e9b00, package_partial=1) at connection.c:3760
#7  0x1c0a6e7a in connection_handle_read_impl (conn=0x7d0e9b00) at connection.c:2656
#8  0x1c0a6f94 in connection_handle_read (conn=0x7d0e9b00) at connection.c:2697
#9  0x1c001cb0 in conn_read_callback (fd=97, event=2, _conn=0x7d0e9b00) at main.c:702
#10 0x1c137b35 in event_base_loop (base=0x87c47800, flags=0) at /usr/src/lib/libevent/event.c:402
#11 0x1c0045e7 in do_main_loop () at main.c:1924
#12 0x1c005cf7 in tor_main (argc=3, argv=0xcfbc0cc4) at main.c:2619
#13 0x1c000406 in main (argc=Cannot access memory at address 0x0
) at tor_main.c:30
(gdb) 

comment:2 Changed 8 years ago by therealditzydoo

I'm using the script shown below to auto-restart tor so there isn't much downtime on my host:

#!/bin/sh
while true;
do true
sleep 1
if ! ps aux|grep [t]orrc-relay > /dev/null
then
echo -n "RESPAWNING AT" 
date
chroot -u _tor -g _tor /home/chrooted/tor /bin/tor -f /etc/tor/torrc-relay
fi

done

comment:3 Changed 8 years ago by therealditzydoo

Still fails when running with a recent libevent:

GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-openbsd5.0"...

warning: exec file is newer than core file.
Core was generated by `tor'.
Program terminated with signal 11, Segmentation fault.
#0  0x1c07347f in entry_guard_register_connect_status (digest=0x7cd57e68 "001\030��>�bR7@\202�TPa�\177", succeeded=1, 
    mark_relay_status=0, now=1329073119) at circuitbuild.c:3909
3909      SMARTLIST_FOREACH(entry_guards, entry_guard_t *, e,
(gdb) bt
#0  0x1c07347f in entry_guard_register_connect_status (digest=0x7cd57e68 "001\030��>�bR7@\202�TPa�\177", succeeded=1, 
    mark_relay_status=0, now=1329073119) at circuitbuild.c:3909
#1  0x1c0b228b in connection_or_set_state_open (conn=0x7cd57e00) at connection_or.c:1700
#2  0x1c083da9 in command_process_netinfo_cell (cell=0xcfbda160, conn=0x7cd57e00) at command.c:916
#3  0x1c081791 in command_process_cell (cell=0xcfbda160, conn=0x7cd57e00) at command.c:201
#4  0x1c0b2955 in connection_or_process_cells_from_inbuf (conn=0x7cd57e00) at connection_or.c:1832
#5  0x1c0af47c in connection_or_process_inbuf (conn=0x7cd57e00) at connection_or.c:390
#6  0x1c0a10df in connection_process_inbuf (conn=0x7cd57e00, package_partial=1) at connection.c:3760
#7  0x1c09ed7e in connection_handle_read_impl (conn=0x7cd57e00) at connection.c:2656
#8  0x1c09ee98 in connection_handle_read (conn=0x7cd57e00) at connection.c:2697
#9  0x1c001cb0 in conn_read_callback (fd=216, event=2, _conn=0x7cd57e00) at main.c:702
#10 0x1c133a4a in event_base_loop (base=0x7cb8d000, flags=0) at event.c:1340
#11 0x1c0045e7 in do_main_loop () at main.c:1924
#12 0x1c005cff in tor_main (argc=3, argv=0xcfbda820) at main.c:2619
#13 0x1c000406 in main (argc=Cannot access memory at address 0x501
) at tor_main.c:30
(gdb) 

comment:4 Changed 8 years ago by nickm

I'm turning that SMARTLIST_FOREACH into a SMARTLIST_FOREACH_BEGIN/END pair in 61452299d1067298a2865deb6398b1fb269b2a81; that should make the line number a tiny bit easier to find.

If you can find the value of the local variable "e" at the point of the crash, as well as the values of *e, entry_guards, and *entry_guards, that would be great.

But looking at the code, it seems the only possibilities are:

  • one of the entries in entry_guards has become null, or corrupt, or something like that.
  • The entry_guards smartlist itself has somehow gotten messed up.

comment:5 Changed 8 years ago by therealditzydoo

GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-openbsd5.0"...

warning: exec file is newer than core file.
Core was generated by `tor'.
Program terminated with signal 11, Segmentation fault.
#0  0x1c07347f in entry_guard_register_connect_status (digest=0x7cd57e68 "001\030��>�bR7@\202�TPa�\177", 
    succeeded=1, mark_relay_status=0, now=1329073119) at circuitbuild.c:3909
3909      SMARTLIST_FOREACH(entry_guards, entry_guard_t *, e,
(gdb) p e
$1 = (entry_guard_t *) 0x1d3f9c2
(gdb) p *e
Cannot access memory at address 0x1d3f9c2
(gdb) p entry_guard
No symbol "entry_guard" in current context.
(gdb) p entry_guards
$2 = (smartlist_t *) 0x862ac856
(gdb) p *entry_guards                                                                                                 
$3 = {list = 0xc2e07cb8, num_used = 34346, capacity = 842137600}
(gdb) 

comment:6 Changed 8 years ago by arma

"num_used = 34346, capacity = 842137600"

Looks like entry_guards got clobbered somewhere.

comment:7 Changed 8 years ago by arma

"warning: exec file is newer than core file." makes me wonder if we're getting good data though.

comment:8 in reply to:  7 Changed 8 years ago by therealditzydoo

Replying to arma:

"warning: exec file is newer than core file." makes me wonder if we're getting good data though.

This is 'cause I used cp to copy the executable but mv to copy the core file. cp makes a new timestamp while mv does not.

comment:9 Changed 8 years ago by arma

Ok. This happens reproducibly?

Is there anything like valgrind for bsd?

comment:10 Changed 7 years ago by nickm

Milestone: Tor: 0.2.3.x-final

comment:11 Changed 7 years ago by nickm

Status: newneeds_information

comment:12 Changed 7 years ago by nickm

Has this happened with any later version of Tor?

comment:13 Changed 7 years ago by nickm

Keywords: tor-bridge added

comment:14 Changed 7 years ago by nickm

Component: Tor BridgeTor

comment:15 Changed 7 years ago by nickm

Milestone: Tor: 0.2.3.x-finalTor: unspecified

Ping? Did this go away, or is anybody still seeing this? Does the new version of Tor get a real line number for the crash?

Oh, here's another question: If this still happens, what's in your torrc?

comment:16 Changed 3 years ago by arma

Severity: Normal

Recommend closing with user-disappeared. This is a way old version of Tor now.

comment:17 Changed 3 years ago by nickm

Resolution: user disappeared
Status: needs_informationclosed

ack

Note: See TracTickets for help on using tickets.