Opened 6 years ago

Closed 5 years ago

#8575 closed defect (worksforme)

Integ tests segfault python 3

Reported by: atagar Owned by: atagar
Priority: Medium Milestone:
Component: Core Tor/Stem Version:
Severity: Keywords: testing
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Both stem's unit and integration tests presently pass under the python 3.x series. However, when the integration tests are finished running python spews a dump and segfaults...

*** glibc detected *** python3: munmap_chunk(): invalid pointer: 0x0a1ca610 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x6f751)[0x302751]
/lib/i386-linux-gnu/libc.so.6(+0x6fefe)[0x302efe]
python3(main+0x19e)[0x805c08e]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x2a9e37]
python3[0x805be51]
======= Memory map: ========
00110000-00117000 r-xp 00000000 08:05 6037079    /lib/i386-linux-gnu/librt-2.13.so
00117000-00118000 r--p 00006000 08:05 6037079    /lib/i386-linux-gnu/librt-2.13.so
00118000-00119000 rw-p 00007000 08:05 6037079    /lib/i386-linux-gnu/librt-2.13.so
00119000-0011b000 r-xp 00000000 00:13 3451724    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/grp.cpython-33m.so
0011b000-0011c000 r--p 00001000 00:13 3451724    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/grp.cpython-33m.so
0011c000-0011d000 rw-p 00002000 00:13 3451724    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/grp.cpython-33m.so
0011d000-00120000 r-xp 00000000 00:13 3451742    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_posixsubprocess.cpython-33m.so
00120000-00121000 r--p 00002000 00:13 3451742    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_posixsubprocess.cpython-33m.so
00121000-00122000 rw-p 00003000 00:13 3451742    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_posixsubprocess.cpython-33m.so
00122000-00135000 r-xp 00000000 08:05 6030228    /lib/i386-linux-gnu/libz.so.1.2.3.4
00135000-00136000 r--p 00012000 08:05 6030228    /lib/i386-linux-gnu/libz.so.1.2.3.4
00136000-00137000 rw-p 00013000 08:05 6030228    /lib/i386-linux-gnu/libz.so.1.2.3.4
00137000-0013a000 r-xp 00000000 00:13 3451722    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/fcntl.cpython-33m.so
0013a000-0013b000 r--p 00002000 00:13 3451722    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/fcntl.cpython-33m.so
0013b000-0013c000 rw-p 00003000 00:13 3451722    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/fcntl.cpython-33m.so
0013c000-00140000 r-xp 00000000 00:13 3451766    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/binascii.cpython-33m.so
00140000-00141000 r--p 00003000 00:13 3451766    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/binascii.cpython-33m.so
00141000-00142000 rw-p 00004000 00:13 3451766    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/binascii.cpython-33m.so
00142000-00147000 r-xp 00000000 00:13 3451740    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_csv.cpython-33m.so
00147000-00148000 r--p 00004000 00:13 3451740    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_csv.cpython-33m.so
00148000-0014a000 rw-p 00005000 00:13 3451740    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_csv.cpython-33m.so
0014a000-001ef000 r-xp 00000000 00:13 3451720    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/unicodedata.cpython-33m.so
001ef000-001f0000 r--p 000a4000 00:13 3451720    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/unicodedata.cpython-33m.so
001f0000-00206000 rw-p 000a5000 00:13 3451720    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/unicodedata.cpython-33m.so
00206000-0020c000 r-xp 00000000 08:05 6037082    /lib/i386-linux-gnu/libnss_compat-2.13.so
0020c000-0020d000 r--p 00005000 08:05 6037082    /lib/i386-linux-gnu/libnss_compat-2.13.so
0020d000-0020e000 rw-p 00006000 08:05 6037082    /lib/i386-linux-gnu/libnss_compat-2.13.so
0020e000-00221000 r-xp 00000000 08:05 6037076    /lib/i386-linux-gnu/libnsl-2.13.so
00221000-00222000 r--p 00012000 08:05 6037076    /lib/i386-linux-gnu/libnsl-2.13.so
00222000-00223000 rw-p 00013000 08:05 6037076    /lib/i386-linux-gnu/libnsl-2.13.so
00223000-00225000 rw-p 00000000 00:00 0 
00225000-0022f000 r-xp 00000000 08:05 6037070    /lib/i386-linux-gnu/libnss_files-2.13.so
0022f000-00230000 r--p 00009000 08:05 6037070    /lib/i386-linux-gnu/libnss_files-2.13.so
00230000-00231000 rw-p 0000a000 08:05 6037070    /lib/i386-linux-gnu/libnss_files-2.13.so
00239000-00255000 r-xp 00000000 08:05 6037078    /lib/i386-linux-gnu/ld-2.13.so
00255000-00256000 r--p 0001b000 08:05 6037078    /lib/i386-linux-gnu/ld-2.13.so
00256000-00257000 rw-p 0001c000 08:05 6037078    /lib/i386-linux-gnu/ld-2.13.so
0028f000-00291000 r-xp 00000000 08:05 6037071    /lib/i386-linux-gnu/libutil-2.13.so
00291000-00292000 r--p 00001000 08:05 6037071    /lib/i386-linux-gnu/libutil-2.13.so
00292000-00293000 rw-p 00002000 08:05 6037071    /lib/i386-linux-gnu/libutil-2.13.so
00293000-003f4000 r-xp 00000000 08:05 6037086    /lib/i386-linux-gnu/libc-2.13.so
003f4000-003f5000 ---p 00161000 08:05 6037086    /lib/i386-linux-gnu/libc-2.13.so
003f5000-003f7000 r--p 00161000 08:05 6037086    /lib/i386-linux-gnu/libc-2.13.so
003f7000-003f8000 rw-p 00163000 08:05 6037086    /lib/i386-linux-gnu/libc-2.13.so
003f8000-003fb000 rw-p 00000000 00:00 0 
0040c000-00428000 r-xp 00000000 00:13 3451955    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_ctypes.cpython-33m.so
00428000-0042c000 r--p 0001b000 00:13 3451955    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_ctypes.cpython-33m.so
0042c000-0042f000 rw-p 0001f000 00:13 3451955    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_ctypes.cpython-33m.so
00481000-004a5000 r-xp 00000000 08:05 6037088    /lib/i386-linux-gnu/libm-2.13.so
004a5000-004a6000 r--p 00023000 08:05 6037088    /lib/i386-linux-gnu/libm-2.13.so
004a6000-004a7000 rw-p 00024000 08:05 6037088    /lib/i386-linux-gnu/libm-2.13.so
004ed000-004ee000 r-xp 00000000 00:00 0          [vdso]
0058c000-0058e000 r-xp 00000000 08:05 6037081    /lib/i386-linux-gnu/libdl-2.13.so
0058e000-0058f000 r--p 00001000 08:05 6037081    /lib/i386-linux-gnu/libdl-2.13.so
0058f000-00590000 rw-p 00002000 08:05 6037081    /lib/i386-linux-gnu/libdl-2.13.so
005f6000-005f8000 r-xp 00000000 00:13 3451709    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/atexit.cpython-33m.so
005f8000-005f9000 r--p 00001000 00:13 3451709    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/atexit.cpython-33m.so
005f9000-005fa000 rw-p 00002000 00:13 3451709    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/atexit.cpython-33m.so
005fa000-0072c000 r-xp 00000000 08:05 6037050    /lib/libcrypto.so.0.9.8
0072c000-00734000 r--p 00132000 08:05 6037050    /lib/libcrypto.so.0.9.8
00734000-00743000 rw-p 0013a000 08:05 6037050    /lib/libcrypto.so.0.9.8
00743000-00746000 rw-p 00000000 00:00 0 
007e9000-007eb000 r-xp 00000000 00:13 3451702    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_bisect.cpython-33m.so
007eb000-007ec000 r--p 00001000 00:13 3451702    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_bisect.cpython-33m.so
007ec000-007ed000 rw-p 00002000 00:13 3451702    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_bisect.cpython-33m.so
00834000-00837000 r-xp 00000000 00:13 3451758    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/termios.cpython-33m.so
00837000-00838000 r--p 00002000 00:13 3451758    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/termios.cpython-33m.so
00838000-0083a000 rw-p 00003000 00:13 3451758    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/termios.cpython-33m.so
00940000-00946000 r-xp 00000000 00:13 3451684    /home/atagar/Python-3.3.0/build/lib.linux-i686-3.3/_struct.cpython-33m.soAborted

This happens to me with both Ubuntu's python3 deb (version 3.2) and version 3.3 I compiled from source while trying to sort a similar issue out with upstream.

At this point I'm kinda resigned to the conclusion that the 3.x series is still simply somewhat unstable. As Guido once said in a presentation I attended "If you get a stacktrace then that's your problem. If you manage to get python to segfault then that's almost certainly mine."

That said, if someone wants to contact upstream and troubleshoot these latest python 3.x segfaults then the help would be greatly appreciated!

Child Tickets

Change History (3)

comment:1 Changed 6 years ago by atagar

A potential cause of this is the stem.util.system integ tests which include a process renaming integ test. That test gets pretty low level, using ctypes to change the present process' name.

This would be pretty easy to either verify or rule out. Before contacting upstream we should do a binary search of our integ tests to narrow them down to the subset needed to reproduce this issue. This should quickly narrow the haystack quite a bit.

Another potential issue could be lingering threads. I once spent the better part of a month troubleshooting orphaned threads that caused TorCtl stacktraces while shutting arm. On very rare occasions (0.1%?) I've seen a similar issue with stem's reader or event daemons. From my read of the code those threads really *should* be getting joined. Unfortunately tracking down such a transient issue has been tricky.

All this said, my money's still on python3 instability. In short we should...

  • check that this still occurs with the latest development version of python 3.x
  • figure out the set of integration tests we need to run to reproduce this
  • contact upstream and work with them to figure out where the segfault is coming from

comment:2 Changed 5 years ago by Foxboron

This does not happen anymore on Python 3.4.2! All integ tests run without failure.

comment:3 Changed 5 years ago by atagar

Resolution: worksforme
Status: newclosed

Yup, I haven't seen this for several months either. I left this ticket open because the segfaults were very inconsistent, but I suppose we can reopen if this crops up again.

Note: See TracTickets for help on using tickets.