Opened 4 weeks ago

Last modified 4 weeks ago

#28078 needs_information defect

nyx hangs (sometimes) when tor process vanishes

Reported by: traumschule Owned by: atagar
Priority: Medium Milestone:
Component: Core Tor/Nyx Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

For the second time nyx became irresponsive after TB has been restarted (via update or after a crash).

What happened:
The process /usr/bin/python3 /usr/bin/nyx -i 127.0.0.1:9151 is connected to TB's tor instance. Some days ago TB upgraded itself and restarted. (Probably) since then nyx is no longer responding. Attaching to it reveals python hangs at PyThread_acquire_lock_timed.

# strace -p 27404
strace: Process 27404 attached
futex(0x9343b40, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 0xffffffff

I tried to attach to the process with pdb-clone after installing python-dbg:

# pdb-attach -p 27404                   
Starting gdb 8.1                                  
+++ -interpreter-exec console "where"           
+++ -gdb-exit                                                          
~"\n"                                                  
^done                                                         
(gdb)                                             
~"#0  0xb7f1cd09 in __kernel_vsyscall ()\n"
~"#1  0xb7ed12d6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x9343b40) at ../sysdeps/unix/sysv/linux/futex-internal.h:205\n"
~"#2  do_futex_wait (sem=sem@entry=0x9343b40, abstime=0x0) at sem_waitcommon.c:115\n"
~"#3  0xb7ed13d7 in __new_sem_wait_slow (sem=0x9343b40, abstime=0x0) at sem_waitcommon.c:282\n"
~"#4  0x081e98cc in PyThread_acquire_lock_timed ()\n"
~"#5  0x0821a63d in ?? ()\n"
~"#6  0x081ab38b in _PyCFunction_FastCallDict ()\n"
~"#7  0x081dc4ed in PyObject_CallFunctionObjArgs ()\n"
~"#8  0x08152cf2 in _PyEval_EvalFrameDefault ()\n"
~"#9  0x0814f4c2 in ?? ()\n"
~"#10 0x081500a0 in ?? ()\n"
~"#11 0x0815186b in _PyEval_EvalFrameDefault ()\n"
~"#12 0x0814d785 in ?? ()\n"
~"#13 0x081ca5be in ?? ()\n"
~"#14 0x081e1054 in PyObject_Call ()\n"
~"#15 0x08152c05 in _PyEval_EvalFrameDefault ()\n"
~"#16 0x0814df2d in ?? ()\n"
~"#17 0x0814f76f in ?? ()\n"
~"#18 0x081500a0 in ?? ()\n"
~"#19 0x0815186b in _PyEval_EvalFrameDefault ()\n"
~"#20 0x0814f4c2 in ?? ()\n"
~"#21 0x081500a0 in ?? ()\n"
~"#22 0x0815186b in _PyEval_EvalFrameDefault ()\n"
~"#23 0x0814f4c2 in ?? ()\n"
~"#24 0x081500a0 in ?? ()\n"
~"#25 0x0815186b in _PyEval_EvalFrameDefault ()\n"
~"#26 0x0814f4c2 in ?? ()\n"
~"#27 0x081500a0 in ?? ()\n"
~"#28 0x0815186b in _PyEval_EvalFrameDefault ()\n"
~"#29 0x0814d785 in ?? ()\n"
~"#30 0x0814f76f in ?? ()\n"
~"#31 0x081500a0 in ?? ()\n"
~"#32 0x08152419 in _PyEval_EvalFrameDefault ()\n"
~"#33 0x0814d785 in ?? ()\n"
~"#34 0x0814f76f in ?? ()\n"
~"#35 0x081500a0 in ?? ()\n"
~"#36 0x08152419 in _PyEval_EvalFrameDefault ()\n"
~"#37 0x0814d785 in ?? ()\n"
~"#38 0x0814f76f in ?? ()\n"
~"#39 0x081500a0 in ?? ()\n"
~"#40 0x08152419 in _PyEval_EvalFrameDefault ()\n"
~"#41 0x0814f4c2 in ?? ()\n"
~"#42 0x081500a0 in ?? ()\n"
~"#43 0x0815186b in _PyEval_EvalFrameDefault ()\n"
~"#44 0x0814df2d in ?? ()\n"
~"#45 0x081ca5be in ?? ()\n"
~"#46 0x081e1054 in PyObject_Call ()\n"
~"#47 0x08152c05 in _PyEval_EvalFrameDefault ()\n"
~"#48 0x0814d785 in ?? ()\n"
~"#49 0x0814f76f in ?? ()\n"
~"#50 0x081500a0 in ?? ()\n"
~"#51 0x0815186b in _PyEval_EvalFrameDefault ()\n"
~"#52 0x0814df2d in ?? ()\n"
~"#53 0x0814f76f in ?? ()\n"
~"#54 0x081500a0 in ?? ()\n"
~"#55 0x08152419 in _PyEval_EvalFrameDefault ()\n"
~"#56 0x0814d785 in ?? ()\n"
~"#57 0x081ca69f in ?? ()\n"
~"#58 0x081e1054 in PyObject_Call ()\n"
~"#59 0x08152c05 in _PyEval_EvalFrameDefault ()\n"
~"#60 0x0814df2d in ?? ()\n"
~"#61 0x0814f76f in ?? ()\n"
~"#62 0x081500a0 in ?? ()\n"
~"#63 0x0815186b in _PyEval_EvalFrameDefault ()\n"
~"#64 0x0814f4c2 in ?? ()\n"
~"#65 0x081500a0 in ?? ()\n"
~"#66 0x0815186b in _PyEval_EvalFrameDefault ()\n"
~"#67 0x0814d785 in ?? ()\n"
~"#68 0x0815097c in PyEval_EvalCode ()\n"
~"#69 0x08267ea1 in ?? ()\n"
~"#70 0x08267f52 in PyRun_FileExFlags ()\n"
~"#71 0x0826b84f in PyRun_SimpleFileExFlags ()\n"
~"#72 0x0826c430 in Py_Main ()\n"
~"#73 0x080f5c50 in main ()\n"
^done
(gdb)
^exit
Unable to setup pdb for remote debugging.
Please use a Python program built with debugging symbols.

Seems i need to restart nyx. Will keep it running though in case you have any idea to inspect its variables.

Child Tickets

Attachments (2)

gdb-nyx.log (5.2 KB) - added by traumschule 4 weeks ago.
gdb log with missing debug symbols
strace_nyx.log (11.8 KB) - added by traumschule 4 weeks ago.
kept strace attached for some time, unfortunately nyx didn't recover

Download all attachments as: .zip

Change History (5)

Changed 4 weeks ago by traumschule

Attachment: gdb-nyx.log added

gdb log with missing debug symbols

Changed 4 weeks ago by traumschule

Attachment: strace_nyx.log added

kept strace attached for some time, unfortunately nyx didn't recover

comment:1 Changed 4 weeks ago by atagar

Hi traumschule, would you mind adding the '--debug' argument to Nyx? That will generate a trace level request/reply log with your tor process. Hopefully that'll give a bit more of a hint on where we're hanging.

comment:2 Changed 4 weeks ago by traumschule

Status: newneeds_information

yes! should have done this from the beginning. Now i start it with pdb ./run_nyx -d nyx-system-tor.log but i wonder what will happen when nyx hangs again, since it's in the same window i won't have a chance to interact with pdb i guess, the log may help still.

comment:3 Changed 4 weeks ago by atagar

Personally I never use pdb, but the hanging shouldn't matter to the log. If we're lucky the messages emitted before and when nyx hangs should hopefully give a clue about where I buggered up.

Note: See TracTickets for help on using tickets.