Opened 7 weeks ago

Last modified 7 weeks ago

#24116 needs_review defect

Torsocks deadlocks every Rust program

Reported by: larsl Owned by: dgoulet
Priority: Medium Milestone:
Component: Core Tor/Torsocks Version:
Severity: Major Keywords: torsocks deadlock rust jemalloc
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Any Rust program that is run with torsocks will deadlock. This has nothing to do with networking, even the program 'fn main() { }' compiled with a recent rustc will deadlock when run as 'torsocks ./rust_torsocks'.

This is a backtrace I got when attaching to the deadlocked process:

#0  0xb7713cf9 in __kernel_vsyscall ()
#1  0xb76b9d92 in __lll_lock_wait ()
  at ../sysdeps/unix/sysv/linux/i386/lowlevellock.S:144
#2  0xb76b38de in __GI___pthread_mutex_lock (mutex=0xb770d024)
  at ../nptl/pthread_mutex_[lock.c:80 lock.c:80]
#3  0xb77001ed in tsocks_mutex_lock ()
  from /usr/lib/i386-linux-gnu/torsocks/libtorsocks.so
#4  0xb7700334 in tsocks_once ()
  from /usr/lib/i386-linux-gnu/torsocks/libtorsocks.so
#5  0xb76fa25e in tsocks_initialize ()
  from /usr/lib/i386-linux-gnu/torsocks/libtorsocks.so
#6  0xb76fd02d in syscall ()
  from /usr/lib/i386-linux-gnu/torsocks/libtorsocks.so
#7  0x004a5049 in os_overcommits_proc ()
  at /checkout/src/liballoc_jemalloc/../jemalloc/src/pages.c:252
#8  je_pages_boot ()
  at /checkout/src/liballoc_jemalloc/../jemalloc/src/pages.c:297
#9  0x004745dd in malloc_init_hard_a0_locked ()
  at /checkout/src/liballoc_jemalloc/../jemalloc/src/jemalloc.c:1366
#10 0x00474768 in malloc_init_hard ()
  at /checkout/src/liballoc_jemalloc/../jemalloc/src/jemalloc.c:1493
#11 0x00489b95 in malloc_init ()
  at /checkout/src/liballoc_jemalloc/../jemalloc/src/jemalloc.c:317
#12 ialloc_body (slow_path=true, usize=<synthetic pointer>,
  tsdn=<synthetic pointer>, zero=true, size=20)at /checkout/src/liballoc_jemalloc/../jemalloc/src/jemalloc.c:1583
#13 calloc (num=1, size=20)
  at /checkout/src/liballoc_jemalloc/../jemalloc/src/jemalloc.c:1824
#14 0xb76d23ec in _dlerror_run (
  operate=operate@entry=0xb76d1b80 <dlopen_doit>, args=args@entry=0xbfb8cd10)
#15 0xb76d1c9e in __dlopen (file=0xb7703345 "libc.so.6", mode=1) at dlopen.c:87
#16 0xb76fa44f in ?? () from /usr/lib/i386-linux-gnu/torsocks/libtorsocks.so
#17 0xb7700352 in tsocks_once ()__
  from /usr/lib/i386-linux-gnu/torsocks/libtorsocks.so
#18 0xb76fa25e in tsocks_initialize ()
  from /usr/lib/i386-linux-gnu/torsocks/libtorsocks.so
#19 0xb7724c65 in call_init (l=<optimized out>, argc=argc@entry=1,
  argv=argv@entry=0xbfb8ce74, env=0xbfb8ce7c) at dl-init.c:72
#20 0xb7724d8e in call_init (env=0xbfb8ce7c, argv=0xbfb8ce74, argc=1,
  l=<optimized out>) at dl-init.c:30
#21 _dl_init (main_map=<optimized out>, argc=1, argv=0xbfb8ce74,
  env=0xbfb8ce7c) at dl-init.c:120
#22 0xb7715a5f in _dl_start_user () from /lib/ld-linux.so.2

It looks like tsocks_initialize() is called when libtorsocks is loaded, it calls tsocks_once() which locks a mutex and then calls dlopen() to get the libc symbols, dlopen() tries to allocate some memory which leads jemalloc (the default allocator for Rust programs) to try to call syscall() (it wants to open a proc file to see if the system overcommits memory or not), which is intercepted by libtorsocks, which leads to another call to tsocks_initialize()... and since the mutex is already locked, it deadlocks.

One way to fix this might be to just let through any syscall() calls that happen during bootstrapping, but i don't know the torsocks code well enough to know if this could cause any dangerous leaks.

Child Tickets

Change History (3)

comment:1 Changed 7 weeks ago by cypherpunks

I believe this is related to the documented issue where a custom malloc() will lock up torsocks. This happens to me when I use an LD_PRELOAD-based hook. If #24037 is taken seriously, this will no longer be an issue.

comment:2 Changed 7 weeks ago by larsl

I have made an attempt at fixing this in the branch rust_deadlock_fix in https://github.com/larsluthman/torsocks.git .

It consists of two commits, the first one simply changes the static initialiser for the mutexes torsocks uses from PTHREAD_MUTEX_INITIALIZER to PTHREAD_ERRORCHECK_MUTEX_INITIALIZER_NP where that is available, which makes a second pthread_mutex_lock() call in the same thread return an error instead of invoking undefined behaviour (which includes the deadlock) and also calls abort() instead of assert() when that error occurs, to avoid allocating memory which can cause further problems if we're currently in the bootstrap code for a custom allocator.

On my system (Debian 9.2, i686) this makes the Rust programs that previously deadlocked abort immediately instead. The mutex change shouldn't have any significant effect on performance since they only get used during initialisation.

The second one makes torsocks's syscall() wrapper less eager to call tsocks_initialize(), so that it only gets called for syscall numbers that explicitly need it, such as those that are redirected to other syscalls intercepted by torsocks. For syscalls that are redirected to actual libc functions, or that are blocked, tsocks_initialize() is not called, particularly not for MMAP (which would cause a recursive allocation when mmap was called by a custom allocator, since tsocks_initialize() tries to allocate memory when calling dlopen()) or OPEN (which would cause an allocation attempt before jemalloc was finished bootstrapping, since it tries to open some /proc files to check system characteristics).

On my system this makes Rust programs work with torsocks again - instead of deadlocking or aborting they simply print "1509634181 WARNING torsocks[15489]: [syscall] Unsupported syscall number 5. Denying the call (in tsocks_syscall() at syscall.c:571)" and continues.

comment:3 Changed 7 weeks ago by larsl

Status: newneeds_review
Note: See TracTickets for help on using tickets.