Opened 20 months ago

Closed 8 months ago

#25245 closed defect (worksforme)

Crash in assert_connection_ok when changing Exit options

Reported by: toralf Owned by: ahf
Priority: Very High Milestone: Tor: 0.3.5.x-final
Component: Core Tor/Tor Version: Tor: 0.3.3.2-alpha
Severity: Normal Keywords: crash, regression?, tor-exit, tor-relay, ipv6, 033-triage-20180320, 033-included-20180320, 035-must, 035-triaged-in-20180711
Cc: Actual Points:
Parent ID: Points: 1
Reviewer: Sponsor:

Description

Got this today:

Feb 13 22:02:49.000 [err] tor_assertion_failed_(): Bug: src/or/connection.c:5113: assert_connection_ok: Assertion (conn->type == CONN_TYPE_EXIT && conn->state == EXIT_CONN_STATE_RESOLVING) || connection_is_writing(conn) || conn->write_blocked_on_bw || (CONN_IS_EDGE(conn) && TO_EDGE_CONN(conn)->edge_blocked_on_circ) failed; aborting. (on Tor 0.3.3.2-alpha efc105716283bbdf)

Before I changed the config from

ExitRelay 1
IPv6Exit 1

to

#ExitRelay 1
#IPv6Exit 1

and a little bit later to

ExitRelay 0
IPv6Exit 0

and run a kill -HUP (after every config change)

Child Tickets

Change History (30)

comment:1 Changed 20 months ago by teor

Keywords: crash regression? tor-exit ipv6 added
Milestone: Tor: 0.3.3.x-final
Points: 1
Priority: MediumVery High

comment:2 Changed 20 months ago by teor

Keywords: tor-relay added
Summary: [err] tor_assertion_failed_(): Bug: src/or/connection.c:5113Crash in assert_connection_ok when changing Exit options

comment:4 Changed 20 months ago by rl1987

I tried untangling the assert statement for better debuggability:

Please try reproducing the crash bug with this Tor compiled from this branch. A stack trace would be nice as well. I wasn't able to make it crash on my local machine as it seems to be related to actual traffic being handled by your relay.

When exit node feature is disabled by force-reloading torrc, what exactly happens to pending exit connections? Do we cancel them properly?

Last edited 20 months ago by rl1987 (previous) (diff)

comment:5 Changed 20 months ago by rl1987

Status: newneeds_information

comment:6 Changed 20 months ago by toralf

Applied that patch and got immediately a crash of both of my Tor exit relays :

Mar 11 16:27:41.000 [err] tor_assertion_failed_(): Bug: src/or/connection.c:5114: assert_connection_ok: Assertion connection_is_writing(conn) || conn->write_blocked_on_bw failed; aborting. (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug: Assertion connection_is_writing(conn) || conn->write_blocked_on_bw failed in assert_connection_ok at src/or/connection.c:5114. Stack trace: (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/bin/tor(log_backtrace+0x53) [0x564a1d49f513] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/bin/tor(tor_assertion_failed_+0x99) [0x564a1d4bc559] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/bin/tor(assert_connection_ok+0x86b) [0x564a1d419e8b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/bin/tor(+0x152b4b) [0x564a1d45fb4b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x3329d) [0x7fd983b4329d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x244d0) [0x7fd983b344d0] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(event_base_loop+0x53f) [0x7fd983b3548f] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/bin/tor(do_main_loop+0x26c) [0x564a1d3616dc] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/bin/tor(tor_run_main+0x27d) [0x564a1d362d5d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/bin/tor(tor_main+0x49) [0x564a1d35bd79] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/bin/tor(main+0x28) [0x564a1d35baa8] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /lib64/libc.so.6(__libc_start_main+0xfd) [0x7fd9826a75ad] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:27:41.000 [err] Bug:     /usr/bin/tor(_start+0x2a) [0x564a1d35bafa] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)

and

ar 11 16:28:07.000 [err] tor_assertion_failed_(): Bug: src/or/connection.c:5114: assert_connection_ok: Assertion connection_is_writing(conn) || conn->write_blocked_on_bw failed; aborting. (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug: Assertion connection_is_writing(conn) || conn->write_blocked_on_bw failed in assert_connection_ok at src/or/connection.c:5114. Stack trace: (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/bin/tor(log_backtrace+0x53) [0x564fb8033513] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/bin/tor(tor_assertion_failed_+0x99) [0x564fb8050559] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/bin/tor(assert_connection_ok+0x86b) [0x564fb7fade8b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/bin/tor(+0x152b4b) [0x564fb7ff3b4b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x3329d) [0x7f6ad207d29d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x244d0) [0x7f6ad206e4d0] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(event_base_loop+0x53f) [0x7f6ad206f48f] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/bin/tor(do_main_loop+0x26c) [0x564fb7ef56dc] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/bin/tor(tor_run_main+0x27d) [0x564fb7ef6d5d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/bin/tor(tor_main+0x49) [0x564fb7eefd79] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/bin/tor(main+0x28) [0x564fb7eefaa8] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f6ad0be15ad] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 16:28:07.000 [err] Bug:     /usr/bin/tor(_start+0x2a) [0x564fb7eefafa] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Last edited 20 months ago by toralf (previous) (diff)

comment:7 Changed 20 months ago by teor

Status: needs_informationnew

comment:8 Changed 20 months ago by rl1987

Okay, now line numbers do not match. Which one of the three new assert statements is at line 5114?

Alternatively, which git commit were you at before applying this?

comment:9 Changed 20 months ago by toralf

Well, I use the Gentoo package which has tor.0.3.3.3-alpha packed.
On that I do apply "git diff tor-0.3.3.3-alpha..tor-0.3.4.0-alpha-dev", the exact command is

torproject@mr-fox ~ $ cd ~/tor; git diff tor-0.3.3.3-alpha..tor-0.3.4.0-alpha-dev -- . ':!doc' ':!contrib' ':!changes' ':!ChangeLog' ':!Release*' ':!scripts' ':!.gitignore' ':!.travis.yml' ':!src/test/Makefile.nmake' ':!.gitlab-ci.yml' ':!.gitmodules' > /etc/portage/patches/net-vpn/tor/master.patch

and because your given patch starts with ebd1fda4b16bb169718 the Gentoo logic appleis first the ebd* patch and then master.
I can change the name to A-master so that your patch is applied on top of the upgrade from 0.3.3.3 to 0.3.4 if you want ?

Version 0, edited 20 months ago by toralf (next)

comment:10 Changed 20 months ago by toralf

And with the opposite order in applying first the 0.3.3.3->0.3.4 and then your patch I get this picture :

Mar 11 21:03:15.000 [err] tor_assertion_failed_(): Bug: src/or/connection.c:5114: assert_connection_ok: Assertion connection_is_writing(conn) || conn->write_blocked_on_bw failed; aborting. (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug: Assertion connection_is_writing(conn) || conn->write_blocked_on_bw failed in assert_connection_ok at src/or/connection.c:5114. Stack trace: (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/bin/tor(log_backtrace+0x53) [0x55c4d18a3513] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/bin/tor(tor_assertion_failed_+0x99) [0x55c4d18c0559] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/bin/tor(assert_connection_ok+0x86b) [0x55c4d181de8b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/bin/tor(+0x152b4b) [0x55c4d1863b4b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x3329d) [0x7f5327c1829d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x244d0) [0x7f5327c094d0] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(event_base_loop+0x53f) [0x7f5327c0a48f] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/bin/tor(do_main_loop+0x26c) [0x55c4d17656dc] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/bin/tor(tor_run_main+0x27d) [0x55c4d1766d5d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/bin/tor(tor_main+0x49) [0x55c4d175fd79] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/bin/tor(main+0x28) [0x55c4d175faa8] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f532677c5ad] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:03:15.000 [err] Bug:     /usr/bin/tor(_start+0x2a) [0x55c4d175fafa] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)

and

Mar 11 21:02:24.000 [err] tor_assertion_failed_(): Bug: src/or/connection.c:5114: assert_connection_ok: Assertion connection_is_writing(conn) || conn->write_blocked_on_bw failed; aborting. (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug: Assertion connection_is_writing(conn) || conn->write_blocked_on_bw failed in assert_connection_ok at src/or/connection.c:5114. Stack trace: (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/bin/tor(log_backtrace+0x53) [0x559eb6f5e513] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/bin/tor(tor_assertion_failed_+0x99) [0x559eb6f7b559] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/bin/tor(assert_connection_ok+0x86b) [0x559eb6ed8e8b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/bin/tor(+0x152b4b) [0x559eb6f1eb4b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x3329d) [0x7ff26e1a729d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x244d0) [0x7ff26e1984d0] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(event_base_loop+0x53f) [0x7ff26e19948f] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/bin/tor(do_main_loop+0x26c) [0x559eb6e206dc] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/bin/tor(tor_run_main+0x27d) [0x559eb6e21d5d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/bin/tor(tor_main+0x49) [0x559eb6e1ad79] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/bin/tor(main+0x28) [0x559eb6e1aaa8] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /lib64/libc.so.6(__libc_start_main+0xfd) [0x7ff26cd0b5ad] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:02:24.000 [err] Bug:     /usr/bin/tor(_start+0x2a) [0x559eb6e1aafa] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)

comment:12 Changed 20 months ago by toralf

applying that patch too gives:

Mar 11 21:45:51.000 [err] tor_assertion_failed_(): Bug: src/or/connection.c:5114: assert_connection_ok: Assertion 1 == 1 && (connection_is_writing(conn) || conn->write_blocked_on_bw) failed; aborting. (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug: Assertion 1 == 1 && (connection_is_writing(conn) || conn->write_blocked_on_bw) failed in assert_connection_ok at src/or/connection.c:5114. Stack trace: (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/bin/tor(log_backtrace+0x53) [0x55d5ad38e513] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/bin/tor(tor_assertion_failed_+0x99) [0x55d5ad3ab559] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/bin/tor(assert_connection_ok+0x86b) [0x55d5ad308e8b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/bin/tor(+0x152b4b) [0x55d5ad34eb4b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x3329d) [0x7f4f6a4be29d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x244d0) [0x7f4f6a4af4d0] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(event_base_loop+0x53f) [0x7f4f6a4b048f] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/bin/tor(do_main_loop+0x26c) [0x55d5ad2506dc] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/bin/tor(tor_run_main+0x27d) [0x55d5ad251d5d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/bin/tor(tor_main+0x49) [0x55d5ad24ad79] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/bin/tor(main+0x28) [0x55d5ad24aaa8] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f4f690225ad] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:45:51.000 [err] Bug:     /usr/bin/tor(_start+0x2a) [0x55d5ad24aafa] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)

and

Mar 11 21:46:26.000 [err] tor_assertion_failed_(): Bug: src/or/connection.c:5114: assert_connection_ok: Assertion 1 == 1 && (connection_is_writing(conn) || conn->write_blocked_on_bw) failed; aborting. (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug: Assertion 1 == 1 && (connection_is_writing(conn) || conn->write_blocked_on_bw) failed in assert_connection_ok at src/or/connection.c:5114. Stack trace: (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/bin/tor(log_backtrace+0x53) [0x56077cb26513] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/bin/tor(tor_assertion_failed_+0x99) [0x56077cb43559] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/bin/tor(assert_connection_ok+0x86b) [0x56077caa0e8b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/bin/tor(+0x152b4b) [0x56077cae6b4b] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x3329d) [0x7f93f338f29d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(+0x244d0) [0x7f93f33804d0] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/lib64/libevent-2.1.so.6(event_base_loop+0x53f) [0x7f93f338148f] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/bin/tor(do_main_loop+0x26c) [0x56077c9e86dc] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/bin/tor(tor_run_main+0x27d) [0x56077c9e9d5d] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/bin/tor(tor_main+0x49) [0x56077c9e2d79] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/bin/tor(main+0x28) [0x56077c9e2aa8] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f93f1ef35ad] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)
Mar 11 21:46:26.000 [err] Bug:     /usr/bin/tor(_start+0x2a) [0x56077c9e2afa] (on Tor 0.3.4.0-alpha-dev d15a7a27bb9e49bd)

comment:13 Changed 19 months ago by rl1987

I figure this happens because we do not properly cleanup/close existing edge connections when ExitRelay is switched off. A solution would be to keep track of child connections that listeners spawn when accept() succeeds and close all of them when closing the listener.

Something like:

Does this make sense?

If this is correct approach I could finalise the patch into mergeable shape.

comment:14 Changed 19 months ago by nickm

Hm. We don't actually want to close all of the "Foo" connections when the "FooListener" is closed. If we did that, then our hibernation code would break, since it assumes that you we continue servicing existing connections after we've closed all the listeners.

I think instead the problem here is that there is something wrong with the tor_assert(). Can you help me understand how the problem here relates to the fix? That is, why does setting ExitRelay 0 make these connections trigger the assertion if they do not get closed?

comment:15 Changed 19 months ago by nickm

Keywords: 033-must added

comment:16 Changed 19 months ago by nickm

Keywords: 033-triage-20180320 added

Marking all tickets reached by current round of 033 triage.

comment:17 Changed 19 months ago by nickm

Keywords: 033-included-20180320 added

Mark 033-must tickets as triaged-in for 0.3.3

comment:18 in reply to:  14 Changed 19 months ago by rl1987

Let's ponder some more about this. Look at the above stack traces. Libevent calls back into tor, then the called function calls assert_connection_ok(), which explodes.

The function in question is conn_write_callback() in main.c. It calls connection_handle_write() which seems to succeed (otherwise it would attempt to close and null out outbuf_flushlen for connection in question).

However, right after that assert_connection_ok() fails. Do we have a race condition here? Is there one more exception case when edge connection cannot be writable despite outbuf_flushlen not being zero? Does libevent de-schedule EV_WRITE for already accepted connections when listener is closed?

comment:19 in reply to:  4 Changed 19 months ago by arma

Replying to rl1987:

I tried untangling the assert statement for better debuggability:

I think this patch is not right. That is, the new more debuggable version is a different set of asserts than the original.

comment:20 Changed 19 months ago by rl1987

I believe the following is logically equivalent (and logs more debug info):

This one checks if we have edge connection first and if so, checks if it is exit connection.

comment:21 Changed 19 months ago by ahf

Owner: set to ahf
Status: newassigned

comment:22 Changed 17 months ago by teor

Milestone: Tor: 0.3.3.x-finalTor: 0.3.5.x-final
Status: assignedneeds_revision

We can fix this in 0.3.5.

comment:23 Changed 16 months ago by rl1987

Idea: maybe we could use this to reproduce the crashbug: https://github.com/antitree/private-tor-network

comment:24 Changed 16 months ago by nickm

Keywords: 035-must added

comment:25 Changed 16 months ago by nickm

Keywords: 035-triaged-in-20180711 added

comment:26 Changed 13 months ago by nickm

Mark all 035-must tickets as "very high"

comment:27 Changed 11 months ago by teor

Keywords: 033-must removed

If this ticket is a -must, it's an 035-must

comment:28 Changed 8 months ago by rl1987

Status: needs_revisionneeds_information

Is this bug still a thing?

comment:29 Changed 8 months ago by toralf

The old hardware is gone entirely.
At my 2 new relays (0.4.0.2-alpha, non-exits) I cannot reproduce the bug.

comment:30 Changed 8 months ago by nickm

Resolution: worksforme
Status: needs_informationclosed

Closing as worksforme, but please reopen if it happens again.

Note: See TracTickets for help on using tickets.