Opened 3 years ago

Closed 3 years ago

#19128 closed defect (fixed)

Bug: src/common/crypto.c:3039: memwipe: Assertion sz < SIZE_T_CEILING failed; aborting.

Reported by: toralf Owned by: nickm
Priority: High Milestone: Tor: 0.2.8.x-final
Component: Core Tor/Tor Version: Tor: 0.2.8.2-alpha
Severity: Blocker Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

git 2b8ba551d3eab2793214ab34a1c9a47a888402cd at a hardened stable Gentoo :

May 19 20:03:46.000 [err] tor_assertion_failed_(): Bug: src/common/crypto.c:3039: memwipe: Assertion sz < SIZE_T_CEILING failed; aborting. (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug: Assertion sz < SIZE_T_CEILING failed in memwipe at src/common/crypto.c:3039. Stack trace: (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(log_backtrace+0x55) [0x64843e1715] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(tor_assertion_failed_+0x9c) [0x64843f145c] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(memwipe+0xbf) [0x648440654f] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(tor_cert_free+0x43) [0x6484341ca3] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(routerinfo_free+0x102) [0x6484329a92] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(+0x8bbcd) [0x6484329bcd] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(router_add_to_routerlist+0x75d) [0x648432b3dd] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(router_load_routers_from_string+0x183) [0x648432ee53] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(+0x113d14) [0x64843b1d14] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(connection_dir_reached_eof+0x3c) [0x64843b27ac] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(+0xf2779) [0x6484390779] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(+0x40024) [0x64842de024] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/lib64/libevent-2.0.so.5(event_base_loop+0x6dd) [0x3cd7614671d] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(do_main_loop+0x235) [0x64842df0c5] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(tor_main+0x1b35) [0x64842e2745] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(main+0x2b) [0x64842da6ab] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /lib64/libc.so.6(__libc_start_main+0x114) [0x3cd74e40734] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 19 20:03:46.000 [err] Bug:     /usr/bin/tor(_start+0x29) [0x64842da6f9] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)

Child Tickets

TicketStatusOwnerSummaryComponent
#19175closednickmfree(): invalid next size (fast)Core Tor/Tor
#19177closedanother free() crashCore Tor/Tor

Change History (19)

comment:1 Changed 3 years ago by nickm

Keywords: must-fix-before-028-alpha TorCoreTeam201605 added
Milestone: Tor: 0.2.8.x-final
Priority: MediumHigh
Severity: NormalBlocker

comment:2 Changed 3 years ago by toralf

likely that the issue is uncovered by something in ef03dc0..2b8ba55

comment:3 Changed 3 years ago by nickm

I merged #19073, which might fix this, or might not.

comment:4 Changed 3 years ago by toralf

This bug is unfortunately still present in latest origin/release-0.2.8 from yesterday, so #19073 is a different bug.

comment:5 Changed 3 years ago by nickm

Okay; what does the stack trace look like now?

comment:6 Changed 3 years ago by nickm

I think I figured this one out.

comment:7 in reply to:  5 Changed 3 years ago by toralf

Replying to nickm:

Okay; what does the stack trace look like now?

May 20 01:03:10.000 [err] tor_assertion_failed_(): Bug: src/common/crypto.c:3039: memwipe: Assertion sz < SIZE_T_CEILING failed; aborting. (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug: Assertion sz < SIZE_T_CEILING failed in memwipe at src/common/crypto.c:3039. Stack trace: (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(log_backtrace+0x55) [0x124a47f6d5] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(tor_assertion_failed_+0x9c) [0x124a48f41c] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(memwipe+0xbf) [0x124a4a450f] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(tor_cert_free+0x43) [0x124a3dfc63] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(routerinfo_free+0xf5) [0x124a3c7a65] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(+0x8bb9d) [0x124a3c7b9d] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(router_add_to_routerlist+0x75d) [0x124a3c93ad] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(router_load_routers_from_string+0x183) [0x124a3cce23] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(+0x113cd4) [0x124a44fcd4] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(connection_dir_reached_eof+0x3c) [0x124a45076c] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(+0xf2739) [0x124a42e739] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(+0x40024) [0x124a37c024] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/lib64/libevent-2.0.so.5(event_base_loop+0x6dd) [0x389f854171d] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(do_main_loop+0x235) [0x124a37d0c5] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(tor_main+0x1b35) [0x124a380745] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(main+0x2b) [0x124a3786ab] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /lib64/libc.so.6(__libc_start_main+0x114) [0x389f723b734] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)
May 20 01:03:10.000 [err] Bug:     /usr/bin/tor(_start+0x29) [0x124a3786f9] (on Tor 0.2.8.2-alpha-dev 684babee8491c3e9)

comment:8 Changed 3 years ago by nickm

So, my branch bug19128_028 has a fix for a dangling pointer bug related to this stuff. But my problem is, I can't see how the dangling pointer bug might actually cause this particular symptom. I say we try it anyway and keep looking.

comment:9 Changed 3 years ago by nickm

Owner: set to nickm
Status: newaccepted

(setting myself as owner here, but I will need others to work on this too to figure it out)

comment:10 Changed 3 years ago by nickm

Status: acceptedneeds_review

comment:11 Changed 3 years ago by nickm

Status: needs_reviewneeds_information

weasel liked it; merging it to maint/release-0.2.8 and to master. Putting this in needs_information: did we fix the bug?

comment:12 Changed 3 years ago by toralf

fd34049 now gives :

May 20 21:57:53.000 [notice] Tor 0.2.8.2-alpha-dev (git-684babee8491c3e9) opening log file.

============================================================ T= 1463774682
Tor 0.2.8.2-alpha-dev (git-684babee8491c3e9) died: Caught signal 11
/usr/bin/tor(+0x1435c9)[0x19aad105c9]
/usr/lib64/libcrypto.so.1.0.0(OPENSSL_cleanse+0x35)[0x36b930375b5]
/usr/lib64/libcrypto.so.1.0.0(OPENSSL_cleanse+0x35)[0x36b930375b5]

comment:13 Changed 3 years ago by nickm

(currently hoping for a gdb backtrace here.)

comment:14 Changed 3 years ago by nickm

Keywords: must-fix-before-028-alpha removed

comment:15 Changed 3 years ago by toralf

The issue appeared again after about 20h (expected due to the lower bandwidth now).
Unfortunately I missed to add the "-ex bt" to the gdb command line, therefore I just got a

Program received signal SIGFPE, Arithmetic exception.
0x00000313c6d855e1 in tls1_enc (s=0x79e1d5f710, send=1) at t1_enc.c:849
849 t1_enc.c: No such file or directory.

related to this of debug.log

May 24 19:06:35.000 [debug] conn_write_callback(): socket 622 wants to write.

============================================================ T= 1464109596
Tor 0.2.8.2-alpha-dev (git-684babee8491c3e9) died: Caught signal 8
/usr/bin/tor(+0x1435c9)[0x79dd1c35c9]
/usr/lib64/libssl.so.1.0.0(tls1_enc+0x1c1)[0x313c6d855e1]
/usr/lib64/libssl.so.1.0.0(tls1_enc+0x1c1)[0x313c6d855e1]
/usr/lib64/libssl.so.1.0.0(+0x320fa)[0x313c6d760fa]
/usr/lib64/libssl.so.1.0.0(ssl3_write_bytes+0xe5)[0x313c6d76555]
/usr/bin/tor(tor_tls_write+0xa3)[0x79dd1ef7e3]
/usr/bin/tor(flush_buf_tls+0xbb)[0x79dd128d3b]
/usr/bin/tor(+0xf2e52)[0x79dd172e52]
/usr/bin/tor(connection_handle_write+0x43)[0x79dd173813]
/usr/bin/tor(+0x3fe41)[0x79dd0bfe41]
/usr/lib64/libevent-2.0.so.5(event_base_loop+0x799)[0x313c6fd4319]
/usr/bin/tor(do_main_loop+0x235)[0x79dd0c10c5]
/usr/bin/tor(tor_main+0x1b35)[0x79dd0c4745]
/usr/bin/tor(main+0x2b)[0x79dd0bc6ab]
/lib64/libc.so.6(__libc_start_main+0x114)[0x313c5cce734]
/usr/bin/tor(_start+0x29)[0x79dd0bc6f9]

I do have the debug.log here and can send it via email if needed.
Currently I do have the following gdb running :

rm nohup.out; nohup gdb -q -p `pgrep tor` -ex "handle SIGPIPE nostop ignore noprint" -ex "handle SIGHUP nostop" -ex cont -ex bt &

Hope to get a better result within the enxt day or so.

comment:16 Changed 3 years ago by nickm

#19175 appears to be a case of this, and appears (I hope) to be the last case of this.

comment:17 Changed 3 years ago by nickm

Merged #19175. Please let me know if this can still happen!

comment:18 Changed 3 years ago by nickm

Keywords: TorCoreTeam201605 removed

Remove "TorCoreTeam201605" keyword. The time machine is broken.

comment:19 Changed 3 years ago by nickm

Resolution: fixed
Status: needs_informationclosed

Calling this probably fixed in #19175 .

Note: See TracTickets for help on using tickets.