Opened 5 months ago

Closed 5 months ago

#23551 closed defect (fixed)

src/common/compress.c:576: tor_compress_process: Non-fatal assertion

Reported by: cypherpunks Owned by: ahf
Priority: High Milestone: Tor: 0.3.1.x-final
Component: Core Tor/Tor Version: Tor: 0.3.1.5-alpha
Severity: Normal Keywords: 0.3.2.2-alpha-must
Cc: vvort@… Actual Points:
Parent ID: Points:
Reviewer: isis Sponsor:

Description

duplicate of #22719 ?

OS: HardenedBSD

Bootstrapped 100%: Done
Performing bandwidth self-test...done.
tor_bug_occurred_: Bug: src/common/compress.c:576: tor_compress_process: Non-fatal assertion !((rv == TOR_COMPRESS_OK) && *in_len == in_len_orig && *out_len == out_len_orig) failed. (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug: Non-fatal assertion !((rv == TOR_COMPRESS_OK) && *in_len == in_len_orig && *out_len == out_len_orig) failed in tor_compress_process at src/common/compress.c:576. Stack trace: (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103e31d87 <log_backtrace+0x37> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103e488a1 <tor_bug_occurred_+0x131> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
\Bug:     0xe103e4dc7b <tor_compress_process+0xcb> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103e4d32c <tor_compress+0x21c> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103e4d7e8 <tor_uncompress+0x28> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103df4c88 <connection_dir_reached_eof+0x818> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103df448d <connection_dir_reached_eof+0x1d> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103dce0b8 <connection_handle_read+0xbb8> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103d19dd1 <connection_add_impl+0x1f1> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0x36578fda06f <event_base_assert_ok_nolock_+0xc9f> at /usr/local/lib/libevent-2.1.so.6 (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0x36578fd5f4e <event_base_loop+0x50e> at /usr/local/lib/libevent-2.1.so.6 (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103d1b834 <do_main_loop+0x594> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103d1dcdb <tor_main+0xcb> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103d19b99 <main+0x9> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)
Bug:     0xe103d089f0 <_start+0x180> at /usr/local/bin/tor (on Tor 0.3.1.5-alpha 83389502ee631465)

Child Tickets

Change History (16)

comment:1 Changed 5 months ago by nickm

Milestone: Tor: 0.3.1.x-final
Priority: MediumHigh

comment:2 Changed 5 months ago by nickm

It sure does look similar to #22719, though we thought we fixed that one 0.3.1.4-alpha

comment:3 Changed 5 months ago by Vort

Cc: vvort@… added

Same thing for 0.3.1.7 (custom build) and Windows 7:

Sep 19 12:33:13.743 [notice] Tor 0.3.1.7 running on Windows 7 with Libevent 2.0.22-stable, OpenSSL 1.0.2l, Zlib 1.2.11, Liblzma 5.2.3, and Libzstd 1.3.1.
...
Sep 19 23:36:30.000 [warn] tor_bug_occurred_(): Bug: compress.c:576: tor_compress_process: Non-fatal assertion !((rv == TOR_COMPRESS_OK) && *in_len == in_len_orig && *out_len == out_len_orig) failed. (on Tor 0.3.1.7 )
Sep 19 23:36:30.000 [warn] Bug: Non-fatal assertion !((rv == TOR_COMPRESS_OK) && *in_len == in_len_orig && *out_len == out_len_orig) failed in tor_compress_process at compress.c:576. (Stack trace not available) (on Tor 0.3.1.7 )

comment:4 Changed 5 months ago by Felixix

Tor 0.3.2.1-alpha (git-290274dbb5428bc5) running on FreeBSD with Libevent 2.1.8-stable, OpenSSL LibreSSL 2.5.4, Zlib 1.2.8, Liblzma 5.2.2, and Libzstd 1.3.1.
More info on the bug: method == Zstandard compressed, finish == 0,  *in_len == in_len_orig == 1136, *out_len == out_len_orig == 0

comment:5 Changed 5 months ago by nickm

Ohhhh. out_len_orig == 0 seems like the problem there...

comment:6 Changed 5 months ago by teor

I suggest we split that BUG() expression into its separate components to aid in diagnosing future issues.

Edit: oh, hang on, that won't work. Just ignore me.

Last edited 5 months ago by teor (previous) (diff)

comment:7 Changed 5 months ago by teor

It wouldn't surprise me if we have a zstd version incompatibility here. Or maybe truncated input data?

Truncated input data would definitely explain why zstd can't make any progress, but isn't returning an error code. In that case, it's our fault for not giving zstd enough data to work with. (And not failing the loop quietly?)

Because if it were corrupt data, it really should return an error:
http://facebook.github.io/zstd/zstd_manual.html#Chapter9

comment:8 Changed 5 months ago by nickm

I think that the reason it's making no progress is because out_len_orig was 0 -- there was no space in the output buffer when it started.

comment:9 Changed 5 months ago by ahf

Owner: set to ahf
Status: newassigned

comment:10 Changed 5 months ago by cypherpunks

this is hitting all my Debian 9 based relays:

Tor 0.3.1.7 (git-5fa14939bca67c23) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.1.0f, Zlib 1.2.8, Liblzma 5.2.2, and Libzstd 1.1.2.
 Bug: ../src/common/compress.c:576: tor_compress_process: Non-fatal assertion !((rv == TOR_COMPRESS_OK) && *in_len == in_len_orig && *out_len == out_len_orig) failed. (on Tor 0.3.1.7 )
 Non-fatal assertion !((rv == TOR_COMPRESS_OK) && *in_len == in_len_orig && *out_len == out_len_orig) failed in tor_compress_process at ../src/common/compress.c:576. Stack trace: (on Tor 0.3.1.7 )
     /usr/bin/tor(log_backtrace+0x44) [0x557fdd02e194] (on Tor 0.3.1.7 )
     /usr/bin/tor(tor_bug_occurred_+0xb9) [0x557fdd047029] (on Tor 0.3.1.7 )
     /usr/bin/tor(tor_compress_process+0x135) [0x557fdd04ffa5] (on Tor 0.3.1.7 )
     /usr/bin/tor(+0x18e171) [0x557fdd050171] (on Tor 0.3.1.7 )
     /usr/bin/tor(tor_uncompress+0x31) [0x557fdd050631] (on Tor 0.3.1.7 )
     /usr/bin/tor(connection_dir_reached_eof+0x118c) [0x557fdcff266c] (on Tor 0.3.1.7 )
     /usr/bin/tor(+0x1089bc) [0x557fdcfca9bc] (on Tor 0.3.1.7 )
     /usr/bin/tor(+0x4d85e) [0x557fdcf0f85e] (on Tor 0.3.1.7 )
     /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5(event_base_loop+0x6a0) [0x7f2bebf2a5a0] (on Tor 0.3.1.7 )
     /usr/bin/tor(do_main_loop+0x29d) [0x557fdcf1098d] (on Tor 0.3.1.7 )
     /usr/bin/tor(tor_main+0x1c35) [0x557fdcf144d5] (on Tor 0.3.1.7 )
     /usr/bin/tor(main+0x19) [0x557fdcf0c189] (on Tor 0.3.1.7 )
     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f2bea98c2b1] (on Tor 0.3.1.7 )
     /usr/bin/tor(_start+0x2a) [0x557fdcf0c1da] (on Tor 0.3.1.7 )

It does NOT affect my Debian 8 relays since they have no zstd:

Tor 0.3.1.7 (git-5fa14939bca67c23) running on Linux ..., and Libzstd **N/A.**

comment:11 Changed 5 months ago by nickm

Keywords: 0.3.2.2-alpha-must added

comment:12 Changed 5 months ago by isis

Reviewer: isis

Assigning myself as a reviewer since we want this is 0.3.2.2-alpha. I know very little to nothing of libzstd, so if someone else would also like to review that would be helpful. (Maybe teor?)

comment:13 Changed 5 months ago by ahf

Status: assignedneeds_review

I have a proposed fix for this issue. It can be found in the following branches:

comment:14 in reply to:  13 Changed 5 months ago by isis

Status: needs_reviewmerge_ready

Replying to ahf:

I have a proposed fix for this issue. It can be found in the following branches:


  • c2fac2c6 LGTM (yay test driven dev)
  • 44dc4b73 LGTM
  • c3b7f9d7 LGTM
  • a196fdb6 LGTM

FWIW this didn't require any libzstd knowledge to review, so I feel comfortable that my assessment is correct. If someone else still wants to also review, though, please feel free.

comment:15 Changed 5 months ago by nickm

Okay. Merged to maint-0.3.1 and master.

comment:16 Changed 5 months ago by nickm

Resolution: fixed
Status: merge_readyclosed
Note: See TracTickets for help on using tickets.