Opened 6 weeks ago

Closed 6 weeks ago

#22252 closed defect (fixed)

get_options_mutable: Assertion global_options failed; SIGABRT

Reported by: stze Owned by: nickm
Priority: Medium Milestone: Tor: 0.3.0.x-final
Component: Core Tor Version: Tor: unspecified
Severity: Normal Keywords: 029-backport 030-backport
Cc: dgoulet, arma, catalyst Actual Points: .2
Parent ID: Points:
Reviewer: Sponsor:

Description

Version:

Tor 0.3.1.0-alpha-dev (git-44102714460aafe5)

Input file hexdump:

00000000  4c 00 20 20 66 6f 6f 2e  6c 6f 67 0a 0a           |L.  foo.log..|
0000000d

How to reproduce:

$ ./src/or/tor -f <attached input file> --verify-config

gdb:

Program terminated with signal SIGABRT, Aborted.
#0  0x00007fb6f1d31a10 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007fb6f1d31a10 in raise () from /usr/lib/libc.so.6
#1  0x00007fb6f1d3313a in abort () from /usr/lib/libc.so.6
#2  0x000055a0ed6e39a3 in get_options_mutable () at src/or/config.c:768
#3  get_options () at src/or/config.c:776
#4  0x000055a0ed5b224f in networkstatus_get_latest_consensus () at src/or/networkstatus.c:1370
#5  networkstatus_get_param (ns=0x2, param_name=0x55a0ed8e2190 "cbtdisabled", default_val=0, min_val=0, 
    max_val=1) at src/or/networkstatus.c:2368
#6  0x000055a0ed6d0f4b in circuit_build_times_disabled (options=0x55a0eeef0c50)
    at src/or/circuitstats.c:113
#7  0x000055a0ed6ef697 in options_validate (old_options=<optimized out>, options=<optimized out>, 
    default_options=<optimized out>, from_setconf=<optimized out>, msg=<optimized out>)
    at src/or/config.c:3493
#8  0x000055a0ed6faa72 in options_init_from_string (cf_defaults=<optimized out>, cf=<optimized out>, 
    command=<optimized out>, command_arg=<optimized out>, msg=<optimized out>) at src/or/config.c:5170
#9  0x000055a0ed6f8eba in options_init_from_torrc (argc=<optimized out>, argv=<optimized out>)
    at src/or/config.c:4968
#10 0x000055a0ed59a184 in tor_init (argc=<optimized out>, argv=<optimized out>) at src/or/main.c:3080
#11 0x000055a0ed59aeb8 in tor_main (argc=2, argv=0x7ffe29220520) at src/or/main.c:3707
#12 0x000055a0ed5923e9 in main (argc=2, argv=0x7ffe29220520) at src/or/tor_main.c:34

valgrind:

==32291== Memcheck, a memory error detector
==32291== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==32291== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==32291== Command: ./src/or/tor -f /tmp/crash --verify-config
==32291== 
May 14 10:44:50.978 [notice] Tor 0.3.1.0-alpha-dev (git-44102714460aafe5) running on Linux with Libevent 2.0.22-stable, OpenSSL 1.1.0e, Zlib 1.2.11, Liblzma 5.2.3, and Libzstd N/A.
May 14 10:44:51.044 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
May 14 10:44:51.046 [notice] This version is not a stable Tor release. Expect more bugs than usual.
May 14 10:44:51.079 [notice] Read configuration file "/tmp/crash".
May 14 10:44:51.445 [warn] The abbreviation 'L' is deprecated. Please use 'LearnCircuitBuildTimeout' instead
May 14 10:44:51.550 [err] tor_assertion_failed_: Bug: src/or/config.c:768: get_options_mutable: Assertion global_options failed; aborting. (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.560 [err] Bug: Assertion global_options failed in get_options_mutable at src/or/config.c:768. Stack trace: (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.561 [err] Bug:     ./src/or/tor(log_backtrace+0x66) [0x3b3d86] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.561 [err] Bug:     ./src/or/tor(tor_assertion_failed_+0xc3) [0x3ea183] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.562 [err] Bug:     ./src/or/tor(get_options+0x9e) [0x2a299e] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.562 [err] Bug:     ./src/or/tor(networkstatus_get_param+0x6f) [0x17124f] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.562 [err] Bug:     ./src/or/tor(circuit_build_times_disabled+0x5b) [0x28ff4b] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.562 [err] Bug:     ./src/or/tor(+0x1a6697) [0x2ae697] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.562 [err] Bug:     ./src/or/tor(options_init_from_string+0x862) [0x2b9a72] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.562 [err] Bug:     ./src/or/tor(options_init_from_torrc+0x6aa) [0x2b7eba] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.562 [err] Bug:     ./src/or/tor(tor_init+0x7e4) [0x159184] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.562 [err] Bug:     ./src/or/tor(tor_main+0x88) [0x159eb8] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.565 [err] Bug:     ./src/or/tor(main+0x39) [0x1513e9] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.565 [err] Bug:     /usr/lib/libc.so.6(__libc_start_main+0xf1) [0x673e511] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
May 14 10:44:51.565 [err] Bug:     ./src/or/tor(_start+0x2a) [0x1512aa] (on Tor 0.3.1.0-alpha-dev 44102714460aafe5)
==32291== 
==32291== Process terminating with default action of signal 6 (SIGABRT): dumping core
==32291==    at 0x6751A10: raise (in /usr/lib/libc-2.25.so)
==32291==    by 0x6753139: abort (in /usr/lib/libc-2.25.so)
==32291==    by 0x2A29A2: get_options_mutable (config.c:768)
==32291==    by 0x2A29A2: get_options (config.c:776)
==32291==    by 0x17124E: networkstatus_get_latest_consensus (networkstatus.c:1370)
==32291==    by 0x17124E: networkstatus_get_param (networkstatus.c:2368)
==32291==    by 0x28FF4A: circuit_build_times_disabled (circuitstats.c:113)
==32291==    by 0x2AE696: options_validate (config.c:3493)
==32291==    by 0x2B9A71: options_init_from_string (config.c:5170)
==32291==    by 0x2B7EB9: options_init_from_torrc (config.c:4968)
==32291==    by 0x159183: tor_init (main.c:3080)
==32291==    by 0x159EB7: tor_main (main.c:3707)
==32291==    by 0x1513E8: main (tor_main.c:34)
==32291== 
==32291== HEAP SUMMARY:
==32291==     in use at exit: 91,573 bytes in 3,013 blocks
==32291==   total heap usage: 5,285 allocs, 2,272 frees, 187,424 bytes allocated
==32291== 
==32291== LEAK SUMMARY:
==32291==    definitely lost: 0 bytes in 0 blocks
==32291==    indirectly lost: 0 bytes in 0 blocks
==32291==      possibly lost: 0 bytes in 0 blocks
==32291==    still reachable: 91,573 bytes in 3,013 blocks
==32291==         suppressed: 0 bytes in 0 blocks
==32291== Rerun with --leak-check=full to see details of leaked memory
==32291== 
==32291== For counts of detected and suppressed errors, rerun with: -v
==32291== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[1]    32291 abort      valgrind ./src/or/tor -f /tmp/crash --verify-config

Best Regards,
Stephan Zeisberg

Child Tickets

Attachments (1)

crash (13 bytes) - added by stze 6 weeks ago.

Download all attachments as: .zip

Change History (16)

Changed 6 weeks ago by stze

comment:1 Changed 6 weeks ago by nickm

  • Milestone set to Tor: 0.3.1.x-final

comment:2 Changed 6 weeks ago by arma

Bug confirmed. It affects at least 0.3.0.6 too.

comment:3 Changed 6 weeks ago by arma

This looks like a fun one:

In config.c:3538, in options_validate(), we have

    if (circuit_build_times_disabled(options)) {
      severity = LOG_INFO;
    }

In circuit_build_times_disabled(), we have

    int consensus_disabled = networkstatus_get_param(NULL, "cbtdisabled",
                                                     0, 0, 1);

In networkstatus_get_param(), we have

  if (!ns) /* if they pass in null, go find it ourselves */
    ns = networkstatus_get_latest_consensus();

In networkstatus_get_latest_consensus(), we have

  if (we_use_microdescriptors_for_circuits(get_options()))

And it's that get_options() that screws us -- it goes to fetch the global options value, and there isn't one yet since we're still in the middle of validating it.

comment:4 Changed 6 weeks ago by arma

Here's a more helpful torrc, rather than the binary one that's attached:

LearnCircuitBuildTimeout 0

comment:5 Changed 6 weeks ago by arma

  • Cc dgoulet arma added

Check out #21062 where we think we solved this bug before. (That patch went into 0.3.0.2-alpha.)

comment:6 Changed 6 weeks ago by arma

I tried out tor-0.3.0.2-alpha, and it asserts. I wonder if anybody tried the #21062 patch, and if it worked at any point?

comment:7 Changed 6 weeks ago by arma

Short-term fixes would be to either introduce the 'options' parameter to all of these functions too, or to refactor circuit_build_times_disabled() so it returns 1 as soon as it discovers a reason to return 1, so we check consensus_disabled last, since this chain of functions only happens when !options->LearnCircuitBuildTimeout. That latter fix is probably the right one for 0.3.0, if we choose to fix this bug there.

For 0.3.1 or 0.3.2, I'm curious if we have ideas for better fixes, since right now the requirement is "while validating or applying options, never call anything that would cause a call to get_options()" which is tricky to stick to because so many things call get_options().

comment:8 Changed 6 weeks ago by nickm

git-bisect results: This bug was introduced in 0.2.9.3-alpha, with one of these:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
c43211fd6cbb82a8016fcc0f81b309c6172e93d2
07d32d2e68daa6ef1ba03d8121998f619c409ff5
4d9d2553baa6856b1d85ec26baa1ac3d2c24832a
3e4a401ead701750218146edde939ef74ce8a5d0
831cf6d1d8a01e0538a4f1eeadc99455237325fb
0285f4f34d72b2b77f36fd55fa46216f6b54efc4
e5ad00330c7e4f63898a15ab6a4d833b732601a2
41f96078c23e3ef1c39a853841332cac3e133a94
75ebbed5576d402ef2929ee043ab2170bff5cc2b
65b2d34c9cb3434c26be71de6f725244444824a7
b560f852f220f5630f6bf5a300d15b40c9c235cf
a4f46ff8ba43b1e635bc5a8543b9354e6de02e14
We cannot bisect more!

comment:9 Changed 6 weeks ago by nickm

4d9d2553baa6856b1d85ec26baa1ac3d2c24832a would appear to be the offender.

comment:10 follow-up: Changed 6 weeks ago by nickm

  • Keywords 029-backport 030-backport added
  • Owner set to nickm
  • Status changed from new to accepted

Fix in branch bug22252_029 of my public repo should apply to 0.2.9 or later. I've opened another ticket (#22281) to fix this pattern of bug in 0.3.2.

An additional observation here -- maybe nobody sets LearnCircuitBuildTimeout 0? If they did, we would have had a report of this a long time ago.

comment:11 Changed 6 weeks ago by nickm

  • Actual Points set to .2
  • Status changed from accepted to needs_review

comment:12 Changed 6 weeks ago by catalyst

  • Cc catalyst added

comment:13 in reply to: ↑ 10 Changed 6 weeks ago by yawning

Replying to nickm:

Fix in branch bug22252_029 of my public repo should apply to 0.2.9 or later. I've opened another ticket (#22281) to fix this pattern of bug in 0.3.2.

The branch appears correct given the root cause analysis in the ticket.

comment:14 Changed 6 weeks ago by nickm

It appears that dgoulet fixed part of this already in #21062 , but the consensus-related part was still broken. So, merging and fixing conflict.

comment:15 Changed 6 weeks ago by nickm

  • Milestone changed from Tor: 0.3.1.x-final to Tor: 0.3.0.x-final
  • Resolution set to fixed
  • Status changed from needs_review to closed

Fixed in 0.3.0 and forward, not backporting to 0.2.9.

Note: See TracTickets for help on using tickets.