Opened 4 years ago

Closed 3 years ago

Last modified 3 years ago

#18930 closed defect (user disappeared)

Segmentation fault: entry->parsed->intro_nodes

Reported by: juha Owned by: andrea
Priority: Medium Milestone:
Component: Core Tor/Tor Version: Tor: 0.2.7.1-alpha
Severity: Normal Keywords: tor-hs, tor2web
Cc: Actual Points:
Parent ID: Points: 1
Reviewer: dgoulet Sponsor:

Description

I have been running Tor on tor2web mode and now it has started to crash.

It crashes after few minutes of use. Before that it works normally.

Program received signal SIGSEGV, Segmentation fault.
0x00005555555c02d6 in rend_client_get_random_intro_impl (
   entry=0x5555555bf1bf <rend_client_fetch_v2_desc+114>, strict=0,
warnings=0)
   at src/or/rendclient.c:1353
1353      smartlist_add_all(usable_nodes, entry->parsed->intro_nodes);

Tor version:
Tor 0.2.7.6 (git-605ae665009853bd)

On log:
Apr 29 03:52:05.920 [notice] Tor v0.2.7.6 running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.1f and Zlib 1.2.8.
...
Apr 29 03:52:18.000 [notice] Tor has successfully opened a circuit. Looks like client functionality is working.
Apr 29 03:52:18.000 [notice] Bootstrapped 100%: Done

(gdb) bt full

#0  0x00005555555c02d6 in rend_client_get_random_intro_impl (
    entry=0x5555555bf1bf <rend_client_fetch_v2_desc+114>, strict=0, warnings=0)
    at src/or/rendclient.c:1353
        i = 32767
        intro = 0x1
        options = 0x555555a74770
        usable_nodes = 0x555556944060
        n_excluded = 0
        __PRETTY_FUNCTION__ = "rend_client_get_random_intro_impl"
        __func__ = "rend_client_get_random_intro_impl"
#1  0x00005555555c05de in rend_client_any_intro_points_usable (
    entry=0x5555555bf1bf <rend_client_fetch_v2_desc+114>)
    at src/or/rendclient.c:1422
        extend_info = 0x555556860078
        rv = 2054450743
#2  0x00005555555bfea1 in rend_client_desc_trynow (
    query=0x5555568ea1f0 "oahmssjdnck7ntzx") at src/or/rendclient.c:1217
        base_conn_sl_idx = 45
        base_conn_sl_len = 97
        base_conn = 0x5555567aef00
        conn = 0x5555567aef00
        entry = 0x5555555bf1bf <rend_client_fetch_v2_desc+114>
        rend_data = 0x5555568661e0
        now = 1461908592
        conns = 0x555555a5c890
        __PRETTY_FUNCTION__ = "rend_client_desc_trynow"
#3  0x00005555555bf366 in rend_client_refetch_v2_renddesc (
    rend_query=0x5555568ea1f0) at src/or/rendclient.c:921
        ret = 0
        e = 0x0
        __func__ = "rend_client_refetch_v2_renddesc"
        __PRETTY_FUNCTION__ = "rend_client_refetch_v2_renddesc"
#4  0x0000555555699e50 in connection_dir_about_to_close (
    dir_conn=0x5555568ee290) at src/or/directory.c:2312
---Type <return> to continue, or q <return> to quit---
        conn = 0x5555568ee290
#5  0x0000555555663b87 in connection_about_to_close_connection (
    conn=0x5555568ee290) at src/or/connection.c:704
        __func__ = "connection_about_to_close_connection"
#6  0x000055555558de1c in connection_unlink (conn=0x5555568ee290)
    at src/or/main.c:430
No locals.
#7  0x000055555558f4c6 in conn_close_if_marked (i=48) at src/or/main.c:982
        conn = 0x5555568ee290
        retval = 1452204688
        now = 1461908592
        __PRETTY_FUNCTION__ = "conn_close_if_marked"
        __func__ = "conn_close_if_marked"
#8  0x000055555558ec09 in close_closeable_connections () at src/or/main.c:760
        conn = 0x5555568ee290
        i = 0
#9  0x000055555558ed88 in conn_read_callback (fd=-1, event=2, 
    _conn=0x5555568ee290) at src/or/main.c:795
        conn = 0x5555568ee290
        __PRETTY_FUNCTION__ = "conn_read_callback"
#10 0x00007ffff7684f24 in event_base_loop ()
   from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
No symbol table info available.
#11 0x0000555555591da6 in run_main_loop_once () at src/or/main.c:2226
        loop_result = 0
        __PRETTY_FUNCTION__ = "run_main_loop_once"
#12 0x0000555555591ef6 in run_main_loop_until_done () at src/or/main.c:2267
        loop_result = 1
#13 0x0000555555591cbf in do_main_loop () at src/or/main.c:2198
        now = 1461907585
        __PRETTY_FUNCTION__ = "do_main_loop"
        __func__ = "do_main_loop"
#14 0x0000555555595d95 in tor_main (argc=1, argv=0x7fffffffe588)
    at src/or/main.c:3294
---Type <return> to continue, or q <return> to quit---
        result = 0
        __PRETTY_FUNCTION__ = "tor_main"
#15 0x000055555558d4a4 in main (argc=1, argv=0x7fffffffe588)
    at src/or/tor_main.c:30
        r = 32767

Child Tickets

Attachments (1)

get_bug18930_descriptor.py (221 bytes) - added by teor 4 years ago.
stem script to fetch a hidden service descriptor

Download all attachments as: .zip

Change History (20)

comment:1 Changed 4 years ago by arma

Component: - Select a componentCore Tor/Tor

comment:2 Changed 4 years ago by teor

Cc: dgoulet added
Keywords: crash added
Milestone: Tor: 0.2.8.x-final
Points: small
Severity: NormalCritical

Further Information

The line numbers in your backtrace don't seem to be the same as the line numbers I have for 0.2.7.6. I'm going to assume they're wrong, and work off the code listings.

It might help to provide the last few entries from an info-level or debug-level log.

query=0x5555568ea1f0 "oahmssjdnck7ntzx") at src/or/rendclient.c:1217
The hidden service in question is oahmssjdnck7ntzx.onion, it appears to be down.
(I've attached a stem script to fetch its descriptor, it says: stem.DescriptorUnavailable: No running hidden service at oahmssjdnck7ntzx.onion.)

Can you tell us if it's always the same hidden service causing the crash?

Analysis

The calls in this backtrace were removed in 0.2.8.2-alpha by dgoulet to fix #15937, a bug where tor over-enthusiastically cancelled connections if too many requests happened for the same hidden services in a short period of time.

This bug could cause all sorts of problems for busy tor2web instances connecting to busy hidden services. I can't see any obvious issues in the code, but I'd like others to have a look at:

  • rend_client_refetch_v2_renddesc (0.2.7.6)
  • rend_client_desc_trynow
  • rend_cache_lookup_entry
  • I wouldn't bother looking in detail at rend_client_any_intro_points_usable, as it crashes on the first line that tries to use a corrupt or NULL entry.

I wonder if you've found a race condition or something?

Suggested Solutions

You could try applying that patch from #15937 and see if it fixes your issue.
You could also try running 0.2.8.2-alpha, it should work for Tor2web, but it's still a little unstable.

Changed 4 years ago by teor

Attachment: get_bug18930_descriptor.py added

stem script to fetch a hidden service descriptor

comment:3 Changed 4 years ago by teor

Version: Tor: 0.2.7.6Tor: 0.2.7.1-alpha

This bug was likely introduced in commit 59f8dced in 0.2.7.1-alpha.

comment:4 Changed 4 years ago by teor

Keywords: CoreTorTeam201605 added

comment:5 Changed 4 years ago by teor

Keywords: must-fix-before-028-rc added

comment:6 Changed 4 years ago by nickm

Another question: 605ae665009853bd isn't a version of Tor that's in our repository, as far as I know. Where did you get your Tor?

comment:7 Changed 4 years ago by nickm

Keywords: TorCoreTeam201605 added

Calling all non-needs_information tickets for May.

comment:8 Changed 4 years ago by bugzilla

Keywords: CoreTorTeam201605 removed

comment:9 Changed 4 years ago by nickm

Status: newneeds_information

comment:10 Changed 4 years ago by andrea

Owner: set to andrea
Status: needs_informationassigned

comment:11 Changed 3 years ago by isabela

Points: small1

comment:12 Changed 3 years ago by dgoulet

Cc: dgoulet removed
Reviewer: dgoulet
Severity: CriticalNormal
Status: assignedneeds_information

We are still waiting on more information from juha here.

The i = 32767 value is super high... There is no chance we have that amount of usable intro points so probably that entry->parsed->intro_nodes has never been cleaned or a crazy amount of nodes have been added.

  i = crypto_rand_int(smartlist_len(usable_nodes));
  intro = smartlist_get(usable_nodes, i);

Also this: intro = 0x1 is simply not possible... So I think we don't have an accurate backtrace here.
This tor version is not from the tor repository thus no way I can confirm anything... Putting this one in needs_information.

comment:13 Changed 3 years ago by nickm

Keywords: must-fix-before-028-rc removed

comment:14 Changed 3 years ago by nickm

Keywords: 028-backport added
Milestone: Tor: 0.2.8.x-finalTor: 0.2.9.x-final

we can take this in 0.2.9 and maybe even backport, if we get a fix.

comment:15 Changed 3 years ago by nickm

Keywords: TorCoreTeam201605 removed

Remove "TorCoreTeam201605" keyword. The time machine is broken.

comment:16 Changed 3 years ago by dgoulet

Keywords: tor-hs added; crash 028-backport removed
Milestone: Tor: 0.2.9.x-finalTor: 0.2.???

comment:17 Changed 3 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:18 Changed 3 years ago by dgoulet

Resolution: user disappeared
Status: needs_informationclosed

I'll go in user disappeared mode here as 7 months is a long time.

comment:19 Changed 3 years ago by nickm

Milestone: Tor: 0.3.???

Milestone deleted

Note: See TracTickets for help on using tickets.