Opened 23 months ago

Closed 22 months ago

Last modified 18 months ago

#17752 closed defect (fixed)

Null pointer deref in connection_ap_attach_pending()

Reported by: dgoulet Owned by: nickm
Priority: Very High Milestone: Tor: 0.2.8.x-final
Component: Core Tor/Tor Version: Tor: unspecified
Severity: Major Keywords: crash, TorCoreTeam201512, 2016-bug-retrospective
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

My tor client is running on version 0.2.8.0-alpha-dev (git-ee5337e90497e31c) and I got a crash with a coredump this morning. It happened when one of my hidden service was rebooted and then the torsocks client did try to reconnect.

Last notice log:

Dec 04 11:39:16.000 [notice] Closing stream for 'SCRUBBED ONION': hidden service is unavailable (try again later).

Here is the gdb backtrace of the coredump:

[snip]
#3  <signal handler called>
No locals.
#4  connection_ap_attach_pending (retry=retry@entry=1) at src/or/connection_edge.c:801
        conn = 0x0
        entry_conn_sl_idx = 3
        entry_conn_sl_len = 4
        entry_conn = 0x0
        __FUNCTION__ = "connection_ap_attach_pending"
        __func__ = "connection_ap_attach_pending"
#5  0x0000561584871bf4 in connection_ap_rescan_and_attach_pending () at src/or/connection_edge.c:779
        entry_conn = 0x561586bcc260
        conns = <optimized out>
        __FUNCTION__ = "connection_ap_rescan_and_attach_pending"
#6  0x0000561584851da8 in circuit_build_needed_circs (now=now@entry=1449247161) at src/or/circuituse.c:1126
        options = 0x561586bcc260
#7  0x00005615847c8288 in run_scheduled_events (now=1449247161) at src/or/main.c:1491
        options = 0x561586bcc260
        have_dir_info = <optimized out>
        i = <optimized out>
[snip]

Apparently conn is NULL at that point thus this line exploded insrc/or/connection_edge.c

    connection_t *conn = ENTRY_TO_CONN(entry_conn);
    if (conn->marked_for_close) {

Child Tickets

Change History (13)

comment:1 Changed 23 months ago by teor

Keywords: crash added
Version: Tor: unspecified

This could have been introduced in the #17590 refactor in 0.2.8.
#17659 could also be related - it's in the same area of the code, and appeared after #17590 was merged.

comment:2 Changed 23 months ago by nickm

Keywords: TorCoreTeam201512 added
Priority: MediumVery High

comment:3 Changed 22 months ago by nickm

I added a couple of assertions in a4ca2ef ; I don't see how a NULL could possibly be getting to that point, but it's worth a try to investigate it.

comment:4 Changed 22 months ago by nickm

See the stack trace in #17874 ; I think that's an explanation for this.

comment:5 Changed 22 months ago by nickm

Status: newneeds_information

I think I got it maybe? #17876 was the root bug here. The commits that fix this bug are, I think:

613e0e1c1ac3e44bad7a876147c49bc232460df2
24fcb6adbb3896395edda38d6ecccb6ad53bddbd

Please let me know if this recurs at any more recent version of Tor. Thanks!

comment:6 Changed 22 months ago by dgoulet

Status: needs_informationneeds_revision

New stack trace. I can reproduce that reliably every time I disconnect from my VPN and try to connect to an HS after that:

Dec 17 16:13:58.000 [notice] Our IP address has changed.  Rotating keys...
Dec 17 16:13:58.000 [notice] Tor now sees network activity. Restoring circuit build timeout recording. Network was down for 184 seconds during 131 circuit attempts.
Dec 17 16:14:05.000 [err] tor_assertion_failed_(): Bug: src/or/connection_edge.c:806: connection_ap_attach_pending: Assertion conn && entry_conn failed; aborting. (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug: Assertion conn && entry_conn failed in connection_ap_attach_pending at src/or/connection_edge.c:806. Stack trace: (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/local/bin/tor(log_backtrace+0x42) [0x564effb77d42] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/local/bin/tor(tor_assertion_failed_+0x8d) [0x564effb85fcd] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/local/bin/tor(connection_ap_attach_pending+0x109) [0x564effb2eb19] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/local/bin/tor(circuit_build_needed_circs+0x38) [0x564effb0e4d8] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/local/bin/tor(+0x3c118) [0x564effa84118] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5(event_base_loop+0x7fc) [0x7f1bde6dea0c] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/local/bin/tor(do_main_loop+0x20d) [0x564effa84b5d] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/local/bin/tor(tor_main+0x19ad) [0x564effa880cd] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/local/bin/tor(main+0x19) [0x564effa80949] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f1bdda5da00] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)
Dec 17 16:14:05.000 [err] Bug:     /usr/local/bin/tor(_start+0x29) [0x564effa80999] (on Tor 0.2.8.0-alpha-dev b9714e1366a19dff)

comment:7 Changed 22 months ago by nickm

Okay, i think I figured it out. I'll work on a fix once I'm done with my next meeting.

comment:8 Changed 22 months ago by nickm

Owner: set to nickm
Status: needs_revisionassigned

comment:9 Changed 22 months ago by nickm

Status: assignedneeds_review

Branch 17752_again in my public repository has a possible fix here.

comment:10 in reply to:  9 Changed 22 months ago by dgoulet

Replying to nickm:

Branch 17752_again in my public repository has a possible fix here.

It clearly fixes the issue for me.

I had an issue with hidden services that previously timed out client side and then I couldn't re-connect to them anymore. I can't reproduce it so in case I stumble upon this issue again, I'll open a new ticket with more info.

comment:11 Changed 22 months ago by nickm

Status: needs_reviewneeds_information

Okay, then I've merged this to master. I think this should solve it, but somebody please let me know if this recurs in the next few days.

comment:12 Changed 22 months ago by nickm

Resolution: fixed
Status: needs_informationclosed

This appears to be fixed.

comment:13 Changed 18 months ago by nickm

Keywords: 2016-bug-retrospective added

Marking these tickets (based on severity and hand-review) for inclusion in 2016 bug retrospective

Note: See TracTickets for help on using tickets.