Opened 4 weeks ago

Closed 12 days ago

#24313 closed defect (fixed)

Crash: died: Caught signal 11 [crash from rend_consider_services_intro_points]

Reported by: cypherpunks Owned by: dgoulet
Priority: High Milestone: Tor: 0.3.2.x-final
Component: Core Tor/Tor Version: Tor: 0.3.2.4-alpha
Severity: Normal Keywords: tor-hs
Cc: asn, dgoulet Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

[notice] Tor has successfully opened a circuit. Looks like client functionality is working.
[notice] Ignoring directory request, since no bridge nodes are available yet.
[notice] Ignoring directory request, since no bridge nodes are available yet.
[notice] Your network connection speed appears to have changed. Resetting timeout to 60s after 18 timeouts and 892 buildtimes.
[notice] Ignoring directory request, since no bridge nodes are available yet.
[notice] Our directory information is no longer up-to-date enough to build circuits: We're missing descriptors for 1/2 of our primary entry guards (total microdescriptors: 6124/6450).
[notice] I learned some more directory information, but not enough to build a circuit: We're missing descriptors for 1/2 of our primary entry guards (total microdescriptors: 6124/6450).
[notice] Ignoring directory request, since no bridge nodes are available yet.
[notice] Ignoring directory request, since no bridge nodes are available yet.
[notice] Ignoring directory request, since no bridge nodes are available yet.
[notice] Ignoring directory request, since no bridge nodes are available yet.
[notice] Ignoring directory request, since no bridge nodes are available yet.
[notice] Tor has successfully opened a circuit. Looks like client functionality is working.
[notice] Tor has successfully opened a circuit. Looks like client functionality is working.
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
============================================================ T= 1510821832
Tor 0.3.2.3-alpha (git-894b61b1d19f6235) died: Caught signal 11
/usr/bin/tor(+0x1830c9)[0x55c71b98d0c9]
/usr/bin/tor(extend_info_free+0x1d)[0x55c71b8e0add]
/usr/bin/tor(extend_info_free+0x1d)[0x55c71b8e0add]
/usr/bin/tor(rend_intro_point_free+0x25)[0x55c71b88a8d5]
/usr/bin/tor(rend_consider_services_intro_points+0x5c4)[0x55c71b894744]
/usr/bin/tor(hs_service_run_scheduled_events+0xa1c)[0x55c71b97c15c]
/usr/bin/tor(+0x4e931)[0x55c71b858931]
/usr/bin/tor(+0x6e6d0)[0x55c71b8786d0]
/usr/lib/x86_64-linux-gnu/libevent-2.0.so.5(event_base_loop+0x6a0)[0x7f0067487420]
/usr/bin/tor(do_main_loop+0x29d)[0x55c71b85be1d]
/usr/bin/tor(tor_main+0x1c25)[0x55c71b85f925]
/usr/bin/tor(main+0x19)[0x55c71b8575c9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f0065ea31c1]
/usr/bin/tor(_start+0x2a)[0x55c71b85761a]

Child Tickets

Change History (9)

comment:1 Changed 4 weeks ago by nickm

Milestone: Tor: 0.3.2.x-final
Priority: MediumHigh

comment:2 Changed 4 weeks ago by nickm

Cc: asn dgoulet added

comment:3 Changed 4 weeks ago by nickm

Summary: Tor 0.3.2.3-alpha crash: died: Caught signal 11Tor 0.3.2.3-alpha crash: died: Caught signal 11 [crash from rend_consider_services_intro_points]

comment:4 Changed 4 weeks ago by asn

Hey, in case you can help with debugging, were you on single onion services? And also, do you have info logs by any chance?

comment:5 in reply to:  4 Changed 4 weeks ago by cypherpunks

Replying to asn:

Hey, in case you can help with debugging, were you on single onion services? And also, do you have info logs by any chance?

No, I was running Ricochet (obviously not a single onion service) when it told me that Tor crashed. I don't think there are info logs.

comment:6 Changed 3 weeks ago by asn

Probs bug introduced by 1125a4876b455d41b4c858cc97e8f8feef0fa8d0 as part of #8239.

comment:7 Changed 3 weeks ago by cypherpunks

Summary: Tor 0.3.2.3-alpha crash: died: Caught signal 11 [crash from rend_consider_services_intro_points]Crash: died: Caught signal 11 [crash from rend_consider_services_intro_points]
Version: Tor: 0.3.2.4-alpha

Hey guys, happened to me again this time with Tor 0.3.2.4-a:

[notice] Your system clock just jumped 38626 seconds forward; assuming established circuits no longer work.
[notice] Tried for 38721 seconds to get a connection to [scrubbed]:9878. Giving up. (waiting for rendezvous desc)
[notice] Tried for 38721 seconds to get a connection to [scrubbed]:9878. Giving up. (waiting for rendezvous desc)
[notice] Tor has successfully opened a circuit. Looks like client functionality is working.
[notice] Tor has successfully opened a circuit. Looks like client functionality is working.
[notice] Our directory information is no longer up-to-date enough to build circuits: We're missing descriptors for 1/2 of our primary entry guards (total microdescriptors: 6162/6497).
[notice] I learned some more directory information, but not enough to build a circuit: We're missing descriptors for 1/2 of our primary entry guards (total microdescriptors: 6162/6497).
[notice] Tor has successfully opened a circuit. Looks like client functionality is working.
[notice] Tor has successfully opened a circuit. Looks like client functionality is working.
[notice] I learned some more directory information, but not enough to build a circuit: We're missing descriptors for 1/2 of our primary entry guards (total microdescriptors: 6273/6497).
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
[warn] Error launching circuit to node [scrubbed] for service [scrubbed].
============================================================ T= 1511254243
Tor 0.3.2.4-alpha (git-023228ae7a88cc0e) died: Caught signal 11
/usr/bin/tor(+0x183429)[0x556b1fd53429]
/usr/bin/tor(extend_info_free+0x1d)[0x556b1fca6d0d]
/usr/bin/tor(extend_info_free+0x1d)[0x556b1fca6d0d]
/usr/bin/tor(rend_intro_point_free+0x25)[0x556b1fc50875]
/usr/bin/tor(rend_consider_services_intro_points+0x5c4)[0x556b1fc5a6e4]
/usr/bin/tor(hs_service_run_scheduled_events+0xa1c)[0x556b1fd424fc]
/usr/bin/tor(+0x4e941)[0x556b1fc1e941]
/usr/bin/tor(+0x6e670)[0x556b1fc3e670]
/usr/lib/x86_64-linux-gnu/libevent-2.0.so.5(event_base_loop+0x6a0)[0x7fdf8923a420]
/usr/bin/tor(do_main_loop+0x29d)[0x556b1fc21e2d]
/usr/bin/tor(tor_main+0x1c25)[0x556b1fc25935]
/usr/bin/tor(main+0x19)[0x556b1fc1d5d9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fdf87c5a1c1]
/usr/bin/tor(_start+0x2a)[0x556b1fc1d62a]

comment:8 Changed 3 weeks ago by dgoulet

Keywords: tor-hs added
Owner: set to dgoulet
Status: newaccepted

Ok theory I have so far with this.

I think the problem could be in remove_invalid_intro_points(). We have a intro point with a node_t but no intro circuit, so the behavior is to add that node to the retry_nodes list.

Then, just after, it is possible for that intro point to expire which in that case, we'll move it to the expiring_nodes list and remove it from the working intro point list.

Then, we are unable to launch an intro circuit (see from the log above) and we remove it from the intro list (not in there so doesn't do anything) and we free() it.

Next tor main loop (a second after), we go again through remove_invalid_intro_points() which will free the intro point object in the expiring list if no intro circuit (which is the case) leading to a double free.

So far, this is the only thing I got that could explain this stacktrace... If I trick tor into going in that path, I get a heap use after free so I suspect that is at least an issue to fix.

comment:9 Changed 12 days ago by asn

Resolution: fixed
Status: acceptedclosed

Fixed as part of #24430.

Note: See TracTickets for help on using tickets.