Opened 2 months ago

Closed 5 weeks ago

Last modified 5 weeks ago

#27800 closed defect (fixed)

Non-fatal assertion !(old) failed in node_add_to_ed25519_map

Reported by: stefani Owned by: catalyst
Priority: Very High Milestone: Tor: 0.3.5.x-final
Component: Core Tor/Tor Version: Tor: 0.3.2.1-alpha
Severity: Normal Keywords: regression, 035-must
Cc: Actual Points: 3
Parent ID: Points:
Reviewer: nickm Sponsor:

Description (last modified by dgoulet)

this error showed up in my logs today:

(happened on the directory authority bastet)

[warn] tor_bug_occurred_(): Bug: ../src/or/nodelist.c:297: node_add_to_ed25519_map: Non-fatal assertion !(old) failed. (on Tor 0.3.4.8 )
 [warn] Bug: Non-fatal assertion !(old) failed in node_add_to_ed25519_map at ../src/or/nodelist.c:297. Stack trace: (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(log_backtrace+0x44) [0x560fb19f0354] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(tor_bug_occurred_+0xb9) [0x560fb1a0b2f9] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(+0x648b1) [0x560fb18cc8b1] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(nodelist_set_routerinfo+0x13f) [0x560fb18ce9ef] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(+0x9d7ec) [0x560fb19057ec] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(router_add_to_routerlist+0x814) [0x560fb190bb94] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(dirserv_add_descriptor+0x228) [0x560fb19aefa8] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(dirserv_add_multiple_descriptors+0x154) [0x560fb19af294] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(connection_dir_process_inbuf+0xabc) [0x560fb19a494c] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(connection_handle_read+0xa27) [0x560fb197d717] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(+0x53afe) [0x560fb18bbafe] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5(event_base_loop+0x6a0) [0x7fefd7df25a0] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(do_main_loop+0x265) [0x560fb18bde95] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(tor_run_main+0x1175) [0x560fb18c0685] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(tor_main+0x3a) [0x560fb18b84ba] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(main+0x19) [0x560fb18b8229] (on Tor 0.3.4.8 )
 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7fefd664e2e1] (on Tor 0.3.4.8 )
 [warn] Bug:     /usr/bin/tor(_start+0x2a) [0x560fb18b827a] (on Tor 0.3.4.8 )

Child Tickets

Change History (16)

comment:1 Changed 2 months ago by dgoulet

Component: Core TorCore Tor/Tor
Description: modified (diff)
Keywords: regression added
Milestone: Tor: 0.3.5.x-final
Priority: MediumHigh
Summary: assertion failed in nodelist.cNon-fatal assertion !(old) failed in node_add_to_ed25519_map

comment:2 Changed 7 weeks ago by nickm

Keywords: 035-must added

Add the 035-must tag to some assertion failures, hangs, ci-blockers, etc.

comment:3 Changed 7 weeks ago by nickm

Priority: HighVery High

Mark all 035-must tickets as "very high"

comment:4 Changed 5 weeks ago by nickm

note: ahf and I are going to try to figure this out....

comment:5 Changed 5 weeks ago by nickm

oops, copy-paste error. I meant: "note: catalyst and I are going to try to figure this out...."

comment:6 Changed 5 weeks ago by catalyst

Owner: set to catalyst
Status: newassigned

comment:7 Changed 5 weeks ago by catalyst

Current hypothesis: someone misconfigures their relay and regenerates their rsa1024 key while leaving their ed25519 key unchanged. nodelist_set_routerinfo() gets a new node from node_get_or_create(), which zeroes the ed25519_id, so node_remove_from_ed25519_map() is a no-op. On the other hand, node_add_to_ed25519_map() looks up the ed25519_id explicitly, which can then collide with an existing node_t.

We probably want to make this log (at LD_DIR?) without indicating a bug, because I'm not convinced it's an internal error. It should log at least both rsa1024 fingerprints and the colliding ed25519 key.

comment:8 Changed 5 weeks ago by catalyst

Version: Tor: 0.3.4.8Tor: 0.3.2.1-alpha

Commit 3cddd6570c6 added the code with the call to BUG(). This commit is contained in 0.3.2.1-alpha.

comment:9 Changed 5 weeks ago by catalyst

Looking at this some more, I'm not sure how this condition can happen if key pinning is functioning correctly. Maybe there are some paths that miss key pinning somehow? Maybe the key pinning table is out of sync with the ed25519 table?

comment:10 Changed 5 weeks ago by nickm

I think possibly bridges could be the problem: They don't always go through authorities, and so they aren't necessarily affected by key pinning

comment:11 Changed 5 weeks ago by catalyst

It looks like only direct uploads to authorities update key pinning. Downloads of missing descriptors from votes seem to go through the same code paths that clients use, so they bypass key pinning. Thanks to arma for helping to confirm this.

comment:12 Changed 5 weeks ago by catalyst

pull requests:

033
035

The 033 one merges cleanly into 034.

comment:13 Changed 5 weeks ago by catalyst

Status: assignedneeds_review

comment:14 Changed 5 weeks ago by nickm

Reviewer: nickm

comment:15 Changed 5 weeks ago by nickm

Resolution: fixed
Status: needs_reviewclosed

This looks clean to me, and CI is passing. Merging to 0.3.3 and forward. Thank you!

comment:16 Changed 5 weeks ago by catalyst

Actual Points: 3
Note: See TracTickets for help on using tickets.