Opened 6 months ago

Closed 5 months ago

#26627 closed defect (fixed)

HSv3 throws many "Tried connecting to router at [IP:port], but RSA identity key was not as expected"

Reported by: asn Owned by: teor
Priority: Medium Milestone: Tor: 0.3.2.x-final
Component: Core Tor/Tor Version: Tor: 0.3.2.4-alpha
Severity: Normal Keywords: tor-hs, tor-relay, certs, handshake, ed25519, 035-roadmap-proposed, 035-must, fast-fix, 035-triaged-in-20180711, 032-backport, 033-backport, 034-backport
Cc: Actual Points:
Parent ID: Points:
Reviewer: asn Sponsor:

Description

A popular-ish HSv3 operator contacted me and told me that they've been getting lots of warnings on their logs:

[warn] Tried connecting to router at [IP:port], but RSA identity key was not as expected: wanted [hex string] + [base64 string] but got [same hex string] + no ed25519 key.

They are afraid it's some sort of downgrade attack. We should look into this.

Child Tickets

TicketStatusOwnerSummaryComponent
#24868closedteorCheck a consensus parameter before activating onion service IPv6 featuresCore Tor/Tor
#26924closedteorMake single onion service to rend and Tor2web to intro link authentication into a protocol warningCore Tor/Tor
#26925closedteorMake link specifier handling in rend-spec-v3 more preciseCore Tor/Tor

Attachments (1)

Figure_1.png (16.0 KB) - added by mahrud 6 months ago.
tor versions

Download all attachments as: .zip

Change History (23)

comment:1 Changed 6 months ago by teor

Status: newneeds_information

Any relay that causes this error on all its connections should be marked as not Running by the directory authorities, and removed from the consensus.

So here are two possibilities:

  • the HS is connecting to Tor versions that have broken ed25519 handshakes
  • the HS tor expects ed25519 in the handshake, even when the other Tor doesn't support ed25519 link handshakes
  • clients are asking the HS to connect with an ed25519 key, but the relay doesn't actually have one

The operator could also provide some additional information:

  • what Tor version are they running?
  • are they using single onion services?
  • are they using any kind of load balancing?
  • do the IP, Port, and keys correspond to relays in a recent consensus?
    • if so, do those relays have anything in common?

comment:2 Changed 6 months ago by mahrud

First, here's some general stats:

  • 1956 occurrences in about a month.
  • 653 different keys and 667 different (RSA public key, IP) combinations.
  • 318 of these were seen only once and one was seen 34 times.
  • Full breakdown: [318, 138, 66, 34, 24, 22, 16, 8, 7, 2, 7, 3, 4, 5, 1, 1, 2, 1, 0, 0, 1, 0, 3, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1]

What would you suggest is the best way of testing those possibilities?

To answer your questions:

  • Partially v0.3.2.10, partially v0.3.3.7.
  • Assuming you mean single-hop, yes.
  • No load balancing.

I'm not sure how to answer the last question, can you point me to how can I query or view the consensus?

comment:3 in reply to:  2 Changed 6 months ago by asn

Replying to mahrud:

I'm not sure how to answer the last question, can you point me to how can I query or view the consensus?

Thanks for the info!

To find your current consensus check in your DataDirectory. e.g. /var/lib/tor/cached-microdesc-consensus

comment:4 Changed 6 months ago by asn

Keywords: security tor-relay certs handshake ed25519 035-roadmap-proposed added; tor-hs removed
Milestone: Tor: unspecifiedTor: 0.3.5.x-final

Potentially triaging this into 035 because it seems like a wider issue client issue.

comment:5 in reply to:  2 Changed 6 months ago by teor

Replying to mahrud:

First, here's some general stats:

  • 1956 occurrences in about a month.
  • 653 different keys and 667 different (RSA public key, IP) combinations.
  • 318 of these were seen only once and one was seen 34 times.
  • Full breakdown: [318, 138, 66, 34, 24, 22, 16, 8, 7, 2, 7, 3, 4, 5, 1, 1, 2, 1, 0, 0, 1, 0, 3, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1]

What would you suggest is the best way of testing those possibilities?

To answer your questions:

  • Partially v0.3.2.10, partially v0.3.3.7.
  • Assuming you mean single-hop, yes.

Single onion services don't use guards, so they eventually connect to most relays.
This means that they see more errors than most other onion services, which only connect to a few guards.

  • No load balancing.

I'm not sure how to answer the last question, can you point me to how can I query or view the consensus?

We are particularly interested in the failing relay versions, because that helps us isolate the bug.

If you want to look up a few relays, you can use Relay Search:
https://metrics.torproject.org/rs.html

You could also use Stem to look up the RSA fingerprints and dump the relay information:
https://stem.torproject.org

If you'd like us to do the analysis, it should be safe to post the RSA/ed25519 pairs as an attachment to this ticket.
But you must remove the timestamps from the log file, then destroy order by sorting the list.

Changed 6 months ago by mahrud

Attachment: Figure_1.png added

tor versions

comment:6 Changed 6 months ago by mahrud

I assume ed25519 keys started in v0.3, so it makes sense that those relays wouldn't support ed25519 handshakes, but why do they have three ed25519 values in their descriptor?

Last edited 6 months ago by mahrud (previous) (diff)

comment:7 in reply to:  6 Changed 6 months ago by teor

Status: needs_informationnew

tor versions

The problematic tor versions are all 0.2.9.

0.2.9 and later support v3 rendezvous, but...

Replying to mahrud:

I assume ed25519 keys started in v0.3, so it makes sense that those relays wouldn't support ed25519 handshakes, but why do they have three ed25519 values in their descriptor?

0.3.0.1-alpha introduced ed25519 link authentication, using the ed25519 identity keys that had been around since 0.2.7.2-alpha. So all supported relays have ed25519 keys, but only relays on 0.3.0.1-alpha or later use them to authenticate their TLS connections:

In 0.3.0.1-alpha:

With the 0.3.0 series, clients and relays now use Ed25519 keys to
authenticate their link connections to relays, rather than the old
RSA1024 keys that they used before.

https://gitweb.torproject.org/tor.git/tree/ChangeLog#n5350

And in 0.2.7.2-alpha:

All relays now maintain a stronger identity key, using the Ed25519
elliptic curve signature format. This master key is designed so
that it can be kept offline. Relays also generate an online
signing key, and a set of other Ed25519 keys and certificates.
These are all automatically regenerated and rotated as needed.
Implements part of ticket 12498.

https://gitweb.torproject.org/tor.git/tree/ChangeLog#n9291

So there are two bugs here:

v3 clients sending introduce cells include an ed25519 key for 0.2.9 and earlier rend points, even though ed25519 link authentication can't possibly work for those rend points:

https://gitweb.torproject.org/tor.git/tree/src/feature/hs/hs_circuit.c#n610

v3 single onion service to rend link authentication is based on untrusted data from clients, so we should log at info, not warn:

https://gitweb.torproject.org/tor.git/tree/src/core/or/connection_or.c#n1961

(There would be a similar bug for v3 Tor2web client to intro, but Tor2web is not supported on v3.)

This bug is similar to #21107, where directory authorities marked 0.2.9 relays as not running, because they had ed25519 identity keys, but did not support authenticating their link handshakes with those keys. See, in particular:

https://trac.torproject.org/projects/tor/ticket/21107#comment:9

And the fix on the client side is a one-line fix similar to:

https://gitweb.torproject.org/nickm/tor.git/commit/?h=bug21107&id=0f79fb51e5653cbc82a0066423c833cafb656542

I'll do up a branch for 0.3.2 and 0.3.5.

Edit: better versions

Last edited 6 months ago by teor (previous) (diff)

comment:8 Changed 6 months ago by teor

Keywords: security removed
Owner: set to teor
Status: newassigned
Version: Tor: 0.3.2.4-alpha

Please see my branches bug26627_032 and bug26627 on https://github.com/teor2345/tor.git

The Windows CI is failing on master due to #26662.

Here's what I fixed:

  • backport #20895 and #23577 from 0.3.3 to 0.3.2
    • without the backport, clients can't check if the node supports ed25519 link auth
    • these backports also make v3 client intro behaviour consistent between 0.3.3+ and 0.3.2
  • only send ed25519 link specifiers in client intros if the rend point supports ed25519 link auth

Here's what I still need to fix:

  • check that we only send ed25519 link specifiers in service descriptors if the intro point supports ed25519 link auth
  • make v3 single onion service to rend link authentication into a protocol warning
  • (we don't need to make v3 Tor2web client to intro link authentication into a protocol warning, because there is no v3 Tor2web)

comment:9 Changed 6 months ago by teor

Status: assignedneeds_revision

comment:10 Changed 6 months ago by teor

  • check that relays log a protocol warning (or lower) when a client asks them to extend to an ed25519 id, but the relay they're extending to doesn't support ed25519 link authentication

comment:11 Changed 5 months ago by asn

Keywords: 035-must fast-fix added
Reviewer: asn

Adding this as fast-fix for 035. teor let me know if you are not up for this, and I will take over.

comment:12 Changed 5 months ago by nickm

Keywords: 035-triaged-in-20180711 added

comment:13 in reply to:  10 Changed 5 months ago by teor

Please see my branches bug26627_032 and bug26627_033 on https://github.com/teor2345/tor.git

  • I did a forced update on bug26627_032 to rebase to get CI working
  • bug26627_033 merges cleanly to master (see bug26627_033_merged_master)

Here's what I fixed:

  • backport #20895 and #23577 from 0.3.3 to 0.3.2
    • without the backport, clients can't check if the node supports ed25519 link auth
    • these backports also make v3 client intro behaviour consistent between 0.3.3+ and 0.3.2
  • only send ed25519 link specifiers in client intros if the rend point supports ed25519 link auth
  • only send ed25519 link specifiers in service descriptors if the intro point supports ed25519 link auth

Relays already behave correctly:

  • all relays log a protocol warning when a client asks them to extend to an ed25519 id, but the relay they're extending to doesn't support ed25519 link authentication

I'm going to split off into #26924:

  • make v3 single onion service to rend link authentication into a protocol warning

comment:14 Changed 5 months ago by teor

Status: needs_revisionneeds_review

comment:15 Changed 5 months ago by teor

Travis CI passed all three branches:

Appveyor CI passed, it's only active for 0.3.5:

Please see the bug26627_032 pull request for review:
https://github.com/torproject/tor/pull/248

(bug26627_033 is a non-trivial merge due to changed context. bug26627_033_merged_master is a trivial merge of bug26627_033.)

comment:16 Changed 5 months ago by teor

Keywords: tor-hs added

comment:17 Changed 5 months ago by teor

Keywords: 032-backport 033-backport 034-backport added

comment:18 Changed 5 months ago by asn

Status: needs_reviewmerge_ready

LGTM!

comment:19 Changed 5 months ago by nickm

Milestone: Tor: 0.3.5.x-finalTor: 0.3.2.x-final

I've merged the 0.3.3 branches, and forward. Let's think a little more about whether we want to backport this to 0.3.2, though? It's only a warning, and the extra dependent backports seem a little tricky to me.

comment:20 Changed 5 months ago by asn

I'm OK with not backporting this to 032. Teor?

comment:21 Changed 5 months ago by teor

I don't think we need to backport to 0.3.2, because we're only supporting 0.3.2 for another 10 weeks.

Instead, we should encourage operators to upgrade to 0.3.3.

comment:22 Changed 5 months ago by teor

Resolution: fixed
Status: merge_readyclosed
Note: See TracTickets for help on using tickets.