HSv3 throws many "Tried connecting to router at [IP:port], but RSA identity key was not as expected"

A popular-ish HSv3 operator contacted me and told me that they've been getting lots of warnings on their logs:

[warn] Tried connecting to router at [IP:port], but RSA identity key was not as expected: wanted [hex string] + [base64 string] but got [same hex string] + no ed25519 key.

They are afraid it's some sort of downgrade attack. We should look into this.

changed milestone to %Tor: 0.3.2.x-final

added 032-backport 033-backport 034-backport 035-must 035-roadmap-proposed 035-triaged-in-20180711 certs component::core tor/tor ed25519 fast-fix handshake milestone::Tor: 0.3.2.x-final owner::teor priority::medium resolution::fixed reviewer::asn severity::normal status::closed tor-hs tor-relay type::defect version::tor 0.3.2.4-alpha labels

Any relay that causes this error on all its connections should be marked as not Running by the directory authorities, and removed from the consensus.

So here are two possibilities:

• the HS is connecting to Tor versions that have broken ed25519 handshakes
• the HS tor expects ed25519 in the handshake, even when the other Tor doesn't support ed25519 link handshakes
• clients are asking the HS to connect with an ed25519 key, but the relay doesn't actually have one

The operator could also provide some additional information:

• what Tor version are they running?
• are they using single onion services?
• are they using any kind of load balancing?
• do the IP, Port, and keys correspond to relays in a recent consensus?
  • if so, do those relays have anything in common?

Trac:
Status: new to needs_information

First, here's some general stats:

• 1956 occurrences in about a month.
• 653 different keys and 667 different (RSA public key, IP) combinations.
• 318 of these were seen only once and one was seen 34 times.
• Full breakdown: [318, 138, 66, 34, 24, 22, 16, 8, 7, 2, 7, 3, 4, 5, 1, 1, 2, 1, 0, 0, 1, 0, 3, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1]

What would you suggest is the best way of testing those possibilities?

To answer your questions:

• Partially v0.3.2.10, partially v0.3.3.7.
• Assuming you mean single-hop, yes.
• No load balancing.

I'm not sure how to answer the last question, can you point me to how can I query or view the consensus?

Trac:
Username: mahrud

Replying to mahrud:

I'm not sure how to answer the last question, can you point me to how can I query or view the consensus?

Thanks for the info!

To find your current consensus check in your DataDirectory. e.g. /var/lib/tor/cached-microdesc-consensus

Potentially triaging this into 035 because it seems like a wider issue client issue.

Trac:
Keywords: tor-hs deleted, security tor-relay certs handshake ed25519 035-roadmap-proposed added
Milestone: Tor: unspecified to Tor: 0.3.5.x-final

Replying to mahrud:

First, here's some general stats:

• 1956 occurrences in about a month.
• 653 different keys and 667 different (RSA public key, IP) combinations.
• 318 of these were seen only once and one was seen 34 times.
• Full breakdown: [318, 138, 66, 34, 24, 22, 16, 8, 7, 2, 7, 3, 4, 5, 1, 1, 2, 1, 0, 0, 1, 0, 3, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1]

What would you suggest is the best way of testing those possibilities?

To answer your questions:

• Partially v0.3.2.10, partially v0.3.3.7.
• Assuming you mean single-hop, yes.

Single onion services don't use guards, so they eventually connect to most relays. This means that they see more errors than most other onion services, which only connect to a few guards.

• No load balancing.

I'm not sure how to answer the last question, can you point me to how can I query or view the consensus?

We are particularly interested in the failing relay versions, because that helps us isolate the bug.

If you want to look up a few relays, you can use Relay Search: https://metrics.torproject.org/rs.html

You could also use Stem to look up the RSA fingerprints and dump the relay information: https://stem.torproject.org

If you'd like us to do the analysis, it should be safe to post the RSA/ed25519 pairs as an attachment to this ticket. But you must remove the timestamps from the log file, then destroy order by sorting the list.

Trac:
Username: mahrud

tor versions

Trac:
Username: mahrud

I assume ed25519 keys started in v0.3, so it makes sense that those relays wouldn't support ed25519 handshakes, but why do they have three ed25519 values in their descriptor?

Trac:
Username: mahrud

tor versions

The problematic tor versions are all 0.2.9.

0.2.9 and later support v3 rendezvous, but...

Replying to mahrud:

I assume ed25519 keys started in v0.3, so it makes sense that those relays wouldn't support ed25519 handshakes, but why do they have three ed25519 values in their descriptor?

0.3.0.1-alpha introduced ed25519 link authentication, using the ed25519 identity keys that had been around since 0.2.7.2-alpha. So all supported relays have ed25519 keys, but only relays on 0.3.0.1-alpha or later use them to authenticate their TLS connections:

In 0.3.0.1-alpha:

With the 0.3.0 series, clients and relays now use Ed25519 keys to authenticate their link connections to relays, rather than the old RSA1024 keys that they used before.

https://gitweb.torproject.org/tor.git/tree/ChangeLog#n5350

And in 0.2.7.2-alpha:

All relays now maintain a stronger identity key, using the Ed25519 elliptic curve signature format. This master key is designed so that it can be kept offline. Relays also generate an online signing key, and a set of other Ed25519 keys and certificates. These are all automatically regenerated and rotated as needed. Implements part of ticket 12498.

https://gitweb.torproject.org/tor.git/tree/ChangeLog#n9291

So there are two bugs here:

v3 clients sending introduce cells include an ed25519 key for 0.2.9 and earlier rend points, even though ed25519 link authentication can't possibly work for those rend points:

https://gitweb.torproject.org/tor.git/tree/src/feature/hs/hs_circuit.c#n610

v3 single onion service to rend link authentication is based on untrusted data from clients, so we should log at info, not warn:

https://gitweb.torproject.org/tor.git/tree/src/core/or/connection_or.c#n1961

(There would be a similar bug for v3 Tor2web client to intro, but Tor2web is not supported on v3.)

This bug is similar to #21107 (moved), where directory authorities marked 0.2.9 relays as not running, because they had ed25519 identity keys, but did not support authenticating their link handshakes with those keys. See, in particular:

https://trac.torproject.org/projects/tor/ticket/21107#comment:9

And the fix on the client side is a one-line fix similar to:

https://gitweb.torproject.org/nickm/tor.git/commit/?h=bug21107&id=0f79fb51e5653cbc82a0066423c833cafb656542

I'll do up a branch for 0.3.2 and 0.3.5.

Edit: better versions

Trac:
Status: needs_information to new

Please see my branches bug26627_032 and bug26627 on https://github.com/teor2345/tor.git

The Windows CI is failing on master due to #26662 (moved).

Here's what I fixed:

• backport #20895 (moved) and #23577 (moved) from 0.3.3 to 0.3.2
  • without the backport, clients can't check if the node supports ed25519 link auth
  • these backports also make v3 client intro behaviour consistent between 0.3.3+ and 0.3.2
• only send ed25519 link specifiers in client intros if the rend point supports ed25519 link auth

Here's what I still need to fix:

• check that we only send ed25519 link specifiers in service descriptors if the intro point supports ed25519 link auth
• make v3 single onion service to rend link authentication into a protocol warning
• (we don't need to make v3 Tor2web client to intro link authentication into a protocol warning, because there is no v3 Tor2web)

Trac:
Version: N/A to Tor: 0.3.2.4-alpha
Owner: N/A to teor
Keywords: security tor-relay certs handshake ed25519 035-roadmap-proposed deleted, tor-relay certs handshake ed25519 035-roadmap-proposed added
Status: new to assigned

Trac:
Status: assigned to needs_revision

• check that relays log a protocol warning (or lower) when a client asks them to extend to an ed25519 id, but the relay they're extending to doesn't support ed25519 link authentication

Adding this as fast-fix for 035. teor let me know if you are not up for this, and I will take over.

Trac:
Keywords: tor-relay certs handshake ed25519 035-roadmap-proposed deleted, tor-relay certs handshake ed25519 035-roadmap-proposed 035-must fast-fix added
Reviewer: N/A to asn

Trac:
Keywords: tor-relay certs handshake ed25519 035-roadmap-proposed 035-must fast-fix deleted, ed25519, handshake, 035-triaged-in-20180711, tor-relay, certs, 035-roadmap-proposed, 035-must, fast-fix added

Please see my branches bug26627_032 and bug26627_033 on https://github.com/teor2345/tor.git

• I did a forced update on bug26627_032 to rebase to get CI working
• bug26627_033 merges cleanly to master (see bug26627_033_merged_master)

Here's what I fixed:

• backport #20895 (moved) and #23577 (moved) from 0.3.3 to 0.3.2
  • without the backport, clients can't check if the node supports ed25519 link auth
  • these backports also make v3 client intro behaviour consistent between 0.3.3+ and 0.3.2
• only send ed25519 link specifiers in client intros if the rend point supports ed25519 link auth
• only send ed25519 link specifiers in service descriptors if the intro point supports ed25519 link auth

Relays already behave correctly:

• all relays log a protocol warning when a client asks them to extend to an ed25519 id, but the relay they're extending to doesn't support ed25519 link authentication

I'm going to split off into #26924 (moved):

• make v3 single onion service to rend link authentication into a protocol warning

Trac:
Status: needs_revision to needs_review

Travis CI passed all three branches:

• bug26627_032: https://travis-ci.org/teor2345/tor/builds/407846838
• bug26627_033: https://travis-ci.org/teor2345/tor/builds/407847054
• bug26627_033_merged_master: https://travis-ci.org/teor2345/tor/builds/407847143

Appveyor CI passed, it's only active for 0.3.5:

• bug26627_033_merged_master: https://ci.appveyor.com/project/teor2345/tor/build/1.0.8

Please see the bug26627_032 pull request for review: https://github.com/torproject/tor/pull/248

(bug26627_033 is a non-trivial merge due to changed context. bug26627_033_merged_master is a trivial merge of bug26627_033.)

Trac:
Keywords: N/A deleted, tor-hs added

Trac:
Keywords: N/A deleted, 034-backport, 032-backport, 033-backport added

LGTM!

Trac:
Status: needs_review to merge_ready

I've merged the 0.3.3 branches, and forward. Let's think a little more about whether we want to backport this to 0.3.2, though? It's only a warning, and the extra dependent backports seem a little tricky to me.

Trac:
Milestone: Tor: 0.3.5.x-final to Tor: 0.3.2.x-final

I'm OK with not backporting this to 032. Teor?

I don't think we need to backport to 0.3.2, because we're only supporting 0.3.2 for another 10 weeks.

Instead, we should encourage operators to upgrade to 0.3.3.

Trac:
Status: merge_ready to closed
Resolution: N/A to fixed

closed

mentioned in issue #26924 (moved)

mentioned in issue #26925 (moved)

moved to tpo/core/tor#26627 (closed)

mentioned in issue tpo/core/tor#26924 (closed)

mentioned in issue tpo/core/tor#26925 (closed)