Opened 5 years ago

Closed 4 years ago

#16685 closed defect (fixed)

Abnormal behavior when signing key expires if Ed25519 master ID key is offline (missing from /datadirectory/keys)

Reported by: s7r Owned by: nickm
Priority: Medium Milestone: Tor: 0.2.7.x-final
Component: Core Tor/Tor Version: Tor: 0.2.7.2-alpha
Severity: Keywords: ed25519, key, offline, id, keys, TorCoreTeam201508
Cc: nickm, arma, asn Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Using Tor 0.2.7.2-alpha from deb.tp.o/tor-nightly-master-wheezy on Debian Wheezy 64 bit.

Want to test the behavior in all possible scenarios for the offline key feature.

I have included in my torrc a short signing key lifetime, to see how Tor behaves. Previosuly I have exported the Ed25519 master id key from /var/lib/tor/keys.

torrc setting:
SigningKeyLifetime 2 days

Tor started fine. Nothing in the log file was warning that the signing key and cert will expire in short time.

After about ~9 hours of usage, the relay went offline according to network status, but Tor was running on the server just fine. This is everything gathered from the logs:

Jul 27 21:16:00.000 [notice] Bootstrapped 100%: Done
Jul 27 21:17:01.000 [notice] Performing bandwidth self-test...done.
Jul 28 03:15:57.000 [notice] Heartbeat: Tor's uptime is 6:00 hours, with 135 circuits open. I've sent 728.54 MB and received 735.52 MB.
Jul 28 03:15:57.000 [notice] Circuit handshake stats since last time: 3409/3409 TAP, 4878/4878 NTor.
Jul 28 03:15:57.000 [notice] Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 3894 v4 connections; and received 0 v1 connections, 11 v2 connections, 0 v3 connections, and 4673 v4 connections.
Jul 28 06:25:02.000 [notice] Received reload signal (hup). Reloading config and resetting internal state.
Jul 28 06:25:02.000 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc".
Jul 28 06:25:02.000 [notice] Configuration file "/etc/tor/torrc" not present, using reasonable defaults.
Jul 28 06:25:02.000 [notice] Opening Socks listener on 127.0.0.1:9050
Jul 28 06:25:02.000 [notice] Closing no-longer-configured Control listener on 127.0.0.1:9051
Jul 28 06:25:02.000 [notice] Closing no-longer-configured OR listener on <ipv6>:port
Jul 28 06:25:02.000 [notice] Closing no-longer-configured OR listener on <ipv4>:port

12 hours later:

Jul 28 18:13:28.000 [warn] Couldn't find $HOME environment variable while expanding "~/.torrc"; defaulting to "".
Jul 28 18:13:28.000 [notice] Configuration file "/etc/tor/torrc" not present, using reasonable defaults.

I have the torrc correct file in /etc/tor/torrc but Tor says it is not there. Nothing changed/altered it. Triple checked - it is there, it is correct and it was working just fine.

Tor also generated a 'keys' folder with full path: /root/.tor/keys which I didn't order. This folder contains these files:
ed25519_master_id_public_key
ed25519_master_id_secret_key
ed25519_signing_cert
ed25519_signing_secret_key

So, it created another (unrequested) Ed25519 identity. Wild thoughts about this.

  • How was it able to write in /root/.tor/ ($HOME or ~/) and why this path? Shouldn't that be owned by root? I run Tor as debian-tor;
  • Does it use this identity instead? There is nothing in the log files about the directory authorities rejecting this relay.
  • Is it still publishing descriptors? Is it using the new unrequested Ed25519 identity? Or is it just publishing descriptors without Ed25519 identity (just with the old style RSA one)?

The previous folder, /var/lib/tor/keys is untouched. All files are there except master ID key which I exported offline, so here I found what was expected.

When Tor starts, it finds the relay's ID old style fingerprint based on the RSA key. Changed the log to info level, and based on this I assume it is actually publishing descriptors:

Jul 28 18:36:28.000 [info] init_keys(): Reading/making identity key "/var/lib/tor/keys/secret_id_key"...
Jul 28 18:36:28.000 [info] tor_getpwnam(): Caching new entry debian-tor for debian-tor
Jul 28 18:36:28.000 [info] tor_getpwnam(): Caching new entry debian-tor for debian-tor
Jul 28 18:36:28.000 [info] init_keys(): Reading/making onion key "/var/lib/tor/keys/secret_onion_key"...
Jul 28 18:36:28.000 [info] mark_my_descriptor_dirty(): Decided to publish new relay descriptor: set onion key

After few minutes, I get this warning (tor process still running):
Jul 28 19:15:10.000 [warn] ConnLimit must be at least 32. Failing.

What needs to be fixed as fast as possible:

  • Make sure Tor is aware and issues warnings in logs that the ed25519 signing key and key-cert are about to expire with reasonable time in advance it becomes effective. Don't start relay functionality if key-cert is expired, don't use it to build an invalid descriptor and don't generate unrequested new Ed25519 identity key;
  • Make sure Tor disables relay functionality and defaults to PublishServerDescriptor 0 in case it has an Ed25519 public key in /datadirectory/keys but an expired signing key and key-cert. Leave client functionality running, since that should not be affected and maybe it is also used on that server;
  • Don't generate Ed25519 identities unless requested or otherwise totally missing from /datadirectory/keys (example: when we configure a fresh relay from scratch with no identity). Do not proceed with generating a new Ed25519 identity if we already have an Ed25519 public key in /datadirectory/keys;
  • Be aware if the directory authorities reject us because our RSA key <-> Ed25519 key pair changed. Stop publishing descriptors in this case and disable relay functionality and wait for user's action, or automatically generate new fresh full ID (both RSA id key and Ed25519 id key unencrypted) so the realy can be included in the consensus;

Child Tickets

Change History (7)

comment:1 Changed 5 years ago by s7r

Found a part of what was wrong.

The signing key and key-cert were generated in /datadirectory/keys by Tor before I changed the SigningKeyLifetime to 2 (so assume the key-cert was generated with a default validity period of 30 days). After I changed SigningKeyLifetime to 2 in torrc, I have reloaded Tor but left the old signing key and key-cert in /datadirectory/keys as they were with validity period of 30 days (only exported the ed25519 master ID key).

I have deleted the signing key manually and the key cert from /datadirectory/keys. Now Tor won't start:

Jul 29 03:37:44.000 [warn] Missing identity key
Jul 29 03:37:44.000 [err] do_main_loop(): Bug: Error initializing keys; exiting (on Tor 0.2.7.2-alpha-dev )

So this answers my question if it's using another unrequested ed25519 identity: No, it is not.

Also manually deleted /root/.tor/keys folder and it wasn't generated again when tried to start Tor and it failed. Still investigating why did that appear there in the first place.

Now I have copied the previous initial Ed25519 master ID key (which I exported offline) to /var/lib/tor/keys, started Tor with SigningKeyLifetime 2 days in torrc and it correclty generated a signing key and key-cert - it did not fail to start. Exported the ed25519 master ID key offline again and tried a service tor reload. I got this again:

Jul 29 03:47:48.000 [notice] Received reload signal (hup). Reloading config and resetting internal state.
Jul 29 03:47:48.000 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc".
Jul 29 03:47:48.000 [notice] Configuration file "/etc/tor/torrc" not present, using reasonable defaults.
Jul 29 03:47:48.000 [notice] Opening Socks listener on 127.0.0.1:9050
Jul 29 03:47:48.000 [notice] Closing no-longer-configured Control listener on 127.0.0.1:9051
Jul 29 03:47:48.000 [notice] Closing no-longer-configured OR listener on <ipv6>:port
Jul 29 03:47:48.000 [notice] Closing no-longer-configured OR listener on <ipv4>:port

The signing key and key-cert available now are valid for at least 48 hours. If this is too short for Tor, it should warn accordingly, not ignore the torrc file in /etc/tor/torrc which worked few seconds before just fine.

Last edited 5 years ago by s7r (previous) (diff)

comment:2 Changed 5 years ago by s7r

I know why it generated an Ed25519 identity in /root/.tor/keys.

I tried to run:
# tor --keygen
Without a --datadirectory argument (I thought this command was actually saving the output/data in working directory - where I typed the command, in my case ~/workspace/tor, not default to $HOME/.tor/keys).

Anyway, it was a surprise for me and it didn't strike my mind when first opening this ticket, I didn't expect any keys to be generated at all, since the --keygen command failed with bug errors. See #16679 which is about why the command failed, but yet it still generates files in $HOME/.tor/keys.

This answers a second question: Is it generating unrequested Ed25519 identity keys? No, it is not. But we should improve the behavior and reporting as soon as possible.

comment:3 Changed 5 years ago by nickm

Keywords: TorCoreTeam201508 added
Owner: set to nickm
Status: newassigned

comment:4 Changed 5 years ago by s7r

All issues fixed in git-018082ef88b688e2 @nickm ed25519_keygen branch. We can close the ticket when we merge the changes into master.

comment:5 Changed 5 years ago by nickm

Status: assignedneeds_review

For this and other stuff, please review my ed25519_keygen branch.

comment:6 Changed 5 years ago by s7r

We discussed a lot on the mailing list about this and didn't update the ticket every time. If you are interested in exactly what ed25519_keygen branch fixes, see this thread:

https://lists.torproject.org/pipermail/tor-dev/2015-August/009204.html

comment:7 Changed 4 years ago by nickm

Resolution: fixed
Status: needs_reviewclosed

This should be solved in ed25519_keygen now, which is merged as ed25519_keygen_squashed

Note: See TracTickets for help on using tickets.