Opened 9 months ago

Closed 7 months ago

#28616 closed defect (fixed)

TLS internal error running Tor 0.3.4.9 on Debian Buster (OpenSSL 1.1.1a)

Reported by: filippo Owned by:
Priority: Medium Milestone: Tor: 0.4.0.x-final
Component: Core Tor/Tor Version: Tor: 0.3.4.9
Severity: Normal Keywords:
Cc: tor@… Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Running a simple Tor relay on Debian Buster seems to report TLS 1.3 related OpenSSL internal errors. Not entirely sure how a function named tls13_hkdf_expand can fail, but I'm sure OpenSSL found a way.

Nov 26 01:07:40.000 [warn] Unhandled OpenSSL errors found at ../src/common/buffers_tls.c:65:
Nov 26 01:07:40.000 [warn] TLS error: internal error (in SSL routines:tls13_hkdf_expand:---)
FROM debian:buster
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get install -y apt-transport-https gnupg ca-certificates

RUN echo "deb https://deb.torproject.org/torproject.org buster main" > /etc/apt/sources.list.d/tor.list
RUN echo "deb-src https://deb.torproject.org/torproject.org buster main" >> /etc/apt/sources.list.d/tor.list

RUN gpg --no-tty --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89
RUN gpg --no-tty --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add -

RUN apt-get update && apt-get install -y tor deb.torproject.org-keyring nyx

ADD torrc /etc/tor/torrc

RUN useradd --user-group --system --create-home tor
USER tor

RUN mkdir -p /home/tor/.tor/keys
VOLUME /home/tor/.tor

EXPOSE 9001

ENTRYPOINT ["tor"]
Nov 26 01:07:27.114 [notice] Tor 0.3.4.9 (git-de9ea9f0dfc5ecae) running on Linux with Libevent 2.1.8-stable, OpenSSL 1.1.1a, Zlib 1.2.11, Liblzma 5.2.2, and Libzstd 1.3.5.
Nov 26 01:07:27.114 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Nov 26 01:07:27.115 [notice] Read configuration file "/etc/tor/torrc".
Nov 26 01:07:27.117 [notice] Based on detected system memory, MaxMemInQueues is set to 5767 MB. You can override this by setting MaxMemInQueues by hand.
Nov 26 01:07:27.118 [notice] Scheduler type KIST has been enabled.
Nov 26 01:07:27.118 [notice] Opening OR listener on 0.0.0.0:9999
Nov 26 01:07:31.000 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip.
Nov 26 01:07:31.000 [notice] Parsing GEOIP IPv6 file /usr/share/tor/geoip6.
Nov 26 01:07:31.000 [notice] Configured to measure statistics. Look for the *-stats files that will first be written to the data directory in 24 hours from now.
Nov 26 01:07:31.000 [notice] Your Tor server's identity key fingerprint is 'ToBeAnnounced 2EC042F4274CC8A54381C78E8D1BF322FA26A095'
Nov 26 01:07:31.000 [notice] Bootstrapped 0%: Starting
Nov 26 01:07:39.000 [notice] Starting with guard context "default"
Nov 26 01:07:39.000 [notice] Bootstrapped 5%: Connecting to directory server
Nov 26 01:07:39.000 [notice] Bootstrapped 10%: Finishing handshake with directory server
Nov 26 01:07:39.000 [notice] Bootstrapped 50%: Loading relay descriptors
Nov 26 01:07:40.000 [warn] Unhandled OpenSSL errors found at ../src/common/buffers_tls.c:65:
Nov 26 01:07:40.000 [warn] TLS error: internal error (in SSL routines:tls13_hkdf_expand:---)
Nov 26 01:07:40.000 [notice] I learned some more directory information, but not enough to build a circuit: We're missing descriptors for 1/2 of our primary entry guards (total microdescriptors: 5519/6239).
Nov 26 01:07:41.000 [notice] I learned some more directory information, but not enough to build a circuit: We're missing descriptors for 1/2 of our primary entry guards (total microdescriptors: 5506/6327).
Nov 26 01:07:42.000 [warn] Unhandled OpenSSL errors found at ../src/common/buffers_tls.c:65:
Nov 26 01:07:42.000 [warn] TLS error: internal error (in SSL routines:tls13_hkdf_expand:---)
Nov 26 01:07:49.000 [notice] Bootstrapped 80%: Connecting to the Tor network
Nov 26 01:07:50.000 [notice] Bootstrapped 85%: Finishing handshake with first hop
Nov 26 01:07:50.000 [notice] Bootstrapped 90%: Establishing a Tor circuit
Nov 26 01:07:51.000 [notice] Tor has successfully opened a circuit. Looks like client functionality is working.
Nov 26 01:07:51.000 [notice] Bootstrapped 100%: Done

Child Tickets

Change History (14)

comment:1 Changed 9 months ago by arma

See #28613 for a very related issue but on windows

Is the issue openssl 1.1.1a?

comment:2 Changed 9 months ago by nickm

Found it! This is an issue with this change in openssl 1.1.1a from commit ec0c5f5693e39c5:

+/*
+ * RFC 8446, 7.1 Key Schedule, says:
+ * Note: With common hash functions, any label longer than 12 characters
+ * requires an additional iteration of the hash function to compute.
+ * The labels in this specification have all been chosen to fit within
+ * this limit.
+ */
+#define TLS13_MAX_LABEL_LEN     12

I've opened an OpenSSL issue at https://github.com/openssl/openssl/issues/7712 . This does not seem like something we can work around ourselves.

Last edited 9 months ago by nickm (previous) (diff)

comment:3 Changed 9 months ago by nickm

Milestone: Tor: 0.4.0.x-final

comment:4 Changed 9 months ago by gvanem

I had the same issue. I reported it to Tor-talk at 13-October:

https://lists.torproject.org/pipermail/tor-talk/2018-October/044547.html

comment:5 Changed 9 months ago by PsiBlade

Can confirm manually reverting ec0c5f5 works perfectly.

Last edited 9 months ago by PsiBlade (previous) (diff)

comment:6 Changed 9 months ago by nickm

There's a candidate fix for openssl as https://github.com/openssl/openssl/pull/7755 -- if anybody can test it out, that would be great.

comment:7 in reply to:  6 Changed 9 months ago by PsiBlade

Replying to nickm:

There's a candidate fix for openssl as https://github.com/openssl/openssl/pull/7755 -- if anybody can test it out, that would be great.

Cloned mattcaswell:tor-regression and rebuilt Tor 0.3.4.9 against it (Debian Stretch on ARM). No errors in Tor's logs so far.

Also manually patched the changes in 7755 into 1.1.1a (instead of 1.1.2-dev) on another machine (Also Tor 0.3.4.9). Works a treat.

Last edited 9 months ago by PsiBlade (previous) (diff)

comment:8 Changed 8 months ago by Pine64ARMv8

For Nick:
Patched openssl 1.1.1a with 7755:"Reduce stack usage in tls13_hkdf_expand" and there is no more "[warn] TLS error: internal error (in SSL routines:tls13_hkdf_expand:---)" in the tor log (Tor version 0.3.5.6-rc) (devuan tor server)

comment:9 Changed 8 months ago by arma

Do we have a handle on when openssl will come out with a new (fixed) version?

I ask because the tor-relays thread involves relays that are shutting down because of this bug. And we should either know that the new openssl is coming soon, or tell them to go ahead and run their relays anyway and don't worry about the potential breakage, or give them some other workaround.

(I thought of this today because a relay operator came by irc to report the bug. I pointed them to this ticket, but there isn't really a resolution here.)

comment:10 Changed 8 months ago by Big

It would be indeed important that there will be a fixed release, or Debian Buster will go stable with this bug and this will shift from only some relays are affected to realy many.

comment:11 Changed 8 months ago by nickm

I've asked on the ticket, but I don't have a great sense of when it will happen.

One possibility might be to write a patch to detect this issue and disable TLS 1.3 when it happens.

comment:12 in reply to:  11 Changed 8 months ago by nickm

Replying to nickm:

I've asked on the ticket, but I don't have a great sense of when it will happen.

They are not sure; it will either happen when the next major security fix happens, or probably 3-4 months after OpenSSL 1.1.1a.

One possibility might be to write a patch to detect this issue and disable TLS 1.3 when it happens.

I've done a minimal patch for this as #28973, but it needs testing, especially with OpenSSL 1.1.1a.

comment:13 Changed 8 months ago by Christian

Cc: tor@… added

Same here on Alpine Linux:

$ apk list -I tor libssl1.1 | awk '{print $1}'
tor-0.3.4.9-r1
libssl1.1-1.1.1a-r1

(Sorry for the noise, I just wanted to subscribe to this ticket to receive updates.)

comment:14 Changed 7 months ago by nickm

Resolution: fixed
Status: newclosed

The workaround for this was committed as #28973. It's not a pretty workaround, but the need should go away entirely once openssl 1.1.1b is out.

Note: See TracTickets for help on using tickets.