Make meek-server easy to use with Let's Encrypt

[dcf]

Currently it's not trivial to get certificates for meek-server using Let's Encrypt. The --webroot option, for example, wants to write a token to the filesystem so the web server can serve it, but meek-server doesn't serve files from the filesystem.

Ideally this works in a way that certificates can be renewed (e.g. in a cron job) without restarting tor or meek-server.

[dcf]

I posted a request for ideas to tor-dev.

​https://lists.torproject.org/pipermail/tor-dev/2016-March/010645.html

I'm looking for ideas of good ways to handle TLS certificates and their renewal for meek bridges. I want to use Let's Encrypt for this process, and I hope that someone who knows Let's Encrypt well can contribute some ideas.

All three of the meek bridges use HTTPS to receive connections from the CDN, so they need TLS certificates. For example, when you use meek-azure, your traffic is forwarded to the bridge at ​https://meek.bamsoftware.com/. How it works now is I do the usual domain validation procedure with a CA, receive an email to show that I control the domain, install the cert and key, and then run the server like this:

meek-server --cert /etc/meek/cert.pem --key /etc/meek/key.pem

When I used Let's Encrypt in the past, using the --webroot option, it wanted to write a file to the URL path /.well-known/acme-challenge/{token}. That won't work for meek-server as it exists today, because meek-server never serves files from the filesystem. But it could. Perhaps we could add an option like --acme-webroot that would allow serving files from a single whitelisted directory.

I notice that there are other ways of proving domain ownership (HTTP, TLS SNI, DNS). Maybe we could use one of those?
​https://ietf-wg-acme.github.io/acme/#identifier-validation-challenges

I also note that there are third-party plugins:
​https://github.com/letsencrypt/letsencrypt/wiki/Plugins
Maybe there could be a plugin for meek-server (ideally not requiring new complicated code in meek-server itself).

Currently you have to restart meek-server in order to make it notice a changed certificate and key file. It would be better if that were not necessary--maybe we could periodically stat the files, and re-load them if they have changed?

This is going to be an issue for Snowflake as well, because we will want to use WebSocket over TLS for the server component.

[dcf]

Here are some other links I found.

​Use on non-web servers? This question is more about using Let's Encrypt for e.g. IMAP or SMTP, but it's related in the sense that it's about a server type for which there is no ACME plugin.

​allow ports other than 443. Somewhat related, says that the HTTPS authenticators can only run on port 443. (If we were not running meek-server on 443, then it would be pretty easy to run Let's Encrypt in ​standalone mode, which binds to 443. But we are running meek-server on 443.)

[yawning]

​https://github.com/xenolf/lego

There's an acme library written in Go that will likely simplify this. Though to be honest the only annoying part is reloading the cert.

[yawning]

I stand corrected, as long as you use Go 1.6 or later, reloading the cert is trivial (See tls.Config.GetCertificate). dcf, is "you must build with Go 1.6 or later" a reasonable requirement for meek-server?

Not having to tear down the Listener on certificate reload is rather convenient. I can probably do the code for this in an evening if so...

[dcf]

Replying to yawning:

I stand corrected, as long as you use Go 1.6 or later, reloading the cert is trivial (See tls.Config.GetCertificate). dcf, is "you must build with Go 1.6 or later" a reasonable requirement for meek-server?

Good idea. Requiring Go 1.6 is fine.

[yawning]

Replying to dcf:

Replying to yawning:

I stand corrected, as long as you use Go 1.6 or later, reloading the cert is trivial (See tls.Config.GetCertificate). dcf, is "you must build with Go 1.6 or later" a reasonable requirement for meek-server?

Good idea. Requiring Go 1.6 is fine.

​https://git.schwanenlied.me/yawning/meek/commit/b749b5846115d10ba5cc409f5f150362bb4dae57

Untested, should work, if there's something wrong it should be trivial. Behavior is:

• If the cert/key fail to stat() or load when creating the listener, fail hard.
• On each incoming connection, if the mtime of either the cert or key has changed, reload the certificate, otherwise use the cached cert.
• Once the listener is created, failures to stat() or reload the certificate result in an error message logged at most once every 60s, and the existing cached certificate to be used.

Note that stat()-ing 2 files every incoming connection may be a tad expensive (though that's outside the mutex). Performance can be improved by only reloading once in a while (gettimeofday() is vDSO-ed on sensible systems), and by using a RWLock instead of a Mutex. Both to me are pre-mature optimizations since TLS handshake crypto blows both 2 syscalls and slight lock contention out of the water overhead wise.

I think the other half of this is easiest to implement via an --acme-webroot that lets meek-server serve files over port 80 like you mentioned in your tor-dev@ e-mail.

[dcf]

Replying to yawning:

​https://git.schwanenlied.me/yawning/meek/commit/b749b5846115d10ba5cc409f5f150362bb4dae57

Untested, should work, if there's something wrong it should be trivial. Behavior is:

• If the cert/key fail to stat() or load when creating the listener, fail hard.
• On each incoming connection, if the mtime of either the cert or key has changed, reload the certificate, otherwise use the cached cert.
• Once the listener is created, failures to stat() or reload the certificate result in an error message logged at most once every 60s, and the existing cached certificate to be used.

Note that stat()-ing 2 files every incoming connection may be a tad expensive (though that's outside the mutex). Performance can be improved by only reloading once in a while (gettimeofday() is vDSO-ed on sensible systems), and by using a RWLock instead of a Mutex. Both to me are pre-mature optimizations since TLS handshake crypto blows both 2 syscalls and slight lock contention out of the water overhead wise.

I think the other half of this is easiest to implement via an --acme-webroot that lets meek-server serve files over port 80 like you mentioned in your tor-dev@ e-mail.

Thanks. That's beautiful. I'll write tests for certContext.

It occurs to me that an --acme-webroot option will only work if meek-server is listening with TLS on port 443 or with non-TLS on port 80. So I think I'll check that one of those is true, and if not, throw an error at startup if the user tried to use --acme-webroot. If they have something else running on 80 or 443, then they'll have to work out their own thing, for example running letsencrypt in standalone mode.

[gtank]

Here's my branch using Russ Cox's LE client library. The mechanism is the same as Yawning's (replacing GetCertificate). It's a new dependency, but only about 50 lines of code in meek-server itself: ​http://github.com/gtank/meek/tree/letsencrypt

The example torrc works fine on a Debian machine and starts bridging without noticeable delay on first fetch. The biggest caveats I can think of are

1) It manages the keys and certificates in a cache file on disk. There are good reasons for this (​https://godoc.org/rsc.io/letsencrypt#hdr-Persistent_Storage) but it isn't strictly necessary.

2) If meek-server isn't listening on 443, this starts a new listener there since Let's Encrypt doesn't give you another option.

[dcf]

Replying to gtank:

Here's my branch using Russ Cox's LE client library. The mechanism is the same as Yawning's (replacing GetCertificate). It's a new dependency, but only about 50 lines of code in meek-server itself: ​http://github.com/gtank/meek/tree/letsencrypt

I want you to know that I used this idea in the WebSocket server plugin for Snowflake: comment:6:ticket:18654.
​https://gitweb.torproject.org/user/dcf/snowflake.git/diff/?h=letsencrypt&id=1f8be86a01&id2=af70d49e96
It's not deployed yet but I'm happy with how it works (review on #18654 is welcome too).

[dcf]

Replying to dcf:

Thanks. That's beautiful. I'll write tests for certContext.

I merged certContext in ​f7f792f28e and wrote tests in ​fe142d4ee3.

[dcf]

Here is a branch for review.

​https://gitweb.torproject.org/pluggable-transports/meek.git/log/?h=bug18655
​https://gitweb.torproject.org/pluggable-transports/meek.git/diff/?h=bug18655&id=1fcae3cad2&id2=93dd339f7

Let's Encrypt support is activated by the --acme-hostnames option, which specifies the set of hostnames for which the server will ask for certificates. For example,

ServerTransportPlugin meek exec /usr/local/bin/meek-server --acme-hostnames meek.bamsoftware.com --acme-email david@bamsoftware.com

I have this running right now on ​https://meek.bamsoftware.com/ (which is the bridge ​maenad).

[dcf]

Merged in ​e1fe61923fa2e22257615b65c5abfbcbc90dbceb.