Opened 2 months ago

Closed 2 months ago

Last modified 2 months ago

#30509 closed defect (fixed)

snowflake-broker certificate fetch failing after update

Reported by: cohosh Owned by:
Priority: Immediate Milestone:
Component: Circumvention/Snowflake Version:
Severity: Normal Keywords:
Cc: dcf Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I recently tried updating the snowflake broker and it now hangs at

2019/05/14 17:21:15 ACME hostnames: ["snowflake-broker.bamsoftware.com"]
2019/05/14 17:21:15 Starting HTTP-01 listener

upon startup. This happens even when I revert back to the old broker code.

Child Tickets

Change History (15)

comment:1 Changed 2 months ago by cohosh

Looks like we're rate-limited:

2019/05/14 17:12:51 http: TLS handshake error from [scrubbed]: 429 urn:acme:error:rateLimited: Error creating new cert [scrubbed] too man
y certificates already issued for exact set of domains: snowflake-broker.bamsoftware.com: see https://letsencrypt.org/docs/rate-limits/
2019/05/14 17:12:51 http: TLS handshake error from [scrubbed]: acme/autocert: missing certificate

comment:2 Changed 2 months ago by cohosh

I don't think I restarted the broker so many times to run into the 50 certificates/week limit... I'll try again in an hour or so and see if I can get one.

comment:3 Changed 2 months ago by cohosh

Turns out we actually bypassed the "5 duplicate certificate requests/week" limit.

A proposed fix is to add another domain to the cert so we generate a new certificate.

I've set up snowflake.agogagave.com to route to the snowflake-broker machine, but it's going to take a while for the DNS update to propagate so that we can pass the Let's Encrypt challenge.

comment:4 Changed 2 months ago by dcf

Cc: dcf added

The WebSocket bridge has a certificate cache:
https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/server/server.go?id=d865b7c252d3a7efd789a84757fc2635b1964921#n309
The broker doesn't cache certificates, but it should, to avoid this kind of problem.

In the meantime I can get a conventional cert, but we'll have to add --cert and --key options to the broker like meek-server has:
https://gitweb.torproject.org/pluggable-transports/meek.git/tree/meek-server/meek-server.go?h=0.33#n394

comment:5 Changed 2 months ago by cohosh

I'm still waiting for a DNS update, not sure how long that will take.

I'm okay with adding options for a conventional cert in the meantime.

comment:6 Changed 2 months ago by dcf

I've put a certificate and key in

/home/snowflake-broker/snowflake-broker.pem
/home/snowflake-broker/snowflake-broker.key

The reference I gave for --cert and --key options in comment:4 is more complicated than it needs to be, because it has some extra code to allow reloading the files at runtime. The minimal solution is easier than that. I think you just have to provide the filenames here:

err = server.ListenAndServeTLS("", "")

You may also have to unset the GetCertificate callback in the tls.Config, I'm not sure.

comment:7 Changed 2 months ago by cohosh

Status: newneeds_review

comment:8 in reply to:  6 Changed 2 months ago by cohosh

Replying to dcf:

You may also have to unset the GetCertificate callback in the tls.Config, I'm not sure.

No need for that: https://golang.org/pkg/net/http/#Server.ListenAndServeTLS

comment:9 in reply to:  7 Changed 2 months ago by dcf

Status: needs_reviewmerge_ready

Replying to cohosh:

How this look? https://github.com/cohosh/snowflake/compare/ticket30509

There's an extra blank line here, otherwise it looks good.

comment:10 Changed 2 months ago by cohosh

Hrm I got the error

2019/05/14 21:12:27 tls: private key does not match public key

comment:11 in reply to:  10 ; Changed 2 months ago by dcf

Replying to cohosh:

Hrm I got the error

2019/05/14 21:12:27 tls: private key does not match public key

It might be because snowflake-broker.pem contains a concatenation of an intermediate certificate and the snowflake-broker certificate. Try editing the file to swap the order of the certificates.

comment:12 in reply to:  11 Changed 2 months ago by dcf

Replying to dcf:

Replying to cohosh:

Hrm I got the error

2019/05/14 21:12:27 tls: private key does not match public key

It might be because snowflake-broker.pem contains a concatenation of an intermediate certificate and the snowflake-broker certificate. Try editing the file to swap the order of the certificates.

No, this was my mistake. I accidentally only put the intermediates in snowflake-broker.pem. Try again now, there should be 3 certificates total.

comment:13 Changed 2 months ago by cohosh

Resolution: fixed
Status: merge_readyclosed

*whew* back up and running :)

I merged the patch to master (and removed the extra line)

comment:14 in reply to:  3 Changed 2 months ago by dcf

Replying to cohosh:

I've set up snowflake.agogagave.com to route to the snowflake-broker machine, but it's going to take a while for the DNS update to propagate so that we can pass the Let's Encrypt challenge.

I think this would not have worked as a quick fix. The broker would get a certificate under the new name, but the Azure domain front would still have been pointing to the old name. So I would have had to update the CDN configuration to point to the new name (takes at least a coupld of days), or (worst case if I was unreachable) set up a whole new domain front and push a new release configured to use it. The SSL cert name isn't really a critical dependency, but the CDN configuration is. We should migrate the CDN configuration somewhere where we can share admin (or I can see if it's possible to delegate adminship on Azure). I opened #30510.

comment:15 in reply to:  4 Changed 2 months ago by dcf

Replying to dcf:

The WebSocket bridge has a certificate cache:
https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/server/server.go?id=d865b7c252d3a7efd789a84757fc2635b1964921#n309
The broker doesn't cache certificates, but it should, to avoid this kind of problem.

I opened #30512.

Note: See TracTickets for help on using tickets.