Opened 4 months ago

Closed 4 months ago

Last modified 4 months ago

#33587 closed defect (fixed)

puppet certificate revocation anomaly

Reported by: anarcat Owned by: anarcat
Priority: High Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Major Keywords: tpa-roadmap-march
Cc: Actual Points: 0.2
Parent ID: Points:
Reviewer: Sponsor:

Description

today i revoked cupani's cert by mistake:

anarcat@curie:tsa-misc(master)$ ./retire -v -H cupani.torproject.org  retire-all -p unifolium.torproject.org 
checking for ganeti master on node unifolium.torproject.org
omeiense.torproject.org
polyanthum.torproject.org

instance cupani.torproject.org not running, no shutdown required
undefining instance cupani.torproject.org on host unifolium.torproject.org
error: failed to get domain 'cupani.torproject.org'
error: Domain not found: no domain with matching name 'cupani.torproject.org'

instance cupani.torproject.org not found on unifolium.torproject.org assuming retired: error: failed to get domain 'cupani.torproject.org'
error: Domain not found: no domain with matching name 'cupani.torproject.org'

scheduling cupani.torproject.org disk deletion on host unifolium.torproject.org
checking for path "/srv/vmstore/cupani.torproject.org/" on unifolium.torproject.org
scheduling rm -rf "/srv/vmstore/cupani.torproject.org/" to run on unifolium.torproject.org in 7 days
warning: commands will be executed using /bin/sh
job 4 at Tue Mar 17 17:45:00 2020
scheduling cupani.torproject.org backup disks removal on host bungei.torproject.org
checking for path "/srv/backups/bacula/cupani.torproject.org/" on bungei.torproject.org
scheduling rm -rf "/srv/backups/bacula/cupani.torproject.org/" to run on bungei.torproject.org in 30 days
warning: commands will be executed using /bin/sh
job 22 at Thu Apr  9 17:45:00 2020
Notice: Revoked certificate with serial 30
Notice: Removing file Puppet::SSL::Certificate cupani.torproject.org at '/var/lib/puppet/ssl/ca/signed/cupani.torproject.org.pem'
cupani.torproject.org
Submitted 'deactivate node' for cupani.torproject.org with UUID 7b5e6d74-cb31-4929-9082-4a2bcda08b88

i was following the migration procedure as part of #33446 and got over enthusiastic about the process. the cert shouldn't have been revoked, of course, as the machine is still up.

but when i tried to see the effect of this, it seemed the certificate still worked! cupani can do puppet runs without problems, even though the on-disk certificate is gone:

root@pauli:~# ls -al /var/lib/puppet/ssl/ca/signed/cupani.torproject.org.pem
ls: cannot access '/var/lib/puppet/ssl/ca/signed/cupani.torproject.org.pem': No such file or directory

so it seems our certificate revocation routine:

    con.run('puppet node clean %s' % instance)
    con.run('puppet node deactivate %s' % instance)

... does not work.

Child Tickets

Change History (3)

comment:1 Changed 4 months ago by anarcat

Owner: changed from tpa to anarcat
Status: newassigned

restarting puppetdb makes the catalog runs fail, which is good:

root@cupani:~# pat
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Could not retrieve facts for cupani.torproject.org: Failed to execute '/pdb/cmd/v1?checksum=83e3d9d88404f5f83bcd7db00c6466870eabd0a9&version=5&certname=cupani.torproject.org&command=replace_facts&producer-timestamp=2020-03-10T18:28:13.324Z' on at least 1 of the following 'server_urls': https://puppet.torproject.org:8081
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute '/pdb/cmd/v1?checksum=9c465faf636eea137c2391ed4cc74caf9daab225&version=5&certname=cupani.torproject.org&command=replace_facts&producer-timestamp=2020-03-10T18:28:16.845Z' on at least 1 of the following 'server_urls': https://puppet.torproject.org:8081
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

then I uncommented this line in the Apache configuration:

SSLCARevocationCheck chain

... and now the puppet run fails earlier:

root@cupani:~# pat
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate revoked
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate revoked
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate revoked
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate revoked
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate revoked
Info: Loading facts
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate revoked
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate revoked

comment:2 Changed 4 months ago by anarcat

Resolution: fixed
Status: assignedclosed

fixed in ecb3df5 fix puppet revocation procedures

comment:3 Changed 4 months ago by anarcat

Actual Points: 0.2
Keywords: tpa-roadmap-march added
Note: See TracTickets for help on using tickets.