Opened 9 months ago

Closed 6 months ago

#33085 closed task (fixed)

decomission unifolium/kvm2, 6 VMs to migrate

Reported by: anarcat Owned by: anarcat
Priority: High Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords: tpa-roadmap-march
Cc: Actual Points:
Parent ID: Points: 20
Reviewer: Sponsor:

Description (last modified by anarcat)

  • [x] cupani.torproject.org (git-rw) migrated in #33446
  • [x] polyanthum.torproject.org (bridges) #33448
  • [x] omeiense.torproject.org (onionoo.torproject.org) (possibly to decom? see #32268) #33447
  • [x] savii.torproject.org (static content backend) retired in #33441
  • [x] build-x86-07.torproject.org (buildbox) retired in #33442)
  • [x] bracteata.torproject.org (sandstorm) retired in #32390

Requires a new gnt node (#33081).

Child Tickets

TicketStatusOwnerSummaryComponent
#33441closedanarcatdecomission saviiInternal Services/Tor Sysadmin Team
#33442closedanarcatdecomission build-x86-07Internal Services/Tor Sysadmin Team
#33446closedanarcatmigrate cupani/git-rw to the ganeti cluster, triggering an IP address changeInternal Services/Tor Sysadmin Team
#33447closedanarcatmigrate omeiense to the ganeti cluster, triggering an IP changeInternal Services/Tor Sysadmin Team
#33448closedanarcatMigrate IP address of polyanthum.torproject.org (BridgeDB)Internal Services/Tor Sysadmin Team

Change History (14)

comment:1 Changed 9 months ago by anarcat

Points: 5

comment:2 Changed 8 months ago by anarcat

Priority: MediumHigh

comment:3 Changed 8 months ago by anarcat

Description: modified (diff)
Status: newassigned

new node (fsn-node-04) online, ready to start migration. also, bracteata not required because migrated already.

comment:4 Changed 8 months ago by anarcat

Description: modified (diff)

comment:5 Changed 8 months ago by anarcat

Description: modified (diff)

comment:6 Changed 8 months ago by anarcat

Description: modified (diff)

savii decom'd, linked more tickets in summary

comment:7 Changed 8 months ago by anarcat

Description: modified (diff)

build-x86-07 done!

comment:8 Changed 8 months ago by anarcat

Keywords: tpa-roadmap-march added; tpa-roadmap-february removed

comment:9 Changed 8 months ago by anarcat

Description: modified (diff)
Points: 520

this is taking much more time than i expected, partly because i'm automating much of this work with fabric.

but now, cupani has been migrated, and i'll head towards the other two next.

comment:10 Changed 7 months ago by anarcat

Description: modified (diff)

all vms moved, now i just need to retire unifolium!

comment:11 Changed 7 months ago by anarcat

Status: assignedaccepted

starting the decom process now.

comment:12 Changed 7 months ago by anarcat

  1. warned already: DONE
  2. removed from nagios: DONE
  3. N/A
  4. N/A
  5. scrubbing: DONE
  6. LDAP: DONE
    337 host=unifolium,ou=hosts,dc=torproject,dc=org
    objectClass: top
    objectClass: debianServer
    host: unifolium
    hostname: unifolium.torproject.org
    architecture: amd64
    admin: torproject-admin@torproject.org
    sshRSAHostKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eQYY314co1YKne5dUi7t5dfbbP40Oad7swbhnWO83aPbEdLYzyH3UtWdUq3bMvhfwzOOU2gdmufOFDZ9GJU7ZqZESPc1AwjSOYFAfzw9EUqOmLw5ayNvqRCQFtMbRjhz/DTDai43bJkZvDdFFHuCXbOdSO0WnElKx8uy2iyFHh+GbvFhDY9Q5CAzFfkBkZ1Ei89ytI0F4B9v9ZDzQmpqkaYJlQx1dm3TBLHFoRdZB6HIvD8xYHq9xVPqt9L+TKijJud1GIj+e3MaeClVfzRCx5ESekNKj2Kg9x4CMCODNoheMy8OLntQVFPSqKKQhtCKFP+/WZchNy50EhhWutvL root@unifolium
    sshRSAHostKey: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEl8F6smRLPt06epUYSJSgYE/i36v7PJvnz6ZSDCNxku root@unifolium
    description: KVM host
    access: restricted
    distribution: Debian
    ipHostNumber: 148.251.180.115
    ipHostNumber: 2a01:4f8:211:6e8::2
    l: Falkenstein, Saxony, Germany
    purpose: KVM host
    purpose: [[kvm2.torproject.org]]
    
  7. DNS: did as part of the magic grep
  8. Puppet: DONE along with backups:
    $ ./retire -v -H unifolium.torproject.org retire-all
    not wiping instance unifolium.torproject.org data: no parent host
    scheduling unifolium.torproject.org backup disks removal on host bungei.torproject.org
    checking for path "/srv/backups/bacula/unifolium.torproject.org/" on bungei.torproject.org
    scheduling rm -rf "/srv/backups/bacula/unifolium.torproject.org/" to run on bungei.torproject.org in 30 days
    warning: commands will be executed using /bin/sh
    job 23 at Sun Apr 12 14:58:00 2020
    Notice: Revoked certificate with serial 71
    Notice: Removing file Puppet::SSL::Certificate unifolium.torproject.org at '/var/lib/puppet/ssl/ca/signed/unifolium.torproject.org.pem'
    unifolium.torproject.org
    Submitted 'deactivate node' for unifolium.torproject.org with UUID 5b467408-c6c4-4623-adf8-d4381d29db9b
    
  9. tor-puppet: DONE
  10. tor-passwords: DONE
  11. let's encrypt: DONE (N/A)
  12. dnswl: DONE (N/A)
  13. backups purge scheduled in step 8: DONE
  14. docs: DONE
  15. upstream: DONE in now +7d ("Please note that this server will be cancelled on 20/03/2020 and all data will be deleted.")

the magic grep in this case was:

grep -nH -r -e 148.251.180.115 -e 2a01:4f8:211:6e8::2 -e kvm2.torproject.org  -e unifolium.torproject.org -e unifolium -e kvm2

an interesting tidbit: the kvm2.torproject.org string was in the server's "purpose" field in LDAP, so it might make sense to search for that. it also makes sense to search for the short names (unifolium and kvm2) as those yielded postives as well.

Last edited 7 months ago by anarcat (previous) (diff)

comment:13 Changed 7 months ago by anarcat

all done here, disks are wiped and the server will be deracked on friday.

comment:14 Changed 6 months ago by anarcat

Resolution: fixed
Status: acceptedclosed
Note: See TracTickets for help on using tickets.