Opened 5 weeks ago

Closed 2 weeks ago

#32383 closed task (fixed)

retire build-arm-* raspi boxes

Reported by: anarcat Owned by: anarcat
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by anarcat)

there are three boxes in our infra that are just too slow to provide the service they were designed for. they are the build-arm-0[123].torproject.org boxes and should be retired.

Child Tickets

Change History (7)

comment:1 Changed 4 weeks ago by anarcat

Description: modified (diff)
Owner: changed from anarcat to weasel

retirement checklist:

  1. hosts have long been unusable, ack'd (requested, even) by weasel
  2. N/A - will leave running so weasel can wipe the machines if needed
  3. N/A - not a VM
  4. N/A - will let weasel wipe the machine or destroy the hardware
  5. removed the hosts from ldap
  6. remove the records from the 172.30.0.0/16 zone (30.172.in-addr.arpa) and associated sbg namespace (commit 593b1a6 in tor/dns)
  7. remove the three hosts from puppet (for host in build-arm-01 build-arm-02 build-arm-03; do puppet node clean $host.torproject.org && puppet node deactivate $host.torproject.org; done)
  8. removed build-arm* traces from the puppet repo (2dcfd012 and da0b4daf])
  9. removed from tor-passwords
  10. removed from the spreadsheet and slight fix in wiki
  11. removed from nagios
  12. N/A hosts not on the backup server
  13. nothing in letsencrypt
  14. ping'd weasel for physical retirement and deletion
  15. not handling mail

those are the LDAP records removed in step 5, in case that's important:

419 host=build-arm-01,ou=hosts,dc=torproject,dc=org
host: build-arm-01
hostname: build-arm-01.torproject.org
objectClass: top
objectClass: debianServer
l: weasel's, Austria
access: restricted
admin: torproject-admin@torproject.org
description: arm build system
ipHostNumber: 172.30.115.11
distribution: Debian
architecture: arm64
purpose: buildbox
purpose: porterbox
sshRSAHostKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0nCJTls+EUO2I68O2PkHprbeNeTN0BNY3HJa1OEywsLs3/VaTKQmTaJRuVagvu6yaZqEivxa5Uu5I5zSF6PqE+pQeYhH13UGIcuz4UMaPIDozBjsxAf3YgOWxsWMEmGp/VTT/UGajicsdbf2EvU+eAmxAIJ2O2GeC100+9QkcEy5ztaqjb0NrpnDWZEq5Y7h9KZcJm6TKwTvVnSLxW62nwMMlMEtD0UlOfGpvv+eB/g4zBAZ78lYo6m4tBXkjNCIcw8VgxDtpFNSMD+CrxUQyA8mTXY3SB4n60OV7cWHrw2ERIY15/uO8wSdMuesrhEasO1pdxQGY6jofE0M7cZxZ root@build-arm-01
sshRSAHostKey: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA52bCa08CAPN2ud7TRY1XPFZFsqvwppFUh3PVk95I7e root@build-arm-01
machine: Raspberry Pi 3 Model B
allowedGroups: jenkins

420 host=build-arm-02,ou=hosts,dc=torproject,dc=org
host: build-arm-02
hostname: build-arm-02.torproject.org
objectClass: top
objectClass: debianServer
l: weasel's, Austria
access: restricted
admin: torproject-admin@torproject.org
description: arm build system
ipHostNumber: 172.30.115.12
distribution: Debian
architecture: arm64
purpose: buildbox
purpose: porterbox
sshRSAHostKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXuRZZPgwbYm82jSZvyQAz+0RtrrYZGYzdn/aX5r76GnM7Oq98/QwaKYl0oOdmn1ZASc+7XLJpNyB2acUpPLn9vhl6xh9WqBkN79dBJo6sHObSAooWn2LaXfWSPBer4njrnHHT6cGqb8iD8wQBXTctF9Smu8rSRuA7XxVfe6sFeoLDz3wz3IfmIdFB+x0h1xA/BFoLgntJb9mdZv30KUEObOb2yKVO2944gCcFyzO21z285mghFoQkyHeQDNotjXmKmDuf402/XKkBeY8IZ9v2HJhjp9wMtpifaNBH8WWhbbqACAjvq6ZszOR1rm00HojT5NjuT45RFK11JfKYdGy5 root@build-arm-02
sshRSAHostKey: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINzK47M11Ls4bTbBqsBPf71fwradRT7yg4QmblBTbnPe root@build-arm-02
machine: Raspberry Pi 3 Model B
allowedGroups: jenkins

421 host=build-arm-03,ou=hosts,dc=torproject,dc=org
host: build-arm-03
hostname: build-arm-03.torproject.org
objectClass: top
objectClass: debianServer
l: weasel's, Austria
access: restricted
admin: torproject-admin@torproject.org
description: arm build system
ipHostNumber: 172.30.115.13
distribution: Debian
architecture: arm64
purpose: buildbox
purpose: porterbox
sshRSAHostKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtGwC+Z1nxg43HHJGKUnkcyM1yU6HIaS8f0aSdEC/t3S26U30svMaS/PqXTNaqP3s6j3st8mAq/75X053/Qtin5Xv3Ye44IjiorKNu+s6TSOHl9Ra7l73VqPp6lu7QLQas1pexNkF8damAlM1UglS4jZ6KXM0bsXPMbqd/mHi/0udlgywdJJq0C0cDUT2wt1NXkoiupKub9AMjsr2ysknm32dvjMNiFz258Ro/ymYCksy7Ap3PEp6wFTizQAu9Gn/JhIgiC51ReaBtArxiLr7Sd5AAqM0ZfUx6ozfuseOzU9AtmX2iwlI57htEt/d1T0oEsUB4lKs9S2xy+TL3SSh root@build-arm-03
sshRSAHostKey: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHr61yI85pa4wxH7dOui75IhyCZMRjrh+tx9FKQUJxXo root@build-arm-03
machine: Raspberry Pi 3 Model B
allowedGroups: jenkins

step 5 also involved removing the subgroup here as well:

91 gid=buildusers,ou=users,dc=torproject,dc=org
gid: buildusers
objectClass: top
objectClass: debianGroup
gidNumber: 1523
subGroup: sbuild@build-arm-01.torproject.org
subGroup: sbuild@build-arm-02.torproject.org
subGroup: sbuild@build-arm-03.torproject.org

there are still some traces of the sbg network left which I haven't removed in case we still need to access the mikrotik for whatever reason:

tor-puppet/modules/torproject_org/misc/hoster.yaml:torsbg:
tor-puppet/modules/ipsec/templates/ferm.erb:peers << "141"+".201.12.0/23" # sbg mikrotik

There's also the hardcoded ipsec config everywhere that should probably be cleaned up (or just left to rot). It's not in puppet, so that requires manual intervention.

the sbg mikrotik host is still present in tor-passwords hosts-extra-info.

so, next steps:

  1. destroying or scrubbing data on the build-arm-* disks
  2. removing torsbg from hoster.yaml
  3. removing sbg from ferm.erb
  4. removing sbg from hosts-extra-info
  5. removing ipsec configuration from other peers (that is *basically* 20-local-peers.conf everywhere)

i'm hesitant in doing the latter 4 steps myself as I am worried i would cut off access to the machine if weasel needed it for the scrubbing or else.

weasel, this ticket yours now, so that you deal with the physical machines themselves. if you want me to scrub the disks myself, i can do so as well, but I figured it would be much easier for you to do that process.

comment:2 Changed 4 weeks ago by anarcat

build-arm-02 and build-arm-03 seemed to have returned in puppet somehow during the night (21:26 UTC) so i cleaned the nodes up again in puppet and they have disappeared again.

weird.

comment:3 Changed 3 weeks ago by weasel

Resolution: fixed
Status: assignedclosed

disks zeroed, hw removed.

comment:4 Changed 3 weeks ago by anarcat

Resolution: fixed
Status: closedreopened

still need to do:

  1. removing torsbg from hoster.yaml
  2. removing sbg from ferm.erb
  3. removing sbg from hosts-extra-info
  4. removing ipsec configuration from other peers (that is *basically* 20-local-peers.conf everywhere)

comment:5 Changed 3 weeks ago by anarcat

Owner: changed from weasel to anarcat
Status: reopenedaccepted

comment:6 Changed 3 weeks ago by anarcat

status update:

  1. [x] removing torsbg from hoster.yaml (done)
  2. [x] removing sbg from ferm.erb (done)
  3. [x] removing sbg from hosts-extra-info (done)
  4. [x] removing ipsec configuration from other peers (that is *basically* 20-local-peers.conf everywhere)
  5. [x] remove sbg gw from nagios
  6. [x] update wiki documentation (ipsec howto)

ie. ipsec still remaining and two more steps todo. the cool thing with this ipsec change is that we'll be able to fully manage the ipsec config even on legacy hosts (that don't use the new ipsec module) everywhere after this.

Last edited 2 weeks ago by anarcat (previous) (diff)

comment:7 Changed 2 weeks ago by anarcat

Resolution: fixed
Status: acceptedclosed

cleared the wiki, ipsec and nagios configuration, now really all done, and yes, ipsec is fully managed in puppet now.

Note: See TracTickets for help on using tickets.