Opened 7 months ago

Closed 7 months ago

#32638 closed task (fixed)

Please remove onionoo-backend-01 from the backend pool

Reported by: irl Owned by: irl
Priority: High Milestone:
Component: Metrics/Onionoo Version:
Severity: Normal Keywords:
Cc: metrics-team Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We've noticed that the data here has become corrupted from a bad update. We plan to restore it next week.

Child Tickets

Change History (3)

comment:1 Changed 7 months ago by anarcat

Owner: changed from tpa to anarcat
Status: newaccepted

comment:2 Changed 7 months ago by anarcat

Component: Internal Services/Tor Sysadmin TeamMetrics/Onionoo
Owner: changed from anarcat to irl
Status: acceptedassigned

first off, thank you so much for the excellent documentation you wrote on this service. it makes my life (and i suspect everyone else who will have to deal with this in the future) so much easier!

i think we're in the Single backend data corruption, no hardware failure scenario, which says this:

sudo -u onionoo -i bash -c 'systemctl --user stop onionoo'
sudo -u onionoo-unpriv -i bash -c 'systemctl --user stop onionoo-web'
rm -rf /srv/onionoo.torproject.org/onionoo/home/{.,}\*
rm -rf /srv/onionoo.torproject.org/onionoo/home-unpriv/{.,}\*
rm -rf /srv/onionoo.torproject.org/onionoo/onionoo/{.,}\*

Then pretend you are deploying a new backend from the instructions above.

It seems the first two step, at least, haven't been ran, because (a) the services are still running on that backend and (b) the varnish frontend agrees that they are still running:

root@onionoo-frontend-01:~# varnishadm -S /etc/varnish/secret -Tlocalhost:6082 backend.list -p
Backend name                   Admin   Probe      Last change
boot.be_omeiense               probe   6/6 good   Tue, 19 Nov 2019 14:21:35 GMT
Current states  good:  6 threshold:  2 window:  6
  Average response time of good probes: 0.011608
  Oldest ================================================== Newest
  4444444444444444444444444444444444444444444444444444444444444444 Good IPv4
  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit
  RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv
  HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy

boot.be_onionoo_backend_01     probe   6/6 good   Thu, 28 Nov 2019 10:09:23 GMT
Current states  good:  6 threshold:  2 window:  6
  Average response time of good probes: 0.002903
  Oldest ================================================== Newest
  4444444444444444444444444444444444444444444444444444444444444444 Good IPv4
  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit
  RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv
  HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy

boot.be_oo_hetzner_03          probe   6/6 good   Tue, 19 Nov 2019 14:04:28 GMT
Current states  good:  6 threshold:  2 window:  6
  Average response time of good probes: 0.011816
  Oldest ================================================== Newest
  4444444444444444444444444444444444444444444444444444444444444444 Good IPv4
  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit
  RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv
  HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy

boot.mybackends                probe   healthy    Tue, 19 Nov 2019 12:40:16 GMT

Varnish is smart: if a backend goes down, it will stop serving from there. So you just need to shutdown the service in the backend, and it will stop being used in the frontend.

At least, if I understand correctly what you want done here. :) I can also just destroy the box or shut it down if that's better for you. Or I can turn off the service myself, or i can remove it from the load balancer, but I think the latter can be done by the onionoo service operators. That might be better since you wouldn't need TSA to wake up to bring it back online...

In short, I think you can do this yourself. :) Let me know if/how that is incorrect!

comment:3 Changed 7 months ago by irl

Cc: metrics-team added
Resolution: fixed
Status: assignedclosed

Correct, we were indeed able to do this ourselves.

Note: See TracTickets for help on using tickets.