replace munin with prometheus and grafana

munin died in a fire and people want to try out prometheus, let's do that.

this will also involve setting up a Grafana instance as the built-in graphs in Prometheus are too limited and/or hard to configure.

Trac:
Child Ticket(s): #29410 (moved), #30028 (moved), #29389 (moved), #29682 (moved), #29683 (moved), #29684 (moved), #29822 (moved)

added component::internal services/tor sysadmin team owner::anarcat priority::medium resolution::fixed severity::normal status::closed type::task labels

Trac:
Description: munin died in a fire and people want to try out prometheus, let's do that.

to

munin died in a fire and people want to try out prometheus, let's do that.

this will also involve setting up a Grafana instance as the built-in graphs in Prometheus are too limited and/or hard to configure.

Summary: replace munin with prometheus to replace munin with prometheus and grafana

Trac:
Owner: N/A to tpa
Type: defect to task
Component: - Select a component to Internal Services/Tor Sysadmin Team

good progress here: setup prom-node-exporter on vineale and the temp server, and samples are coming in.

next step is to figure out how to deploy grafana and start working on puppetizing the entire thing.

update: grafana is now deployed, although not completely automatically. the last stage of this is to deploy more "exporters" to cover things that might have been built into munin, like mail and database server monitoring. apache webserver monitoring is already partly implemented.

once that is done and munin is cleaned up, i think we'll be done here.

Trac:
Owner: tpa to anarcat
Status: new to accepted

the apache dashboard has been deployed on all hosts running apache.

i did a quick investigation of the other possible exporters and dashboards that we could use in our infrastructure. notes moved into #30028 (moved).

for what it's worth, I opened a ticket about the other side of Prometheus, alerting, and replacing Nagios, in #29864 (moved)

after approximately two weeks of scraping, we are nearing 11GiB of disk usage, which is below the estimates established in #29388 (moved) (20GiB), which is good.

i moved the dashboard / exporter evaluation stuff into #30028 (moved) instead of blowing up this ticket with even more of those.

i think most of the work here is done. there are still open issues on grafana, namely authentication (#30023 (moved)) and deployment (#30026 (moved)) but I don't think those are a priority in the grand scheme of things right now. there's also the wider question of monitoring of third-party services, for example in #30152 (moved) that i consider out of scope for this original deployment, along with an eventual nagios replacement.

because everything is covered by other tickets, I will make the call and claim the first phase of this project is complete.

TL;DR: done! Next steps:

• #30023 (moved) - grafana authentication
• #30026 (moved) - grafana docker deployment
• #30152 (moved) - third-party monitoring
• #29864 (moved) - nagios replacement

Trac:
Status: accepted to closed
Resolution: N/A to fixed

closed

mentioned in issue #29822 (moved)

mentioned in issue #29864 (moved)

mentioned in issue #30006 (moved)

mentioned in issue #30023 (moved)

mentioned in issue #30026 (moved)

mentioned in issue #30028 (moved)

mentioned in issue #29389 (moved)

mentioned in issue #29410 (moved)

mentioned in issue #29682 (moved)

mentioned in issue #29684 (moved)

mentioned in issue #29683 (moved)