Opened 3 months ago

Last modified 2 days ago

#29863 merge_ready task

Add disk space monitoring for snowflake infrastructure

Reported by: cohosh Owned by:
Priority: Medium Milestone:
Component: Circumvention/Snowflake Version:
Severity: Normal Keywords: anti-censorship-roadmap, budget_needed
Cc: dcf, arlolra, cohosh, anarcat, gaba Actual Points:
Parent ID: #30152 Points:
Reviewer: Sponsor: Sponsor30-can

Description (last modified by cohosh)

We've run out of disk space at both the snowflake bridge (#26661, #28390) and the broker (#29861), which has caused snowflake to stop working. We've set up rotating and compressed logs but it would be nice to have some disk space monitoring to alert us if/when this happens again

Also, as discussed on IRC, we should eventually move the broker to a TPA machine.

Child Tickets

Change History (30)

comment:1 Changed 3 months ago by dcf

Description: modified (diff)

comment:2 Changed 3 months ago by anarcat

Cc: anarcat added

comment:3 Changed 3 months ago by dcf

On IRC, armadev and anarcat suggest using prometheus, which is being used to monitor torproject machines.

https://help.torproject.org/tsa/howto/prometheus/

If the Snowflake bridge and broker were hosted and managed by Tor, then I gather that we would get disk space monitoring (and other things) automatically through their common puppet deployment. But as they are currently separately managed, anarcat says says we need to

  • apt install prometheus-node-exporter (from stretch-backports please)
  • provide the sysadmins with canonical hostnames

comment:4 Changed 3 months ago by anarcat

  • apt install prometheus-node-exporter (from stretch-backports please)

is actually:

apt install -t stretch-backports prometheus-node-exporter

... assuming that backports is already configured. if it isn't, such a line in /etc/apt/sources.list.d/backports.debian.org.list should suffice:

deb	https://deb.debian.org/debian/	stretch-backports	main contrib non-free

... followed by an apt update, naturally.

I believe the two hostnames to scrape are snowflake-broker.bamsoftware.com and snowflake.bamsoftware.com, correct me if that's wrong. So that second bullet point is probably already done. :)

The plan from TPA side is to hook those as part of a new node job in the scrape_configs in prometheus.yml, from Puppet, in profile::prometheus::server.

comment:5 Changed 3 months ago by anarcat

ah, and keep in mind that prometheus doesn't currently send alerts although it's planned, in the long term, to replace our current alerting system (nagios) with Prometheus, once we're confident it doesn't misbehave and performs well. this will take at least a few weeks.

comment:6 Changed 3 months ago by cohosh

Description: modified (diff)

comment:7 Changed 3 months ago by anarcat

the alerting project, that is the move from Nagios to Prometheus, has officially been documented in #29864

comment:8 Changed 2 months ago by cohosh

This should be set up on this end. I'm adding it only for the broker at the moment, just see how it goes. I added a firewall rule to allow connections from the TPA prometheus machine.

I believe the two hostnames to scrape are snowflake-broker.bamsoftware.com and snowflake.bamsoftware.com, correct me if that's wrong. So that second bullet point is probably already done. :)

Yup, that's correct. Right now it's just set up for snowflake-broker.

comment:9 Changed 2 months ago by cohosh

Okay, taking a step back for bit.

Right now prometheus is running and the output can be accessed by locally visiting localhost:9100/metrics. I'm going to hold off on allowing prometheus1.torproject.org to access it until we are confident that the data exported won't introduce new side channels.

In a discussion with anarcat in IRC, it looks like at the moment there is anonymous access to the prometheus server that shows the graphs, and the authorization for accessing this is very light. We should make sure we're comfortable with this before doing it.

comment:10 Changed 2 months ago by cohosh

Here is a summary of the features available: https://github.com/prometheus/node_exporter

comment:11 in reply to:  9 Changed 2 months ago by dcf

Replying to cohosh:

Right now prometheus is running and the output can be accessed by locally visiting localhost:9100/metrics. I'm going to hold off on allowing prometheus1.torproject.org to access it until we are confident that the data exported won't introduce new side channels.

I did ssh -N -L 9100:127.0.0.1:9100 snowflake-broker and then browsed to http://127.0.0.1:9100/metrics. And yeah, looks like there is some potentially useful (to an attacker) stuff in there:

# HELP apt_upgrades_pending Apt package pending updates by origin.
# HELP node_boot_time_seconds Node boot time, in unixtime.
# HELP node_entropy_available_bits Bits of available entropy.
# HELP node_intr_total Total number of interrupts serviced.
# HELP node_uname_info Labeled system information as provided by the uname system call.
# HELP node_network_transmit_bytes_total Network device statistic transmit_bytes.
# HELP node_sockstat_TCP_inuse Number of TCP sockets in state inuse.

It doesn't look disastrous on its own, but I think you're right to be cautious.

It looks like this information is not encrypted in transit? Maybe we could expose it on an authenticated onion service instead?

comment:12 Changed 2 months ago by anarcat

that's right: endpoints are in cleartext. in general, Prometheus does not offer any form of encryption or authentication, that's delegated to reverse proxies or whatever you want to put in front:

https://prometheus.io/docs/operating/security/#authentication-authorization-and-encryption

so far we've avoided doing this on the exporters because of the sheer complexity of adding stunnel or tor or whatever to all those targets. it would be quite annoying, to say the least, to wrap probes in tor - i don't actually know if we could do that. but i suspect we *might* be able to scrape metrics off HTTPS remotes, i just haven't done that yet.

comment:13 Changed 2 months ago by cohosh

I edited /etc/init.d/prometheus-node-exporter and /etc/default/prometheus-node-exporter to add the following line:

ARGS="--no-collector.arp --no-collector.bcache --no-collector.bonding --no-collector.conntrack --no-collector.cpu --no-collector.edac --no-collector.entropy --no-collector.filefd --no-collector.hwmon --no-collector.infiniband --no-collector.ipvs --no-collector.loadavg --no-collector.mdadm --no-collector.meminfo --no-collector.netclass --no-collector.netdev --no-collector.netstat --no-collector.nfs --no-collector.nfsd --no-collector.sockstat --no-collector.stat --no-collector.textfile --no-collector.timex --no-collector.uname --no-collector.vmstat --no-collector.xfs --no-collector.zfs"

Unfortunately there's no --disable-all option or config file so we have to disable all of the collectors individual except the ones that we want. The collectors listed here aren't quite what we got. The full list is in /etc/default/prometheus-node-exporter.

The above arguments will leave only the diskstats, filesystem, systemd, and time collectors enabled for now:

Apr  9 15:13:03 broker systemd[1]: Started Prometheus exporter for machine metrics.
Apr  9 15:13:03 broker prometheus-node-exporter[5469]: time="2019-04-09T15:13:03Z" level=info msg="Starting node_exporter (version=0.17.0+ds, branch=debian/sid, revision=0.17.0+ds-2~bpo9+1)" source="node_exporter.go:156"
Apr  9 15:13:03 broker prometheus-node-exporter[5469]: time="2019-04-09T15:13:03Z" level=info msg="Build context (go=go1.10.5, user=pkg-go-maintainers@lists.alioth.debian.org, date=20181223-17:51:48)" source="node_exporter.go:157"
Apr  9 15:13:03 broker prometheus-node-exporter[5469]: time="2019-04-09T15:13:03Z" level=info msg="Enabled collectors:" source="node_exporter.go:97"
Apr  9 15:13:03 broker prometheus-node-exporter[5469]: time="2019-04-09T15:13:03Z" level=info msg=" - diskstats" source="node_exporter.go:104"
Apr  9 15:13:03 broker prometheus-node-exporter[5469]: time="2019-04-09T15:13:03Z" level=info msg=" - filesystem" source="node_exporter.go:104"
Apr  9 15:13:03 broker prometheus-node-exporter[5469]: time="2019-04-09T15:13:03Z" level=info msg=" - systemd" source="node_exporter.go:104"
Apr  9 15:13:03 broker prometheus-node-exporter[5469]: time="2019-04-09T15:13:03Z" level=info msg=" - time" source="node_exporter.go:104"
Apr  9 15:13:03 broker prometheus-node-exporter[5469]: time="2019-04-09T15:13:03Z" level=info msg="Listening on :9100" source="node_exporter.go:170"

We might also want to enable collectors related to memory/CPU to make sure we're not straining the system too badly.

I think the biggest ones to stay away from are related to network data (like netstat).

I'll put this in needs_review and if we decide it's merge_ready we can allow the Tor Project prometheus scraper to access port 9100.

comment:14 Changed 2 months ago by cohosh

Status: newneeds_review

comment:15 Changed 2 months ago by dcf

I didn't necessarily mean to imply that we need to turn off a lot of the exported metrics; just that some of them are sensitive enough that we can't just expose port 9100 to everyone. It feels a little weird in this day and age to rely on IP address–based authentication, but personally I can live with it. I'm fine with going ahead with what you've set up, cohosh, and making incremental adjustments in the future if we decide they're necessary.

comment:16 in reply to:  12 Changed 2 months ago by dcf

Replying to anarcat:

it would be quite annoying, to say the least, to wrap probes in tor - i don't actually know if we could do that.

The way this would work is we would give you an onion name and an auth cookie. You put those in HidServAuth in torrc as

HidServAuth xxxxxxxxxxxxxxxx.onion authcookieauthcookie

Then, instead of configuring prometheus to fetch from http://snowflake.bamsoftware.com:9100/, you configure it to fetch from http://xxxxxxxxxxxxxxxx.onion:9100/ with a proxy_url of socks5://127.0.0.1:9050/.

On the server side, we would add HiddenServiceAuthorizeClient to torrc:

HiddenServiceDir /var/lib/tor/prometheus_node_exporter
HiddenServicePort 9100 127.0.0.1:9100
HiddenServiceAuthorizeClient basic prometheus

and then get the auth cookie from /var/lib/tor/prometheus_node_exporter/hostname.

comment:17 Changed 2 months ago by cohosh

Nice! This looks like a really neat solution.

In IRC it also sounded like there was little-to-no authentication on the server that displays these metrics after scraping. Is that the case?

comment:18 Changed 2 months ago by phw

Parent ID: #30152

comment:19 in reply to:  17 Changed 2 months ago by cohosh

Replying to cohosh:

In IRC it also sounded like there was little-to-no authentication on the server that displays these metrics after scraping. Is that the case?

anarcat has opened this ticket: #30023 to deal with authentication on the graphana server. It's also worth noting that snowflake as well as other third-party services TPO decides to monitor will be on the same server.

We can move the discussion on authentication to that ticket.

comment:20 Changed 8 weeks ago by cohosh

Just to summarize some discussion in IRC:

There are some difficulties in adding stronger authentication to access the grafana graphs that include but are not necessarily limited to:

  • needing to set up separate instances for TPA/3rd party resources and also for projects that have different authentication requirements
  • relying on LDAP is complicated (see #30023)
  • we shouldn't really be exporting metrics we aren't comfortable with being public in the first place

So given that, I think we should evaluate whether or not this ticket is ready with the current limited prometheus exports based on the fact that anyone can access the exports or grafana graphs (even though there will be some light access control on both).

comment:21 Changed 8 weeks ago by anarcat

@cohosh do you mind dumping the result of curl localhost:9100/metrics | grep '# HELP' so that people can evaluate what metrics are currently available here and therefore judge the impact of a possible disclosure?

my current opinion (yes, it changes! :)) is that we should not export metrics that are sensitive in the first place. this would alleviate the need for stronger (but also more complex) authentication systems on the monitoring servers which could then stay in the "semi-public" mode they have always been.

the problem is we have a two-dimension matrix of servers right now, based on those parameters:

  • external/internal
  • private/public

.. which would mean four monitoring servers. :p too complicated. let's drop that second part and stick with external/internal as a distinction. how does that sound? if we clear the exporters so that the stuff they generate is safe (a good idea anyways, given they lack authentication themselves, other than IP-level blocking), it seems that the private/public distinction isn't necessary anymore...

comment:22 Changed 8 weeks ago by cohosh

Here it is:

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# HELP go_goroutines Number of goroutines that currently exist.
# HELP go_info Information about the Go environment.
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# HELP go_memstats_frees_total Total number of frees.
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# HELP go_memstats_heap_objects Number of allocated objects.
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# HELP go_memstats_lookups_total Total number of pointer lookups.
# HELP go_memstats_mallocs_total Total number of mallocs.
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# HELP go_threads Number of OS threads created.
# HELP node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built.
# HELP node_filesystem_avail_bytes Filesystem space available to non-root users in bytes.
# HELP node_filesystem_device_error Whether an error occurred while getting statistics for the given device.
# HELP node_filesystem_files Filesystem total file nodes.
# HELP node_filesystem_files_free Filesystem total free file nodes.
# HELP node_filesystem_free_bytes Filesystem free space in bytes.
# HELP node_filesystem_readonly Filesystem read-only status.
# HELP node_filesystem_size_bytes Filesystem size in bytes.
# HELP node_scrape_collector_duration_seconds node_exporter: Duration of a collector scrape.
# HELP node_scrape_collector_success node_exporter: Whether a collector succeeded.
# HELP node_systemd_socket_accepted_connections_total Total number of accepted socket connections
# HELP node_systemd_socket_current_connections Current number of socket connections
# HELP node_systemd_system_running Whether the system is operational (see 'systemctl is-system-running')
# HELP node_systemd_timer_last_trigger_seconds Seconds since epoch of last trigger.
# HELP node_systemd_unit_start_time_seconds Start time of the unit since unix epoch in seconds.
# HELP node_systemd_unit_state Systemd unit
# HELP node_systemd_unit_tasks_current Current number of tasks per Systemd unit
# HELP node_systemd_unit_tasks_max Maximum number of tasks per Systemd unit
# HELP node_systemd_units Summary of systemd unit states
# HELP node_time_seconds System time in seconds since epoch (1970).
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# HELP process_max_fds Maximum number of open file descriptors.
# HELP process_open_fds Number of open file descriptors.
# HELP process_resident_memory_bytes Resident memory size in bytes.
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.

comment:23 in reply to:  21 Changed 8 weeks ago by cohosh

Replying to anarcat:

my current opinion (yes, it changes! :)) is that we should not export metrics that are sensitive in the first place. this would alleviate the need for stronger (but also more complex) authentication systems on the monitoring servers which could then stay in the "semi-public" mode they have always been.

the problem is we have a two-dimension matrix of servers right now, based on those parameters:

  • external/internal
  • private/public

.. which would mean four monitoring servers. :p too complicated. let's drop that second part and stick with external/internal as a distinction. how does that sound? if we clear the exporters so that the stuff they generate is safe (a good idea anyways, given they lack authentication themselves, other than IP-level blocking), it seems that the private/public distinction isn't necessary anymore...

This sounds good to me!

comment:24 in reply to:  13 Changed 8 weeks ago by dcf

Status: needs_reviewmerge_ready

Replying to cohosh:

I edited /etc/init.d/prometheus-node-exporter and /etc/default/prometheus-node-exporter to add the following line:

ARGS="--no-collector.arp --no-collector.bcache --no-collector.bonding --no-collector.conntrack --no-collector.cpu --no-collector.edac --no-collector.entropy --no-collector.filefd --no-collector.hwmon --no-collector.infiniband --no-collector.ipvs --no-collector.loadavg --no-collector.mdadm --no-collector.meminfo --no-collector.netclass --no-collector.netdev --no-collector.netstat --no-collector.nfs --no-collector.nfsd --no-collector.sockstat --no-collector.stat --no-collector.textfile --no-collector.timex --no-collector.uname --no-collector.vmstat --no-collector.xfs --no-collector.zfs"

Okay. The measurements in comment:22 look ok to export, even if they get accidentally disclosed. I'm thinking it's prudent to keep the IP address authentication for port 9100, to mitigate against potential vulnerabilities in prometheus-node-exporter itself.

It's probably better not to modify /etc/init.d/prometheus-node-exporter because that will cause a conflict when upgrading. It looks to me like /etc/default/prometheus-node-exporter is the right place and is sufficient.

comment:25 Changed 7 weeks ago by anarcat

I confirm that /etc/default/prometheus-node-exporter is the right place and sufficient for configuration. I can also confirm there are firewall rules on nodes we configure at TPO, but I believe you'll have to set that up yourself on snowflake, as it's not managed by us.

As I previously mentioned, I believe we'll need to setup a new machine for this. It would be great if someone from the anti-censorship team could get budget approval to get VM from Hetzner like we did for the other prometheus server. I tried to document the requirements for such a machine in #29388 and we ended up picking a CX21 instance (5€/mth, 2vCPU, 4GB RAM, 40GB disk, 20TB traffic) in #29389.

Once budget is approved, I can take it from there and setup the machine, install prometheus/grafana and give you the keys (more or less). :) Specifically, I think it would be fair to give anti-censorship folks admin access to the Grafana instance so you can build your own graphs, create user accounts and so on. It would also be necessary to decide how that authentication be done on the Grafana instance (semi-public with the easy-to-guess password or real accounts just for that instance or LDAP).

How does that sound?

comment:26 in reply to:  25 Changed 7 weeks ago by cohosh

Replying to anarcat:

I confirm that /etc/default/prometheus-node-exporter is the right place and sufficient for configuration. I can also confirm there are firewall rules on nodes we configure at TPO, but I believe you'll have to set that up yourself on snowflake, as it's not managed by us.

Okay, I've reverted /etc/init.d/prometheus-node-exporter and kept our configuration in /etc/default.

As I previously mentioned, I believe we'll need to setup a new machine for this. It would be great if someone from the anti-censorship team could get budget approval to get VM from Hetzner like we did for the other prometheus server. I tried to document the requirements for such a machine in #29388 and we ended up picking a CX21 instance (5€/mth, 2vCPU, 4GB RAM, 40GB disk, 20TB traffic) in #29389.

Once budget is approved, I can take it from there and setup the machine, install prometheus/grafana and give you the keys (more or less). :) Specifically, I think it would be fair to give anti-censorship folks admin access to the Grafana instance so you can build your own graphs, create user accounts and so on. It would also be necessary to decide how that authentication be done on the Grafana instance (semi-public with the easy-to-guess password or real accounts just for that instance or LDAP).

How does that sound?

This sounds great, I'll send an email for budget approval of this.

comment:27 Changed 11 days ago by phw

Sponsor: Sponsor19Sponsor30-can

Moving from Sponsor 19 to Sponsor 30.

comment:28 Changed 3 days ago by gaba

Keywords: anti-censorship-roadmap added

comment:29 Changed 2 days ago by gaba

Cc: gaba added

comment:30 Changed 2 days ago by gaba

Keywords: budget_needed added; snowflake removed
Note: See TracTickets for help on using tickets.