Don't tweet statistics from Onionoo when Onionoo is down

This has led to an interesting bug that tweets that there are currently zero relays running.

Onionoo is returning "Error 503 Backend fetch failed" currently.

There needs to be some error handling that marks the data source as unusable.

added component::archived/metrics bot owner::irl priority::very high resolution::fixed severity::critical status::closed type::defect labels

Trac:
Priority: Medium to Very High
Status: new to accepted
Severity: Normal to Critical

Replying to irl:

Onionoo is returning "Error 503 Backend fetch failed" currently.

Do you have more information on this incident? Does it happen more often? Do you maybe even have logs?

Trac:
Cc: N/A to karsten

I had the same problem with onionoo and observed the following: details?limit=4 (the example url) worked my actual request: details?limit=0 failed with the above error message.

A few attempts later it worked fine again.

Next time you observe this problem, can you somehow find out which Onionoo host sent you this error message?

karsten: My current workaround is die if Onionoo is down and then I restart it when I notice. I plan to have this be more useful.

Is there a way to determine the host that was used? I don't see anything obvious in the Onionoo spec.

Replying to irl:

Is there a way to determine the host that was used? I don't see anything obvious in the Onionoo spec.

There's nothing in the response body, no. But can you access the response header and print out the remote host IP address?

Or if you (or somebody else) spot this issue while developing/testing something, maybe run a curl -v on the same machine and see what that prints out?

The remote IP address isn't in the Headers afaics.

HTTP/1.1 200 OK
Date: Mon, 23 Oct 2017 15:49:36 GMT
Last-Modified: Mon, 23 Oct 2017 14:52:23 GMT
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=UTF-8
Cache-Control: public, max-age=300
Content-Length: 1045
X-Varnish: 606629251
Via: 1.1 varnish (Varnish/5.0)
X-Varnish: 789232853
Age: 0
Via: 1.1 varnish (Varnish/5.0)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15768000; preload

I can get the headers out when I get the response, but there's nothing to log.

I'm not sure I can actually even do this with the HC fluent API. I think I need to use the full HTTP client API instead. This probably will allow me to pull out the IP address after receiving the response, and I need to do this anyway to take advantage of caching.

So, I believe it's related to one of the Onionoo frontends. I sent mail to the admins and will comment here as soon as I know what's going on.

In the meantime, adding host information to the logs somehow might make sense. If not for this time, then for the next time.

it is happening again now: https://onionoo.torproject.org/details?limit=1

Error 503 Backend fetch failed

Backend fetch failed
Guru Meditation:

XID: 370513704

Varnish cache server

X-Varnish is set to: 370513703

The onionoo part of this is now in #23982 (moved)

The critical portion of this, don't tweet things that aren't true, is fixed.

After 3 attempts to update from Onionoo, metrics-bot will send a cry for help to #tor-bots and then quit. This isn't the best solution to this, but the real solution will go in a new ticket.

Trac:
Resolution: N/A to fixed
Status: accepted to closed

closed

mentioned in issue #23982 (moved)