Opened 4 weeks ago

Closed 3 weeks ago

#23929 closed defect (fixed)

Don't tweet statistics from Onionoo when Onionoo is down

Reported by: irl Owned by: irl
Priority: Very High Milestone:
Component: Metrics/Bot Version:
Severity: Critical Keywords:
Cc: karsten Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

This has led to an interesting bug that tweets that there are currently zero relays running.

Onionoo is returning "Error 503 Backend fetch failed" currently.

There needs to be some error handling that marks the data source as unusable.

Child Tickets

Change History (12)

comment:1 Changed 4 weeks ago by irl

Priority: MediumVery High
Severity: NormalCritical
Status: newaccepted

comment:2 in reply to:  description Changed 4 weeks ago by karsten

Cc: karsten added

Replying to irl:

Onionoo is returning "Error 503 Backend fetch failed" currently.

Do you have more information on this incident? Does it happen more often? Do you maybe even have logs?

comment:3 Changed 4 weeks ago by cypherpunks

I had the same problem with onionoo and observed the following:
details?limit=4 (the example url) worked
my actual request:
details?limit=0 failed with the above error message.

A few attempts later it worked fine again.

comment:4 Changed 4 weeks ago by karsten

Next time you observe this problem, can you somehow find out which Onionoo host sent you this error message?

comment:5 Changed 4 weeks ago by irl

karsten: My current workaround is die if Onionoo is down and then I restart it when I notice. I plan to have this be more useful.

Is there a way to determine the host that was used? I don't see anything obvious in the Onionoo spec.

comment:6 in reply to:  5 Changed 4 weeks ago by karsten

Replying to irl:

Is there a way to determine the host that was used? I don't see anything obvious in the Onionoo spec.

There's nothing in the response body, no. But can you access the response header and print out the remote host IP address?

Or if you (or somebody else) spot this issue while developing/testing something, maybe run a curl -v on the same machine and see what that prints out?

comment:7 Changed 4 weeks ago by irl

The remote IP address isn't in the Headers afaics.

HTTP/1.1 200 OK
Date: Mon, 23 Oct 2017 15:49:36 GMT
Last-Modified: Mon, 23 Oct 2017 14:52:23 GMT
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=UTF-8
Cache-Control: public, max-age=300
Content-Length: 1045
X-Varnish: 606629251
Via: 1.1 varnish (Varnish/5.0)
X-Varnish: 789232853
Age: 0
Via: 1.1 varnish (Varnish/5.0)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15768000; preload

I can get the headers out when I get the response, but there's nothing to log.

comment:8 Changed 4 weeks ago by irl

I'm not sure I can actually even do this with the HC fluent API. I think I need to use the full HTTP client API instead. This probably will allow me to pull out the IP address after receiving the response, and I need to do this anyway to take advantage of caching.

comment:9 Changed 4 weeks ago by karsten

So, I believe it's related to one of the Onionoo frontends. I sent mail to the admins and will comment here as soon as I know what's going on.

In the meantime, adding host information to the logs somehow might make sense. If not for this time, then for the next time.

comment:10 Changed 4 weeks ago by cypherpunks

it is happening again now:
https://onionoo.torproject.org/details?limit=1


Error 503 Backend fetch failed

Backend fetch failed
Guru Meditation:

XID: 370513704

Varnish cache server

X-Varnish is set to:
370513703

comment:11 Changed 4 weeks ago by cypherpunks

The onionoo part of this is now in #23982

comment:12 Changed 3 weeks ago by irl

Resolution: fixed
Status: acceptedclosed

The critical portion of this, don't tweet things that aren't true, is fixed.

After 3 attempts to update from Onionoo, metrics-bot will send a cry for help to #tor-bots and then quit. This isn't the best solution to this, but the real solution will go in a new ticket.

Note: See TracTickets for help on using tickets.