Opened 6 years ago

Closed 3 years ago

#10697 closed task (wontfix)

Fetch hourly lists from Onionoo with relay status

Reported by: baumanno Owned by: baumanno
Priority: Medium Milestone:
Component: Metrics/Tor Weather Version:
Severity: Keywords: weather-rewrite, hourly
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

From the Tor Weather (Rewrite) wiki page wiki:doc/weather-in-2014:

Once per hour, download a list from Onionoo that contains relays that have
been running in the last week. Check if there are any relays that have
been offline for long enough to notify a subscribed user. Send out emails. 

For now, we will concentrate on this subset:

  • query Onionoo for a list of relays containing the appropriate information
  • keep this query as lightweight as possible
  • considering the current weather-design, work out which relays qualify for a 'NodeDown' event
  • output identifier (this could be the contact data directly from the relay, or, to patch this into current weather sometime, the fingerprint)
  • do this hourly

Email notifications will be left out for this initial phase.

Child Tickets

Change History (10)

comment:1 Changed 6 years ago by karsten

Keywords: weather-rewrite added; weather rewrite removed
Owner: changed from kaner to baumanno
Status: newassigned

Assigning this ticket to you, baumanno, and using the weather-rewrite keyword that I used in the other ticket.

comment:2 Changed 6 years ago by baumanno

Current status of implementation of this script can be viewed at https://github.com/baumanno/tor-weather-rewrite

comment:3 Changed 6 years ago by baumanno

Notes from the last meeting on #tor-dev

  • include If-Modified-Since header in query to Onionoo
  • act apropriately according to 304/200 response
  • work on email scraping/check out parsers from current weather

comment:4 Changed 6 years ago by karsten

Minor tweak to your code: you can avoid composing the URL manually: url = OO_URL + urllib.urlencode(OO_PARAMS). As a result you also avoid the trailing &.

comment:5 Changed 6 years ago by baumanno

Caching-logic has been implemented, based around the response-code returned by the request to Onionoo.
There are two possible cases:

  1. 200 OK : store the (subset of the) details document along with the Last-Modified-header to database; Last-Modified will become If-Modified-Since in any subsequent requests
  2. 304 Not modified : work on the cached document (or do nothing at all)

Please see https://github.com/baumanno/tor-weather-rewrite for code and feel free to comment!

comment:6 Changed 6 years ago by karsten

Please base your changes on my "next" branch and tell me where to pull from, and I'll review and merge.

comment:7 Changed 6 years ago by Falken

Was reviewing the code for the hourly script with the idea that I would tackle the daily one, also was reading up on Karstens t-shirt script and think that all of them (and more) could be helped of the caching mechanism that baumanno has included in the hourly script. I would suggest that we break that code out into a separate module to avoid duplication of code and have all of the weather scripts benifit from faster caching mechanism.

comment:8 Changed 6 years ago by karsten

If you mean a separate module in the Weather source tree, then sure, why not. baumanno, what do you think?

comment:9 Changed 6 years ago by Falken

Yup, thats what I mean. If you all think it's a good idea I would love to implement that if it's ok.

Last edited 6 years ago by Falken (previous) (diff)

comment:10 Changed 3 years ago by karsten

Resolution: wontfix
Status: assignedclosed

Tor Weather has been discontinued as of May 24, 2016: https://lists.torproject.org/pipermail/tor-relays/2016-June/009424.html. Batch-closing all remaining tickets as announced in #19382. A list of these tickets and any other Weather tickets modified after June 26, 2016 will be available here: https://trac.torproject.org/projects/tor/query?changetime=Jun+27%2C+2016..&component=^Metrics%2FTor+Weather

Note: See TracTickets for help on using tickets.