Opened 7 years ago

Closed 7 years ago

#6509 closed enhancement (implemented)

Add fields and parameters to Onionoo to search for disappearing or newly joining relays

Reported by: karsten Owned by:
Priority: Medium Milestone:
Component: Metrics/Onionoo Version:
Severity: Keywords:
Cc: delber@…, mo, gsathya Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We should add the information when a relay first joined the Tor network and when it was last seen online. When combined with search parameters, a community manager person could detect any disappearing or newly joining relays and contact relay operators.

Step 1 is to add two new fields first_seen and last_seen to details files. They would contain the timestamps of the first/last consensus (bridge status) that contained the relay (bridge). Once we have that, we can look for relays that have been inactive for up to 7 days (after which they drop out of Onionoo's data) or for relays that have been first seen in the past 7 days. Due to the current design of Onionoo's data storage, a relay that disappears for over a week and returns will have first_seen set to the consensus when it first reappeared.

Step 2 is to extend Onionoo's database to store relays that are inactive for a longer time, too. We'd have to import all the descriptor archives to get the first_seen field of very old relays right, but that's fine.

Step 3 is to add new parameters last_seen_days and first_seen_days which accept a number or a range. If one passes "-7" to last_seen_days, that would means "last seen in last 7 days" which is the current default. "7" or "7-" could stand for "last seen 7 days ago or longer", "7-14" for "between 7 and 14 days in the past", etc.

Once we have these fields and parameters, we can extend #6329 or Atlas to make use of them.

Child Tickets

Change History (4)

comment:1 Changed 7 years ago by delber

Cc: delber@… added

comment:2 Changed 7 years ago by karsten

Cc: mo gsathya added

Step 1 is implemented and deployed since December 5. Please keep in mind that the first_seen field may be wrong if a relay drops out of the consensus for more than a week.

Steps 2 and 3 are blocking on pyonionoo, because we shouldn't make too many changes to Onionoo while pyonionoo is trying to catch up.

comment:3 Changed 7 years ago by karsten

I Implemented steps 2 and 3, but would appreciate some feedback before committing, because I don't want to change the interface more often than necessary. I just deployed step 3, but step 2 is still chewing on a few gigs of data.

Here are some example URLs to test searching for disappearing or newly joining relays. Limits should be limited to 0-7 days for now, or results may be quite inaccurate (that's what step 2 is going to fix tomorrow or the day after).

I admit the syntax could be somewhat more intuitive, but I chose it because it's quite expressive and can be applied to similar parameters in the future. Obviously, this interface is not made for humans, but for a human-friendly interface like Atlas or Compass. Here's the spec:

  • first_seen_days: Return only relays or bridges which have first been seen during the given range of days ago. A parameter value "x-y" with x >= y returns relays or bridges that have first been seen at least x and at most y days ago. Accepted short forms are "x", "x-", and "-y" which are interpreted as "x-x", "x-infinity", and "0-y".
  • last_seen_days: Return only relays or bridges which have last been seen during the given range of days ago. A parameter value "x-y" with x >= y returns relays or bridges that have last been seen at least x and at most y days ago. Accepted short forms are "x", "x-", and "-y" which are interpreted as "x-x", "x-infinity", and "0-y". Note that relays and bridges that haven't been running in the past week are never included in results, so that setting x to 8 or higher will always lead to an empty result set.

Comments? Suggestions? I'm planning to merge this tomorrow.

comment:4 Changed 7 years ago by karsten

Resolution: implemented
Status: newclosed

mo says the two new parameters work for him. Merged and deployed. Closing.

Note: See TracTickets for help on using tickets.