Opened 3 years ago

Closed 2 years ago

#19452 closed enhancement (wontfix)

Make single request for Onionoo details document

Reported by: phw Owned by: irl
Priority: Medium Milestone:
Component: Metrics/Relay Search Version:
Severity: Normal Keywords: onionoo
Cc: irl, karsten Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

This ticket came out of the discussion over at #15395. Quoting Karsten:

Rather than making 1 request for an Onionoo summary document and then n requests for full Onionoo details documents, let's make a single request for 1 Onionoo details document. Exact parameters are subject to further discussion but would include limit=51 (50 results max and 1 to see if there are even more), fields=... (all fields we'd like to display in the results table), search=... (whatever we pass to the summary request right now), and maybe more.

Child Tickets

Attachments (2)

varnish_cache-year.png (22.2 KB) - added by irl 2 years ago.
varnish_cache-day.png (17.5 KB) - added by irl 2 years ago.

Download all attachments as: .zip

Change History (8)

comment:1 Changed 3 years ago by irl

Do we still need a limit if we're only doing one request?

comment:2 Changed 3 years ago by irl

Owner: changed from phw to irl
Status: newaccepted

comment:3 Changed 3 years ago by karsten

Replying to irl:

Do we still need a limit if we're only doing one request?

I'd say yes. Onionoo currently knows about 13k relays and bridges that are either running or have been running in the past week. Displaying them all in a table won't go so well. And putting together such a response requires some effort on the server, too.

In the future, we could easily add an offset parameter to implement pagination. For example, displaying 50 results per page and showing page 2 with results 51 to 100: offset=50&limit=51, where the 51st result would again be to learn if there's a page 3; and oh dear, your next question will be how to find out how many hits there are in total... How about we move that to a separate issue after this one is implemented? :)

comment:4 Changed 2 years ago by karsten

Please disregard my comment above starting with "In the future", which really belongs in another ticket, like #17938. Let's focus on reducing the number of requests from Onionoo internally while keeping functionality unchanged.

Here's an example for requests made by Atlas to Onionoo for displaying "Top 10 Relays by Consensus Weight". Atlas first asks for the first 10 running relays by consensus weight in decreasing order:

/summary?type=relay&order=-consensus_weight&limit=10&running=true

The result of that request is this list of relays:

{"version":"3.1",
"relays_published":"2017-01-09 08:00:00",
"relays":[
{"n":"TorLand1","f":"E1E922A20AF608728824A620BADC6EFC8CB8C2B8","a":["37.130.227.133","[2a02:2498:e001:3c::133]"],"r":true},
{"n":"PrivacyRepublic0001","f":"5CECC5C30ACC4B3DE462792323967087CC53D947","a":["178.32.181.96","37.187.129.166"],"r":true},
{"n":"xshells","f":"E8E71987BCB8C24DBDF1C3BB0BF3B6C76550A108","a":["178.217.187.39","[2001:67c:2044:3226::2]"],"r":true},
{"n":"hviv104","f":"81B75D534F91BFB7C57AB67DA10BCEF622582AE8","a":["192.42.116.16"],"r":true},
{"n":"colosimo","f":"51939625169E2C7E0DC83D38BAE628BDE67E9A22","a":["109.236.90.209"],"r":true},
{"n":"IPredator","f":"BC630CBBB518BE7E9F4E09712AB0269E9DC7D626","a":["197.231.221.211"],"r":true},
{"n":"dopper","f":"D665C959571041972EA8C0DD77559EF5579BA112","a":["192.42.113.102"],"r":true},
{"n":"0x3d001","f":"CFBBA0D858F02E40B1432A65F6D13C9BDFE7A46B","a":["91.121.23.100"],"r":true},
{"n":"Onyx","f":"6DFEB41C04CCE846871338E85DD5ACF5CFB6C1DD","a":["192.42.115.102","[2001:610:510:115:192:42:115:102]"],"r":true},
{"n":"radia2","f":"DE684E6C6B7773B8BE74B4D941E4178988E15E26","a":["91.121.230.212","[2001:41d0:d:22a6::1:4]"],"r":true}
],
"bridges_published":"2017-01-09 06:41:04",
"bridges":[
]}

Now it goes to fetch details documents for all contained fingerprints:

https://onionoo.torproject.org/details?lookup=E1E922A20AF608728824A620BADC6EFC8CB8C2B8
https://onionoo.torproject.org/details?lookup=5CECC5C30ACC4B3DE462792323967087CC53D947
https://onionoo.torproject.org/details?lookup=E8E71987BCB8C24DBDF1C3BB0BF3B6C76550A108
[...]

What Atlas should really do is make one request that doesn't return summary documents but instead details documents, and with just the fields that will be displayed in the results table or used internally otherwise:

/details?type=relay&order=-consensus_weight&limit=10&running=true&fields=nickname,...

I didn't look at the code which fields are used, though that shouldn't be hard to figure out. Also note that "Top 10 Relays by Consensus Weight" is just one type of request and that the search by user-provided search term is probably found somewhere else in the code. Basically, Atlas should not request /summary from Onionoo anymore but instead request /details directly.

comment:5 Changed 2 years ago by irl

Status: acceptednew

Setting back to new, to give someone else a chance at this in case they have time before I do.

Changed 2 years ago by irl

Attachment: varnish_cache-year.png added

Changed 2 years ago by irl

Attachment: varnish_cache-day.png added

comment:6 Changed 2 years ago by irl

Resolution: wontfix
Status: newclosed

Given the high cache hit rate on Onionoo, as illustrated in the graphs above, we've decided to not fix this for now as it may actually cause a decrease in performance.

Note: See TracTickets for help on using tickets.