Opened 6 years ago

Closed 6 years ago

#9183 closed enhancement (fixed)

Avoid parsing server descriptors more than once

Reported by: karsten Owned by: karsten
Priority: High Milestone:
Component: Metrics/Onionoo Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

In the past 48 hours, Onionoo's hourly cronjob have taken between 6 and 61 minutes. The latter number is particularly problematic, because two cronjobs must not overlap, in theory.

An analysis of substeps shows that I/O-heavy steps have highest variance. For example, relay and bridge server descriptors are parsed in three places in the code, which takes between 0:16 and 18:18 minutes, between 0:08 and 22:34 minutes, and between 0:11 and 14:30 minutes.

We can save some time here by avoiding to parse server descriptors more than once. In the mentioned cases, we simply parse all server descriptors published in the last 72 hours. What we should do instead is keep a parse history to only parse those descriptors published in the last hour, and read contents of older descriptors from our own state files.

Start with WeightsDataWriter and then tweak DetailsDataWriter.

Child Tickets

Change History (1)

comment:1 Changed 6 years ago by karsten

Resolution: fixed
Status: newclosed

Implemented and deployed on yatei. Results are quite promising, though there's still room for improvement. Closing, because this particular improvement is done.

Note: See TracTickets for help on using tickets.