Opened 7 years ago

Closed 7 years ago

#5806 closed defect (fixed)

Investigate why metrics-db sometimes takes longer than 5 minutes to run

Reported by: karsten Owned by: karsten
Priority: Medium Milestone:
Component: Metrics/CollecTor Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points: 4
Reviewer: Sponsor:

Description

metrics-db runs once every hour and shouldn't take longer than a few minutes. Ideally, we should run it twice per hour to catch consensuses published at :30 of an hour (see #5504). That's only possible if it reliably finishes in under 30 minutes, ideally under 5 minutes.

My current idea why metrics-db takes so long is that it implements a quite crappy algorithm to provide the last 3 days of data via rsync: enumerate all files in the current rsync/ directory and store their names and last modified times in memory, iterate over our various output directories, copy files that are missing in rsync/, and delete the ones in rsync/ that are older than 3 days. The problem is the step where we iterate over output directories, because files are not deleted automatically after a given time. Any change needs to be implemented for relay descriptors, bridge descriptors, bridge pool assignments, exit lists, etc.

This task took 1.5 points so far and may take another 2.5 points for a real fix.

Child Tickets

Change History (1)

comment:1 Changed 7 years ago by karsten

Resolution: fixed
Status: newclosed
Summary: Investigate why metrics-db sometimes takes long than 5 minutes to runInvestigate why metrics-db sometimes takes longer than 5 minutes to run

Fixed by a) processing the five kinds of metrics data in parallel and by b) implementing a more efficient algorithm to copy recent files to the rsync/ directory. Closing.

Note: See TracTickets for help on using tickets.