backend: hourly update without crontab or the like

Since I didn't want to bother with crontab, I've been using the attached patch for a while.

If no property is given the backend update will run one time. When -Donionoo.cron.runonce=false is given the updater will run after waiting a minute and then every hour after a successful update run. (Thus, it might also be a solution for #13003 (moved).)

Please review.

changed milestone to %Onionoo 3.1-1.0.0

added component::metrics/onionoo milestone::Onionoo 3.1-1.0.0 priority::medium resolution::fixed severity::normal status::closed type::enhancement labels

Trac:
0001-Run-updater-every-hour-depending-on-property-onionoo.patch

Trac:
Status: new to needs_review

That patch starts the first execution 1 minute after starting and then waits for 60 minutes between termination of one execution to start of the next.

Can we try to get closer to the current crontab behavior? That would be start at 15 minutes after the hour, unless an earlier execution is still running. I think scheduleAtFixedRate does that, right? And I think we don't need the lock file anymore if we go that route, correct?

Trac:
Status: needs_review to needs_revision

Why should the crontab behavior be mimicked so closely?

Fixed rate scheduling would change the repeated update times also, if a run takes longer than the set interval. Depending on the configuration this might be another interval or an immediate restart.

The successive update-start depends only on the start time of the java process. Started at 00:14 it would begin at 00:15. But I do not mind setting it to zero.

Especially, when there are several mirrors it might be advisable not to do the downloads at the same time. The runtime of each run would spread the downloading over time.

Should the update interval be shorter, in order to have an update at about every hour?

The lock file will still be needed for the single run using crontab for scheduling the executions. So, it should only show up there, I think.

The idea to run around :15 of the hour is that we can be quite confident that new data will be available at that time. The consensus gets published at :00, and CollecTor starts fetching the consensus and other descriptors at :05. That process usually takes a few minutes. If we have Onionoo download new data from CollecTor at :15 or :20, it's going to be new data, and the earlier we process new data the fresher Onionoo's data will be.

You have a point about not having all mirrors start at the exact same time. What we could do is start between :15 and :20, depending on when the program starts. For example, if the program starts at :01, we start the first run at :16 and the next run 60 minutes later.

Note that we also must avoid running more than once per hour. There's a bug where if we don't parse a fresh consensus, the exit/guard/middle probabilities cannot be calculated correctly. If we plan to run more often, we'll have to fix that bug first. I could imagine running every five minutes and stopping a run if we figure out that CollecTor has no new data for us.

Thoughts? Thanks for starting this!

Thanks for the explanation!

I attached a patch;

• that will start the updater according to the 'onionoo.cron.runonce' property as above (default is the single run).
• the update will be started between (incl.) :15 and :19 depending on start time
• then it will run at a fixed rate of 60 minutes.

Is there a track-task for the incorrect probability calculation bug?

Trac:
0001-task-14826-depending-on-the-starting-time-the-update.patch

Trac:
Status: needs_revision to needs_review

Please take a look at my branch task-14826 which contains your patch plus whitespace fixes. If you still like it, I'll merge that to master.

I'll look if there's already a ticket for the mentioned bug. If there's one, I'll cc you, and if not, I'll create one.

The branch is fine; good to remove the redundant %60-operation.

Okay, great. Merged to master. Closing. Thanks!

Trac:
Resolution: N/A to fixed
Status: needs_review to closed

Trac:
Milestone: N/A to Onionoo 3.1-1.0.0
Severity: N/A to Normal
Reviewer: N/A to N/A
Sponsor: N/A to N/A

closed

mentioned in issue #19018 (moved)

mentioned in issue #19265 (moved)