Does using rephist uptime for calculating HSDir produce too much flapping?

This ticket came out of the solution for #2709 (moved).

One of our original goals with the "uptime > 24 hours" constraint for hsdir flag assignment was to make sure that the hsdir flag assignment was quite robust over time: a hidden service needs to store its descriptor in the place that a client will fetch it hours later.

Uptime is quite monotonic -- for a relay that stays up. How monotonic is the rephist-calculated uptime?

added SponsorR component::metrics/analysis parent::13209 priority::medium resolution::fixed severity::normal status::closed tor-auth tor-hs type::task labels

A relay that fails to be available to a majority of dirauths counts as a relay that is not up, even if the relay itself believes otherwise IMO.

Replying to Sebastian:

A relay that fails to be available to a majority of dirauths counts as a relay that is not up, even if the relay itself believes otherwise IMO.

But if it's away for a short time and then back, it can still be useful at serving the hidden service descriptors that had been assigned to it.

I guess one of the next questions to ask is "what time periods are we talking about here?"

That is, how long does a relay have to be up to be a suitable hidden service directory point?

Trac:
Milestone: N/A to Tor: 0.2.3.x-final

Trac:
hsdir-set-instability-2011-partial.gz

hsdir-set-instability intermediate output file for metrics-tasks.git/task-2649

I've attached the 2011 portion of the hsdir-set-instability intermediate output file from my task-2649 analysis to this ticket so other people can view the graph interactively with gnuplot more easily.

Based on a zoomed-in view of 2011-04-25 through 2011-05-21 (when I downloaded the consensus tarball for May), instability in the HSDir set (as a fraction of the size of the set) fluctuates on a 24-hour cycle, and that cycle's amplitude increased significantly around 2011-04-28.

To view the HSDir set instability graph:

• Uncompress the file attached to this ticket and copy it to task-2649/out/hsdir-set-instability.
• Start gnuplot in the task-2649 directory.
• Run load "graph-hsdir-set-instability.gnuplot", then set format x "%Y-%m-%d-%H", then replot.
• Zoom in on a portion of the graph by right-clicking on opposite corners of the area you want to view.

Trac:
hsdir-set-instability.gz

hsdir-set-instability intermediate output file for metrics-tasks.git/task-2649 (full)

Trac:
hsdir-set-absolute-instability-graph.pdf

graph of absolute (unscaled) instability of the HSDir set

See [hsdir-set-absolute-instability-graph.pdf hsdir-set-absolute-instability-graph.pdf] for a graph of the unscaled instability of the HSDir set; see task-2714 ( git://git.torproject.org/rransom/metrics-tasks.git task-2714 ) for sources.

Did what we found out about this let us know anything about why this is happening and what we could do about it?

Trac:
Milestone: Tor: 0.2.3.x-final to Tor: 0.2.4.x-final

Trac:
Keywords: N/A deleted, tor-auth added

Trac:
Component: Tor Directory Authority to Tor

I'm kicking this out of the Tor component until it represents a Tor task. Please toss it back in if somebody gets a Tor task out of this.

Trac:
Component: Tor to Analysis
Milestone: Tor: 0.2.4.x-final to N/A

Trac:
Parent: N/A to #13209 (moved)

David: you've been gaining some intuition about this topic lately from the hs health measurer. Did you get any closer to answering this ticket? Are there specific queries we could make to the consensus documents to answer it?

Trac:
Cc: karsten to karsten, dgoulet

Trac:
Keywords: N/A deleted, SponsorR added

Some basic conclusion out of #13209 (moved) experiment. A client getting the latest consensus at every hour, the churn rate was less than 3% and that was only one single HSDir would change out of the set of 6 in that hour. But it never happened over almost a year of running the HS health that a service descriptor was not found in the set.

However, this is not really representing reality as clients have different consensuses and not only the latest one... See #13209 (moved) for more information on the issues of this experiment.

Sooooo we can keep that thing opened forever or flag it that "OK, HSDir so far aren't moving around much as we observed and we are happy with this". We should try to measure this a bit better with a "v2.0" of #13209 (moved) so we can overtime measure it and act accordingly if shit hit the fan. We already have some good ticket going on about improving stability of HSDir flag and raising the bar as well so those relays stop snooping on .onion (#19162 (moved)).

I'm closing this ticket and we'll re-open a new one if we think that it's becoming an issue at some point.

Trac:
Resolution: N/A to fixed
Reviewer: N/A to N/A
Sponsor: N/A to N/A
Keywords: N/A deleted, tor-hs added
Severity: N/A to Normal
Status: new to closed

closed

mentioned in issue #2715 (moved)

mentioned in issue #2716 (moved)

mentioned in issue tpo/core/tor#2715 (closed)

mentioned in issue tpo/core/tor#2716 (closed)