Opened 9 years ago

Closed 19 months ago

#1991 closed enhancement (wontfix)

Recognize poor guard performance and switch?

Reported by: arma Owned by:
Priority: Medium Milestone:
Component: Metrics/Analysis Version:
Severity: Keywords: performance roundtrip
Cc: iang, tariq.ee Actual Points:
Parent ID: #2769 Points: 5
Reviewer: Sponsor:

Description

Ticket #1919 showed that the expected performance when using guards is comparable to the expected performance when picking the fastest guards. The penalty from using the slowest guards is especially noticeable in the 1MB and 5MB torperf runs.

What property is it about the guards that lets us guess when we have the slow ones? Can we sum the bandwidth of our top three guards and realize that it's significantly below the average?

Ultimately I expect we'll want the Tor client to be able to recognize that it's chosen a poor set of guards and swap in a faster one so it's not in the bottom, say, 20%.

Child Tickets

Attachments (1)

torperf-guard-bandwidths.png (193.7 KB) - added by karsten 8 years ago.
Tor performance as a function of guard bandwidth

Download all attachments as: .zip

Change History (20)

comment:1 Changed 8 years ago by nickm

Milestone: Tor: post 0.2.2.xTor: unspecified

comment:2 Changed 8 years ago by arma

Component: Tor ClientMetrics
Milestone: Tor: unspecified
Owner: set to karsten

Changing this to a metrics project since our first step is to answer the measurements question.

comment:3 Changed 8 years ago by arma

Now that we have some torperf path data: can we plot the torperf result (y axis) vs guard bandwidth weight in consensus at the time (x axis) and see if half the guards are ugly or just the really slow ones?

comment:4 in reply to:  3 Changed 8 years ago by karsten

Owner: karsten deleted
Status: newassigned

Replying to arma:

Now that we have some torperf path data: can we plot the torperf result (y axis) vs guard bandwidth weight in consensus at the time (x axis) and see if half the guards are ugly or just the really slow ones?

Yes, we should be able to answer this question. Here are the necessary steps to do so:

  • Wait for the consolidate_stats.py script to work again. The version in Mike's branch ticket2618 should work, but waiting for #2672 to be implemented is probably smarter.
  • Extract consensus bandwidths of all relays and write them to a CSV file. It's probably easiest for me to run an SQL query on the metrics database, like SELECT validafter, fingerprint, bandwidth FROM statusentry.
  • Merge the .mergedata file with the consensus bandwidth file to have a single CSV file that contains Torperf completion times and Guard node consensus bandwidths.
  • Make a scatter plot for a single such file, or concatenate multiple files with qualifiers and make a matrix of scatter plots.

Unassigning this ticket, so that people (including me at a later time) can grab it.

comment:5 Changed 8 years ago by karsten

Component: MetricsTorperf
Points: 5

Leaving priority at normal, because this is an interesting question we can answer with our current data, but nothing depends on this ticket immediately. Blocking on #2672.

comment:6 Changed 8 years ago by karsten

Not blocking on #2672 anymore, because we fixed that one.

comment:7 Changed 8 years ago by mikeperry

Parent ID: #2769

comment:8 Changed 8 years ago by mikeperry

According to torperf, the "slow" guards were worse than the "slow ratio" guards, right? We should make sure we're very certain about this fact.

Once we know this, shouldn't we just try to find the new cutoff in the consensus where we stop assigning Guard flags? Unless we're trying to say something like "1 slow guard is OK, but 3 is not"?

Changed 8 years ago by karsten

Tor performance as a function of guard bandwidth

comment:9 Changed 8 years ago by karsten

I attached a graph that compares guard bandwidths to Torperf completion times. Does this graph look plausible? I would have expected a higher influence of guard bandwidth here.

comment:10 Changed 8 years ago by mikeperry

Yeah, it looks kind of all over the place. It seems like the only thing we know for sure right now is that the very bottom of the barrel of guard nodes suck, but after that it's all just noise...

We could have a look at the CBT parameters for the slow guard runs and see how they differ from the fast guard runs. I bet the curves are flatter. That could cue us into poor guards. But it might also be the same for people on unreliable links. I'd expect people on slow, but uniformly reliable links to just have a shifted distribution with a higher Xm. But people with either poor guards or unreliable links will have a low Xm and also a shallow, flat, long curve.

comment:11 Changed 8 years ago by mikeperry

I looked at the CBT extradata.. While there appears to be some minor correlation here between flatter curves and the slower guards, the correlation isn't major enough to be obvious just by looking at the parameters themselves.

I googled around for ways to measure cdf "flatness" and the only metric I got was entropy. We could run the entropy numbers of these cbt curves and see what they look like. Maybe this will tell us something.

comment:12 Changed 8 years ago by mikeperry

Cool. A basic test calculating continuous entropy seems to show that the slow guard runs can get entropy values as high as 6-7 bits, and hang out around 4, where as the fast guards usually hang out around 2.. Looks promising. However, I'm not sure if continuous entropy is what we want. We may want to use the state file and compute actual entropy there.

I think this means I should write a tor patch to add an extra output item to the BUILDTIMEOUT_SET info that calculates the discrete entropy of the build times array.. Then we can look at our set of torperf runs and see if anything pops out.

comment:13 Changed 8 years ago by karsten

Component: TorperfAnalysis

comment:14 Changed 7 years ago by arma

Keywords: performance added

My understanding is that the current status here is that *most* guard selections are fine, and a small fraction of guard selections are really crappy. We should come up with a way to realize that we're in the 'really crappy' situation and repick.

comment:15 Changed 7 years ago by mikeperry

Do we agree that trying to measure the entropy of the circuit build times could be a good way to recognize "really crappy"? The intuition is that the more disperse your build times data is, the more likely you have erratically performing guards (which is a close approximation of really crappy).

If so, should I be thinking about doing the test I suggested in comment 12?

comment:16 Changed 7 years ago by arma

Keywords: firsthop added

comment:17 Changed 7 years ago by arma

Keywords: roundtrip added; firsthop removed

comment:18 Changed 6 years ago by arma

Cc: iang tariq.ee added

Adding Ian and Tariq since this is on-topic for them as well.

It seems clear that we can only make this decision based on public global info (if we do it based on our observed performance from the guard, we open ourselves to attacks where a local adversary makes us switch our guards).

I'm not sure what the right statistic is for deciding that we're 'far enough' down the performance tail. Is it different when the guard set is 2 guards rather than 3? How about 1 guard? Sounds like a generalizable research problem -- and one that would benefit from knowing how actual stats people approach these questions.

comment:19 Changed 19 months ago by karsten

Resolution: wontfix
Status: assignedclosed

Closing tickets in Metrics/Analysis that have been created 5+ years ago and not seen progress recently, except for the ones that "nickm-cares" about.

Note: See TracTickets for help on using tickets.