Graph consensus and vote information from Rob's experiments

Trac:
Parent Ticket: #33327 (moved)

added actualpoints::3 component::metrics/analysis metrics-team-roadmap-2020 owner::metrics-team parent::33327 points::6 priority::medium sbws-roadmap severity::normal status::needs-review type::task labels

Trac:
onionperf-cdf-ttfb-2020-01-28.pdf

Okay, let's continue this discussion on this ticket.

I'm attaching new graphs with CDF-TTFB for all OnionPerfs running during that time.

These graphs are using colors from ColorBrewer that are supposed to be easier to distinguish for colorblind people, but we can still colors that as we move forward.

CDF-DL requires some more processing, and I'm yet not sure how to do the other two. I'll see when I get there. Posting updates as I have them.

Trac:
Keywords: N/A deleted, metrics-team-roadmap-2020Q1 added

Trac:
onionperf-cdf-ttfb-2020-01-28a.pdf

This is a bit embarrassing, but the reason for the 50% bump was that I mixed public and onion server results... Fixed here! That file also contains CDF-DL. Will try to do the other two later tonight.

Trac:
cdf-relay-utilization-2020-01-29.pdf

The other two are about as hard as expected. I just finished a very first version of CDF-Relay-Utilization that I'm attaching here. Expect bugs!

I'll work on CDF-Relay-Stream-Capacity now, but only for consensuses for the moment; votes will have to wait.

Trac:
cdf-relay-utilization-and-stream-capacity-2020-01-29a.pdf

And here's another document with both CDF-Relay-* graphs; without votes. Expect bugs!

This might be a good point for you to provide feedback whether this is roughly going into the right direction. Setting to needs_review for this purpose. I'll pause working on this until I hear back. Thanks!

Trac:
Status: new to needs_review

Trac:
Cc: N/A to teor

Hrmm. CDF-Relay-Capacity should have a X axis range of [0.0, 1.0]. I just realized that there was some incorrect wording in the definition of the metric on https://trac.torproject.org/projects/tor/wiki/org/roadmaps/CoreTor/PerformanceMetrics. It should be average read/write history divided by peak observed bandwidth, for each relay in the network. In other words, you average the read/write history over time for a relay, and divide it by the peak advertised bandwidth over that period of time. This should produce a value between 0 and 1.

There is also a bug in the CDF-Relay-Stream-Capacity, though I am not sure what it is. It should be centered around 1.0, not 0.01. Can you write the formula you used for this? Perhaps you just forgot to include the scale multiplier for the measured bandwidth?

Trac:
Status: needs_review to needs_revision

I'll have to be AFK for the next couple hours, but until then here's a tiny subset of the data I used for the CDF-Relay-* graphs:

fingerprint,validafter,hasexitflag,hasguardflag,read,written,rate,burst,observed,measured
000A10D4,2019-08-13T09:00,f,f,451,286,102400,204800,112448,2
000C1F7C,2019-08-13T09:00,f,t,3578899,3591896,19660800,1073741824,8023040,27200
000CFDEC,2019-08-13T09:00,f,f,1057727,1057703,4194304,6291456,4117504,4360
0011BD24,2019-08-13T09:00,t,t,17419119,17556018,1073741824,1073741824,34104843,44200
001524DD,2019-08-13T09:00,f,t,2065558,2119272,5242880,10485760,5455106,9870
0020D8A2,2019-08-13T09:00,t,t,3686378,3740564,1073741824,1073741824,9030923,11000
00342C0E,2019-08-13T09:00,f,t,1015702,1020588,4194304,5242880,3717941,4500
003BFA1B,2019-08-13T09:00,f,t,6623297,6664212,26214400,1073741824,18836759,29800
003D7882,2019-08-13T09:00,t,f,1280946,1304984,1073741824,1073741824,5083154,3640
0041E015,2019-08-13T09:00,f,f,175940,176294,10485760,10485760,1102303,994
00451DF7,2019-08-13T09:00,f,f,539348,539309,4194304,6291456,2820096,3100
004573FE,2019-08-13T09:00,f,f,15712,15637,819200,1024000,480169,243
0059D929,2019-08-13T09:00,f,f,82583,82628,409600,1048576,450570,309
005ED972,2019-08-13T09:00,f,f,8810311,8828818,1073741824,1073741824,22108491,41000
00723AF1,2019-08-13T09:00,f,t,123404,124200,1073741824,1073741824,21775327,988
00727F3A,2019-08-13T09:00,f,f,699,698,512000,1024000,240510,2
0077BCBA,2019-08-13T09:00,t,t,21706519,22054066,1073741824,1073741824,38740943,63000
0077DDDD,2019-08-13T09:00,f,f,1049,892,13107200,14417920,1969480,3
008E7B70,2019-08-13T09:00,f,t,3060672,3082137,1073741824,1073741824,9123191,9860
0095792C,2019-08-13T09:00,f,f,324405,324452,1048576,1228800,963827,1250
00962D2D,2019-08-13T09:00,f,f,427773,427954,1048576,2097152,1150132,1200
009851DF,2019-08-13T09:00,f,f,553,391,131072,524288,172355,40
00B3C9FB,2019-08-13T09:00,f,f,1673,1096,102400,204800,119949,18
00CCE6A8,2019-08-13T09:00,t,f,355872,375660,1073741824,1073741824,11663560,35
00D2269D,2019-08-13T09:00,f,t,4218721,4238104,19660800,26214400,9658368,14400
00E1649E,2019-08-13T09:00,f,t,8092131,8137499,13107200,13107200,12435058,22100
00E89DDE,2019-08-13T09:00,f,t,1847757,1854884,1073741824,1073741824,6900558,10200
00FBC7DB,2019-08-13T09:00,f,f,83514,19986,307200,614400,970523,172

There's one line per consensus entry, which is where we get the following columns from: fingerprint,validafter,hasexitflag,hasguardflag,measured. The read and write columns come from bandwidth histories contained in extra-info descriptors. rate,burst,observed come from the server descriptor referenced by the consensus entry.

For CDF-Relay-Utilization I used (pseudo code):

read_write := (read + written) / 2;
advertised := min(rate, burst, observed)
plot -> advertised / read_write

For CDF-Relay-Stream-Capacity I used:

plot -> measured / observed

Replying to karsten:

There's one line per consensus entry, which is where we get the following columns from: fingerprint,validafter,hasexitflag,hasguardflag,measured. The read and write columns come from bandwidth histories contained in extra-info descriptors. rate,burst,observed come from the server descriptor referenced by the consensus entry.

For CDF-Relay-Utilization I used (pseudo code):
read_write := (read + written) / 2;
advertised := min(rate, burst, observed)
plot -> advertised / read_write
}}}

Let's do: {{{ read_write := (read + written) / 2; peak := max(rate, burst, observed) plot -> read_write / peak


Ideally, peak would be that peak-in-30-days thing we grinded out in Whistler, but for this we actually wanna see what the instantaneous change to peak that the experiment caused did to results.

The of plot should still be between 0 and 1.0. Any relay that has a value over 1.0 would be very interesting to look at.

> For CDF-Relay-Stream-Capacity I used:
> 
> ```
> plot -> measured / observed
> }}}

Yes, I think this is just off by a factor of 1000 then. It should be:

{{{
plot -> 1000*measured / observed

And then the CDFs will be centered on 1.0.

Trac:
cdf-relay-utilization-and-stream-capacity-2020-01-29b.pdf

I attached a new set of graphs here. They are all cut off at percentile 95, and they all contain the plotted formula in the subtitle.

Regarding the max(rate, burst, observed) part, I'm worried that this number is not very meaningful. In theory, the operator can pick any numbers for rate and burst which the relay can never provide. I plotted one graph with that number, but I don't think we should use that.

The min(rate, burst, observed) number is what we typically use as advertised bandwidth. Maybe it's sufficient to ignore what the operator thought the relay could/should provide and look at observed bandwidth only. I included a plot for this, too.

I recall the peak advertised bandwidth thing we talked about in Whistler. It's significantly harder to compute than the current advertised (or observed) bandwidth, because we need to include lots of descriptors for that. We should pick a formula that we use for all experiments, not just for this one. Maybe we can start with the single value and leave it as a possible extension for the future to consider a moving window of 30 days.

Ok I went and dug through the Tor and TorFlow source. For CDF-Relay-Stream-Capacity, we actually need:

  plot -> 1000*measured/min(rate, observed)

As min(rate,observed) is what TorFlow uses to multiply out the measured bw weights. It does not include the middle "burst" value in this minimum.

For CDF−Relay−Utilization, we def want:

 read_write := (read + written) / 2;
 peak := observed
 plot -> read_write / peak

I recall the peak advertised bandwidth thing we talked about in Whistler. It's significantly harder to compute than the current advertised (or observed) bandwidth, because we need to include lots of descriptors for that. We should pick a formula that we use for all experiments, not just for this one. Maybe we can start with the single value and leave it as a possible extension for the future to consider a moving window of 30 days.

Yeah.. so we really need this 30 day peak of the observed value, as that gets us closer to the true network capacity and utilization. Rob's experiment is useful exactly because it forces relays closer to their peak capacity. Long term, I think metrics should be computing these 30 day maxes continually and providing them as an auxillary csv or other data stream for graphs like these.

For this experiment, it is interesting to see the direct change from 5 day peak observed to Rob's new values, but if I had to pick only one graph, I would still prefer using 30 day peaks.

Trac:
cdf-relay-utilization-and-stream-capacity-2020-01-30.pdf

Here are the new graphs, including the new formulas with peak observed bandwidths and including votes.

I can make smaller changes later today, but larger changes will have to wait until next week.

Graph consensus and vote information from Rob's experiments

Child items ...

Activity