Opened 9 months ago

Closed 8 months ago

#28603 closed defect (fixed)

OnionPerf vantage points are not comparable

Reported by: irl Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics/Statistics Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

OnionPerf results look to be comparable over time, but between vantage points there are systematic deltas between the results. The "all" plots show rises and falls where they actually don't exist, it's just that a particular vantage point was offline so the average of the two remaining moves noticeably.

We should work out how to have the aggregate value be resistant to the systematic delta due to the vantage point's location.

Child Tickets

Attachments (4)

torperf.png (143.3 KB) - added by karsten 9 months ago.
torperf-failures.png (127.3 KB) - added by karsten 9 months ago.
onionperf-buildtimes.png (136.5 KB) - added by karsten 9 months ago.
onionperf-latencies.png (138.5 KB) - added by karsten 9 months ago.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 9 months ago by karsten

Agreed. This was not so much the case with Torperf measurements, but it is indeed the case with OnionPerf measurements now.

What can we do? It seems that the "all" plots don't make as much sense anymore, so we should take them out. But what should we put on the graph instead?

  • A somewhat randomly chosen measurement, say, op-us, with the option to select another source (except "all" which doesn't exist anymore)?
  • All sources at once but either in separate subplots (like in this graph) or as different colors in the same plot (like in this graph)? We might even combine this with checkboxes to uncheck some of the sources down to a single source.

comment:2 Changed 9 months ago by irl

I think that different colors in the same plot, showing the IQR, is probably the way to go. It's going to be very busy if we have multiple graphs.

comment:3 Changed 9 months ago by karsten

Just the IQR, or median plus IQR?

comment:4 Changed 9 months ago by irl

Ah, I did mean median + IQR, yes.

Changed 9 months ago by karsten

Attachment: torperf.png added

Changed 9 months ago by karsten

Attachment: torperf-failures.png added

Changed 9 months ago by karsten

Attachment: onionperf-buildtimes.png added

Changed 9 months ago by karsten

Attachment: onionperf-latencies.png added

comment:5 Changed 9 months ago by karsten

Status: newneeds_review

This turned out to be a bit more work than expected, because there are four graphs affected by this issue.

Here are possible new graphs for them, which all have in common that they don't need a source parameter anymore:





Any suggestions for further improving these graphs from here?

Please also review the two new commits in my task-28603 branch.

comment:6 Changed 9 months ago by irl

Status: needs_reviewmerge_ready

This looks good to me.

comment:7 Changed 9 months ago by karsten

Great! Pushed and deployed the first commit that announces the change. Leaving this ticket in merge_ready for the next two weeks and merging the second commit then. Thanks!

comment:8 Changed 8 months ago by karsten

Resolution: fixed
Status: merge_readyclosed

Merged with minor tweaks and deployed. Closing. Thanks!

Note: See TracTickets for help on using tickets.