Opened 4 weeks ago

Last modified 13 days ago

#28603 merge_ready defect

OnionPerf vantage points are not comparable

Reported by: irl Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics/Statistics Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

OnionPerf results look to be comparable over time, but between vantage points there are systematic deltas between the results. The "all" plots show rises and falls where they actually don't exist, it's just that a particular vantage point was offline so the average of the two remaining moves noticeably.

We should work out how to have the aggregate value be resistant to the systematic delta due to the vantage point's location.

Child Tickets

Attachments (4)

torperf.png (143.3 KB) - added by karsten 13 days ago.
torperf-failures.png (127.3 KB) - added by karsten 13 days ago.
onionperf-buildtimes.png (136.5 KB) - added by karsten 13 days ago.
onionperf-latencies.png (138.5 KB) - added by karsten 13 days ago.

Download all attachments as: .zip

Change History (11)

comment:1 Changed 3 weeks ago by karsten

Agreed. This was not so much the case with Torperf measurements, but it is indeed the case with OnionPerf measurements now.

What can we do? It seems that the "all" plots don't make as much sense anymore, so we should take them out. But what should we put on the graph instead?

  • A somewhat randomly chosen measurement, say, op-us, with the option to select another source (except "all" which doesn't exist anymore)?
  • All sources at once but either in separate subplots (like in this graph) or as different colors in the same plot (like in this graph)? We might even combine this with checkboxes to uncheck some of the sources down to a single source.

comment:2 Changed 3 weeks ago by irl

I think that different colors in the same plot, showing the IQR, is probably the way to go. It's going to be very busy if we have multiple graphs.

comment:3 Changed 3 weeks ago by karsten

Just the IQR, or median plus IQR?

comment:4 Changed 3 weeks ago by irl

Ah, I did mean median + IQR, yes.

Changed 13 days ago by karsten

Attachment: torperf.png added

Changed 13 days ago by karsten

Attachment: torperf-failures.png added

Changed 13 days ago by karsten

Attachment: onionperf-buildtimes.png added

Changed 13 days ago by karsten

Attachment: onionperf-latencies.png added

comment:5 Changed 13 days ago by karsten

Status: newneeds_review

This turned out to be a bit more work than expected, because there are four graphs affected by this issue.

Here are possible new graphs for them, which all have in common that they don't need a source parameter anymore:





Any suggestions for further improving these graphs from here?

Please also review the two new commits in my task-28603 branch.

comment:6 Changed 13 days ago by irl

Status: needs_reviewmerge_ready

This looks good to me.

comment:7 Changed 13 days ago by karsten

Great! Pushed and deployed the first commit that announces the change. Leaving this ticket in merge_ready for the next two weeks and merging the second commit then. Thanks!

Note: See TracTickets for help on using tickets.