Opened 7 years ago

Closed 19 months ago

#5398 closed task (wontfix)

Compare variance between simulation runs (Shadow and Experimentor)

Reported by: arma Owned by:
Priority: Medium Milestone:
Component: Metrics/Analysis Version:
Severity: Keywords: performance simulation
Cc: robgjansen, kevin, karsten Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

In #4086 we noticed that a given simulation can have quite a bit of variance depending on the topology selected, and what paths clients pick.

How much exactly, and what can we do to quantify it and to reduce it?

The suggestion so far is to do k runs, and then either add them all up as data points that go into the CDF, or look at the CDFs separately and see how different they are from one run to the next.

Child Tickets

Change History (4)

comment:1 Changed 6 years ago by arma

Keywords: performance simulation added

comment:2 Changed 6 years ago by arma

Cc: karsten added

comment:3 Changed 6 years ago by robgjansen

We can run a kstest on the download time distributions to get a distance value between the distributions. See here:
https://github.com/shadow/shadow/issues/156

comment:4 Changed 19 months ago by karsten

Resolution: wontfix
Status: newclosed

Closing tickets in Metrics/Analysis that have been created 5+ years ago and not seen progress recently, except for the ones that "nickm-cares" about.

Note: See TracTickets for help on using tickets.