Goal: Decide how much to work/not work on chutney in next 6 months.

Each possibility has costs.

Ideal I-want-a-pony state:
   * new integration tests easy to add in chutney
   * chutney runs on every PR
   * changes to chutney get CI run against all supported tors
   * mixed chutney network runs the whole build matrix against itself after
     each CI session

practical ideas:
   * just turn on chutney-on-travis, do it a bunch of times, try to get it
     passing, put up to X hours of effort into that.  See how far we get in a

   * make verify extensible.

   * have chutney run N times, report number failures, tolerate up to K.
      * Drive K down.

   * Do "See how far we get" option above, see how far we get, and see what
     K has to be, then see if that's acceptable.

   * Can we extract artifacts from travis?
      * Whole net/nodes directory needs to get out somehow so we can tell
        what the failures were.

      * Maybe we can ask travis people? [ACTION ITEM]

   * work on failure-reporting better.

   * Work on better chutney reporting.

   * Make a branch that turns on chutney on travis
   * Run that branch a bunch of times; try to get it to pass every time.
   * Stop after X days. -- a week or so?

   * Find out what kind of error tolerance we would need to enable (K
     failures allowed out of N runs) to have acceptable time and
     false-positive rates.  See whether that is something we can put up with
     for now.


Last modified 21 months ago Last modified on Jan 31, 2019, 11:49:27 AM