Opened 5 weeks ago

Closed 4 weeks ago

#29761 closed defect (fixed)

Track chutney CI failures, and tweak the allow failures settings

Reported by: teor Owned by: teor
Priority: Medium Milestone:
Component: Core Tor/Chutney Version:
Severity: Normal Keywords: chutney-ci, network-team-roadmap-2019-Q1Q2
Cc: teor Actual Points:
Parent ID: #29729 Points: 1
Reviewer: Sponsor: Sponsor19-must

Description

CI failed one job on 0.2.9, but the debug run succeeded:
https://travis-ci.org/torproject/chutney/jobs/505547963

We might need to increase the allowed failures for 0.2.9, or put a sleep in between the allow failures tests.

But we won't know which one to do until we see how CI works for a few days.

Child Tickets

Change History (9)

comment:1 Changed 5 weeks ago by teor

Here's another failure on 0.2.9, but the debug job succeeds:
https://travis-ci.org/torproject/chutney/jobs/505552407

Maybe we should add a sleep between the allow failures tests, until 0.2.9 is unsupported.

comment:2 Changed 5 weeks ago by teor

The test sometimes succeeds, so 0.2.9 isn't reliably broken:
https://travis-ci.org/torproject/chutney/jobs/505552915

comment:3 Changed 5 weeks ago by teor

(We also had a Travis network failure in another test, so we should tolerate some level of failure.)

comment:4 Changed 5 weeks ago by teor

Owner: set to teor
Status: newassigned

comment:5 Changed 5 weeks ago by teor

Same 0.2.9 failures:
https://travis-ci.org/torproject/chutney/jobs/505567525
https://travis-ci.org/teor2345/chutney/jobs/505046093

0.3.4 failure, before test-network-forgiving:
https://travis-ci.org/teor2345/chutney/jobs/505554142

Travis network failures:
https://travis-ci.org/teor2345/chutney/jobs/505554152
https://travis-ci.org/teor2345/chutney/jobs/505562574

There have been about 13 builds, times 14 jobs per build.

0.2.9 failed on 4/13 jobs, failing 4 builds.
0.3.4 failed on 1/13 jobs, failing 1 build.
Travis network failed on 2/182 jobs, failing 2 builds.

Let's get the chutney failure rate below the Travis failure rate?

Here's what I'm thinking of doing:

  • add a sleep between the allow failure rounds, to hopefully decorrelate the 0.2.9 failures
  • continue to monitor the Travis and 0.3.4 failure rate

comment:6 Changed 5 weeks ago by teor

Keywords: network-team-roadmap-2019-Q1Q2 added

These chutney tickets are on the network team roadmap, or they are required for tickets that are on the network team roadmap.

comment:7 Changed 4 weeks ago by teor

I haven't seen any of these failures since we implemented #22132. Let's continue to monitor and close this bug if we don't see any in the next week?

comment:8 Changed 4 weeks ago by teor

Parent ID: #29729

We might need this for #29729.

comment:9 Changed 4 weeks ago by teor

Resolution: fixed
Status: assignedclosed

This issue seems to be fixed by the bootstrap changes, let's open specific bugs for failures.

Note: See TracTickets for help on using tickets.