Opened 6 weeks ago

Last modified 4 days ago

#29729 needs_review enhancement

Work out which networks to run in Chutney's CI

Reported by: teor Owned by: teor
Priority: Medium Milestone:
Component: Core Tor/Chutney Version:
Severity: Normal Keywords: chutney-ci, network-team-roadmap-2019-Q1Q2
Cc: teor Actual Points: 3
Parent ID: Points: 1
Reviewer: nickm Sponsor: Sponsor19


Work out which networks to run, and if they should have separate jobs.

We don't need to test all the networks, just the ones that test chutney features. In particular, we don't need a tor version vs chutney network matrix: that should go in the tor branch CI, not the chutney CI.

The default network should test as many of tor/chutney's features as possible. If this network breaks on a particular tor version, we look at that Tor version's CI to see which particular networks have broken.

Then we want a separate job for each separate feature:

  • a bridge network from "make test-network-all"
  • an onion service v2 and v3 network from "make test-network-all"
  • an IPv6 network from "make test-network-all"

We can add other networks as needed:

  • when we add new features to chutney, like PTs
  • if we break existing features without breaking chutney's CI
  • if we break tor's CI without breaking chutney's CI

Child Tickets

#28170enhancementassignedTest stem pull requests against all supported tor versions, and tor nightly builds
#29761defectclosedteorTrack chutney CI failures, and tweak the allow failures settings
#29763defectclosedteorFix 0.2.9 failures in chutney CI
#30058defectclosedteorChutney bootstrap-network script uses the wrong network flavour
#30059defectclosedteorUpdate chutney's README
#30060defectnewWork out how (and where) to test chutney's coverage of tor
#30063enhancementclosedteorAdd unit tests to chutney, and run them in Travis
#30064defectclosedteorChutney often fails because Tor hasn't bootstrapped yet
#30065enhancementclosedteorAdd shellcheck tests to chutney
#30067defectnewUpdate sbws' travis config, based on chutney's travis config

Change History (9)

comment:1 Changed 6 weeks ago by teor

Here is the list of features from the README:

We might also want to cover these feature sets:

  • offline (maybe offline should be the default, and we should do one online)
  • send extra data / connections / increase timings
  • multiple rounds (2 verifies is probably enough for testing)

I don't think we need to cover this feature, or any other minor features:

  • bwfile

comment:2 Changed 6 weeks ago by teor

We can do all these networks on Linux.
We also need to do the offline network on macOS.

Here's the reasoning:

  • we want to find out if changes to chutney break an important feature
  • to get a minimal matrix, we want to check all plausible independent failures
  • so we only need to test features on multiple OSes / Tor versions / python versions if a chutney change could break the feature on one matrix alternative, but not the others

For example:

  • offline/online work differently enough on macOS and Linux that we need to test them on both
  • we'll already test v2 onion services on Tor's CI for the 0.2.9 branch, and v2 and v3 on all other Tor CI branches
  • all other features are essentially the same across OSes, Tor versions, and python versions

comment:3 Changed 6 weeks ago by teor

Parent ID: #27912

We'll do this right after #27912 merges.

comment:4 Changed 6 weeks ago by teor

Keywords: network-team-roadmap-2019-Q1Q2 added

These chutney tickets are on the network team roadmap, or they are required for tickets that are on the network team roadmap.

comment:5 Changed 6 weeks ago by teor

It seems plausible that 0.2.9 could fail independently of other releases, see #29763.

comment:6 Changed 2 weeks ago by teor

I have a nice diagram paper diagram for our different CI dimensions. I'll work out how to put the final version in text.

Here's a list of the networks that tor's make test-network-all runs. We need to test all these networks, because breaking them would break Tor's CI.


0.3.4 and later:

Here's a list of the network sets and arguments I want to test:

No Tor installed (?):

  • --dry-run

master only:

  • --coverage --debug --all-warnings
  • --quiet --no-warnings
  • --net-dir mktemp -d
  • --data 100000000 (or 10s worth) --start-time 70 --bootstrap-time 70 --stop-time 10 --rounds 2 --connections 2

0.2.9 and master:

  • bridges networks
  • --offline (Mac & Linux)

0.2.9, 0.3.4, and master:

  • onion networks
  • IPv6 networks --ipv4 --ipv6 :1

This seems like a lot, but we only modify chutney once a week (or less). So we can afford to do exhaustive tests.

comment:7 Changed 6 days ago by teor

Status: assignedneeds_review

I think the latest branch should work.

Please review:

I'll squash all the Travis commits before merging.

Also check CI on the branch:

And the previous version of the branch, before tuning:

If it's still unstable, my next step is --offline by default (#30183).

comment:8 Changed 4 days ago by teor

Reviewer: nickm

See my pull request:

And also check CI on the branch:

I think nickm is the best person to review this branch.

comment:9 Changed 4 days ago by teor

Actual Points: 3

I'm leaving #28170 and #30067 as children, because we should be able to do them really quickly (a cut & paste job) once this ticket is reviewed.

Note: See TracTickets for help on using tickets.