Opened 9 months ago

Closed 6 months ago

#29729 closed enhancement (fixed)

Work out which networks to run in Chutney's CI

Reported by: teor Owned by: teor
Priority: Medium Milestone:
Component: Core Tor/Chutney Version:
Severity: Normal Keywords: chutney-ci, network-team-roadmap-2019-Q1Q2
Cc: teor Actual Points: 3
Parent ID: Points: 1
Reviewer: nickm Sponsor: Sponsor19

Description

Work out which networks to run, and if they should have separate jobs.

We don't need to test all the networks, just the ones that test chutney features. In particular, we don't need a tor version vs chutney network matrix: that should go in the tor branch CI, not the chutney CI.

The default network should test as many of tor/chutney's features as possible. If this network breaks on a particular tor version, we look at that Tor version's CI to see which particular networks have broken.

Then we want a separate job for each separate feature:

  • a bridge network from "make test-network-all"
  • an onion service v2 and v3 network from "make test-network-all"
  • an IPv6 network from "make test-network-all"

We can add other networks as needed:

  • when we add new features to chutney, like PTs
  • if we break existing features without breaking chutney's CI
  • if we break tor's CI without breaking chutney's CI

Child Tickets

TicketTypeStatusOwnerSummary
#29761defectclosedteorTrack chutney CI failures, and tweak the allow failures settings
#29763defectclosedteorFix 0.2.9 failures in chutney CI
#30058defectclosedteorChutney bootstrap-network script uses the wrong network flavour
#30059defectclosedteorUpdate chutney's README
#30063enhancementclosedteorAdd unit tests to chutney, and run them in Travis
#30064defectclosedteorChutney often fails because Tor hasn't bootstrapped yet
#30065enhancementclosedteorAdd shellcheck tests to chutney

Change History (13)

comment:1 Changed 9 months ago by teor

Here is the list of features from the README:
https://gitweb.torproject.org/chutney.git/tree/README#n45

We might also want to cover these feature sets:

  • offline (maybe offline should be the default, and we should do one online)
  • send extra data / connections / increase timings
  • multiple rounds (2 verifies is probably enough for testing)

I don't think we need to cover this feature, or any other minor features:

  • bwfile

comment:2 Changed 9 months ago by teor

We can do all these networks on Linux.
We also need to do the offline network on macOS.

Here's the reasoning:

  • we want to find out if changes to chutney break an important feature
  • to get a minimal matrix, we want to check all plausible independent failures
  • so we only need to test features on multiple OSes / Tor versions / python versions if a chutney change could break the feature on one matrix alternative, but not the others

For example:

  • offline/online work differently enough on macOS and Linux that we need to test them on both
  • we'll already test v2 onion services on Tor's CI for the 0.2.9 branch, and v2 and v3 on all other Tor CI branches
  • all other features are essentially the same across OSes, Tor versions, and python versions

comment:3 Changed 8 months ago by teor

Parent ID: #27912

We'll do this right after #27912 merges.

comment:4 Changed 8 months ago by teor

Keywords: network-team-roadmap-2019-Q1Q2 added

These chutney tickets are on the network team roadmap, or they are required for tickets that are on the network team roadmap.

comment:5 Changed 8 months ago by teor

It seems plausible that 0.2.9 could fail independently of other releases, see #29763.

comment:6 Changed 8 months ago by teor

I have a nice diagram paper diagram for our different CI dimensions. I'll work out how to put the final version in text.

Here's a list of the networks that tor's make test-network-all runs. We need to test all these networks, because breaking them would break Tor's CI.

0.2.9:
basic-min
bridges-min
hs-min
single-onion
bridges+ipv6-min
ipv6-exit-min
hs-ipv6
single-onion-ipv6

0.3.4 and later:
basic-min
bridges-min
hs-v2-min
hs-v3-min
single-onion-v23
bridges+ipv6-min
ipv6-exit-min
hs-v23-ipv6-md
single-onion-ipv6-md

Here's a list of the network sets and arguments I want to test:

No Tor installed (?):

  • --dry-run

master only:

  • --coverage --debug --all-warnings
  • --quiet --no-warnings
  • --net-dir mktemp -d
  • --data 100000000 (or 10s worth) --start-time 70 --bootstrap-time 70 --stop-time 10 --rounds 2 --connections 2

0.2.9 and master:

  • bridges networks
  • --offline (Mac & Linux)

0.2.9, 0.3.4, and master:

  • onion networks
  • IPv6 networks --ipv4 127.0.0.1 --ipv6 :1

This seems like a lot, but we only modify chutney once a week (or less). So we can afford to do exhaustive tests.

comment:7 Changed 7 months ago by teor

Status: assignedneeds_review

I think the latest branch should work.

Please review:

I'll squash all the Travis commits before merging.

Also check CI on the branch:

And the previous version of the branch, before tuning:

If it's still unstable, my next step is --offline by default (#30183).

comment:8 Changed 7 months ago by teor

Reviewer: nickm

See my pull request:
https://github.com/torproject/chutney/pull/25

And also check CI on the branch:
https://travis-ci.org/teor2345/chutney/builds/521045036

I think nickm is the best person to review this branch.

comment:9 Changed 7 months ago by teor

Actual Points: 3

I'm leaving #28170 and #30067 as children, because we should be able to do them really quickly (a cut & paste job) once this ticket is reviewed.

comment:10 Changed 7 months ago by nickm

Status: needs_reviewmerge_ready

This branch (in PR 25) LGTM. It also looks like a lot of work -- wow!

Please merge whenever you think appropriate.

comment:11 in reply to:  10 Changed 7 months ago by teor

Replying to nickm:

This branch (in PR 25) LGTM. It also looks like a lot of work -- wow!

Please merge whenever you think appropriate.

I'd like to block chutney merges until I get back.
That also means blocking the chutney job in Tor's CI (#29280).

I think that's the wisest course of action: I want to be around to fix any CI failures after we merge.

comment:12 Changed 6 months ago by teor

I'm going to merge #29729 soon, so we can merge #29024 and #30459.

comment:13 Changed 6 months ago by teor

Resolution: fixed
Status: merge_readyclosed

Merged to master as f90f8e4.

Note: See TracTickets for help on using tickets.