Work out which networks to run in Chutney's CI

Work out which networks to run, and if they should have separate jobs.

We don't need to test all the networks, just the ones that test chutney features. In particular, we don't need a tor version vs chutney network matrix: that should go in the tor branch CI, not the chutney CI.

The default network should test as many of tor/chutney's features as possible. If this network breaks on a particular tor version, we look at that Tor version's CI to see which particular networks have broken.

Then we want a separate job for each separate feature:

• a bridge network from "make test-network-all"
• an onion service v2 and v3 network from "make test-network-all"
• an IPv6 network from "make test-network-all"

We can add other networks as needed:

• when we add new features to chutney, like PTs
• if we break existing features without breaking chutney's CI
• if we break tor's CI without breaking chutney's CI

Trac:
Child Ticket(s): #29761 (moved), #29763 (moved), #30058 (moved), #30059 (moved), #30063 (moved), #30064 (moved), #30065 (moved)

added actualpoints::3 chutney-ci component::core tor/chutney network-team-roadmap-2019-Q1Q2 owner::teor points::1 priority::medium resolution::fixed reviewer::nickm severity::normal sponsor::19 status::closed type::enhancement labels

Here is the list of features from the README: https://gitweb.torproject.org/chutney.git/tree/README#n45

We might also want to cover these feature sets:

• offline (maybe offline should be the default, and we should do one online)
• send extra data / connections / increase timings
• multiple rounds (2 verifies is probably enough for testing)

I don't think we need to cover this feature, or any other minor features:

• bwfile

We can do all these networks on Linux. We also need to do the offline network on macOS.

Here's the reasoning:

• we want to find out if changes to chutney break an important feature
• to get a minimal matrix, we want to check all plausible independent failures
• so we only need to test features on multiple OSes / Tor versions / python versions if a chutney change could break the feature on one matrix alternative, but not the others

For example:

• offline/online work differently enough on macOS and Linux that we need to test them on both
• we'll already test v2 onion services on Tor's CI for the 0.2.9 branch, and v2 and v3 on all other Tor CI branches
• all other features are essentially the same across OSes, Tor versions, and python versions

We'll do this right after #27912 (moved) merges.

Trac:
Parent: #27912 (moved) to N/A

These chutney tickets are on the network team roadmap, or they are required for tickets that are on the network team roadmap.

Trac:
Keywords: N/A deleted, network-team-roadmap-2019-Q1Q2 added

It seems plausible that 0.2.9 could fail independently of other releases, see #29763 (moved).

I have a nice diagram paper diagram for our different CI dimensions. I'll work out how to put the final version in text.

Here's a list of the networks that tor's make test-network-all runs. We need to test all these networks, because breaking them would break Tor's CI.

0.2.9: basic-min bridges-min hs-min single-onion bridges+ipv6-min ipv6-exit-min hs-ipv6 single-onion-ipv6

0.3.4 and later: basic-min bridges-min hs-v2-min hs-v3-min single-onion-v23 bridges+ipv6-min ipv6-exit-min hs-v23-ipv6-md single-onion-ipv6-md

Here's a list of the network sets and arguments I want to test:

No Tor installed (?):

• --dry-run

master only:

• --coverage --debug --all-warnings
• --quiet --no-warnings
• --net-dir mktemp -d
• --data 100000000 (or 10s worth) --start-time 70 --bootstrap-time 70 --stop-time 10 --rounds 2 --connections 2

0.2.9 and master:

• bridges networks
• --offline (Mac & Linux)

0.2.9, 0.3.4, and master:

• onion networks
• IPv6 networks --ipv4 127.0.0.1 --ipv6 :1

This seems like a lot, but we only modify chutney once a week (or less). So we can afford to do exhaustive tests.

I think the latest branch should work.

Please review:

• https://github.com/torproject/chutney/pull/24

I'll squash all the Travis commits before merging.

Also check CI on the branch:

• https://travis-ci.org/teor2345/chutney/builds/520124726 And the previous version of the branch, before tuning:
• https://travis-ci.org/teor2345/chutney/builds/520120778
• https://travis-ci.org/torproject/chutney/builds/520120842

If it's still unstable, my next step is --offline by default (#30183 (moved)).

Trac:
Status: assigned to needs_review

See my pull request: https://github.com/torproject/chutney/pull/25

And also check CI on the branch: https://travis-ci.org/teor2345/chutney/builds/521045036

I think nickm is the best person to review this branch.

Trac:
Reviewer: N/A to nickm

I'm leaving #28170 (closed) and #30067 (moved) as children, because we should be able to do them really quickly (a cut & paste job) once this ticket is reviewed.

Trac:
Actualpoints: N/A to 3

This branch (in PR 25) LGTM. It also looks like a lot of work -- wow!

Please merge whenever you think appropriate.

Trac:
Status: needs_review to merge_ready

Replying to nickm:

This branch (in PR 25) LGTM. It also looks like a lot of work -- wow!

Please merge whenever you think appropriate.

I'd like to block chutney merges until I get back. That also means blocking the chutney job in Tor's CI (#29280 (moved)).

I think that's the wisest course of action: I want to be around to fix any CI failures after we merge.

I'm going to merge #29729 (moved) soon, so we can merge #29024 (moved) and #30459 (moved).

Merged to master as f90f8e4.

Trac:
Status: merge_ready to closed
Resolution: N/A to fixed

closed

changed time estimate to 8h

added 24h of time spent

mentioned in issue #29761 (moved)

mentioned in issue #29763 (moved)