Use Chutney in Tor's CI

Get Chutney working for continuous integration. This will also help with Tor in general and so has broader benefits than just CI for pluggable transports.

changed milestone to %Tor: 0.2.9.x-final

Trac:
Parent Ticket: #29267 (moved)
Child Ticket(s): #30212 (moved), #29703 (moved), #30859 (moved), #28013 (moved), #29748 (moved)

added 029-backport 035-backport 040-backport 041-deferred-20190530 CI PTs actualpoints::2 component::core tor/tor dgoulet-merge milestone::Tor: 0.2.9.x-final network-team-roadmap-2019-Q1Q2 owner::teor parent::29267 points::1 priority::high resolution::fixed reviewer-was-teor-20190422 reviewer::nickm severity::normal sponsor::19 status::closed tor-ci type::task labels

Trac:
Component: Obfuscation to Core Tor/Tor
Milestone: N/A to Tor: 0.4.1.x-final

Please see the branches below. They add a new make target, "make test-network-forgiving" that tries to run test-network.sh a few times, and succeeds if it succeeds once. They add a builder to do this on travis.

• chutney_ci_029, PR at https://github.com/torproject/tor/pull/734
• chutney_ci_034, PR at https://github.com/torproject/tor/pull/735 (fixes conflicts)
• chutney_ci_035, PR at https://github.com/torproject/tor/pull/736 (fixes conflicts)
• chutney_ci_040, PR at https://github.com/torproject/tor/pull/737 (no conflicts to fix; CI only)
• chutney_ci_041, PR at https://github.com/torproject/tor/pull/738 (no conflicts to fix; CI only)

All of them passed on my first attempt when I pushed them to github myself, so let's see whether CI likes them a second time.

Trac:
Keywords: PTs deleted, PTs 029-backport 034-backport 035-backport 040-backport added
Actualpoints: N/A to 1
Status: new to needs_review

Trac:
Reviewer: N/A to teor

Trac:
Keywords: N/A deleted, network-team-roadmap-2019-Q1Q2 added

I (accidentally) did a review on the 0.3.5 pull request: https://github.com/torproject/tor/pull/736

In general, the code looks good. I have one design question, and one code question.

Trac:
Status: needs_review to needs_revision

I am not sure if we need to fix #29689 (moved) as part of this ticket.

It seems like it might be annoying if someone uses tor/src/test/test-network.sh from an unusual path, or with an unusual directory structure.

I think we can do #29689 (moved) separately, as it's logically independent of this?

Hi! Answered your questions on the tickets; please let me know which way you think I should proceed.

Trac:
Status: needs_revision to needs_review

Replying to nickm:

I think we can do #29689 (moved) separately, as it's logically independent of this?

I think you're right, let's do it some other time.

I'll put it on my fast-fix backlog.

Replying to nickm:

Hi! Answered your questions on the tickets; please let me know which way you think I should proceed.

I think we should add new chutney-related code to the chutney repository: that's been our policy since 0.2.9. And in this particular case, we will need test-network-forgiving.sh to have useful CI for the chutney repository.

There's a longer explanation on the pull request.

I opened #29703 (moved) to backport some test-network.sh fixes from 0.3.0 to 0.2.9 so that we can reliably call chutney's test-network.sh from tor's test-network.sh.

It's also worth being aware of #29060 (moved): I think one of its commits may break tor's chutney CI on master.

Trac:
Status: needs_review to needs_revision

Replying to teor:

I opened #29703 (moved) to backport some test-network.sh fixes from 0.3.0 to 0.2.9 so that we can reliably call chutney's test-network.sh from tor's test-network.sh.

Hmm, there's another option if you don't want to backport:

1. put test-network-forgiving.sh in 0.2.9, and call it from the 0.2.9 Makefile
2. delete test-network-forgiving.sh in 0.3.4 and later
3. implement a test-network-forgiving mode in chutney's test-network.sh, and call it from 0.3.4 and later Makefiles

This seems like the worst of both worlds: we'll be making changes in 0.2.9 and chutney if there are any bugs. Historically, we're not good at keeping tor and chutney in sync.

It's also worth being aware of #29060 (moved): I think one of its commits may break tor's chutney CI on master.

I've asked for a revert on that ticket.

nickm, #29703 (moved) is ready for you to rebase onto.

I've made revised branches:

• chutney_ci_v2_029 -- primary branch to review; based on #29703 (moved). https://github.com/torproject/tor/pull/828
• chutney_ci_v2_034 -- resolves merge conflict. https://github.com/torproject/tor/pull/829
• chutney_ci_v2_035 -- resolves merge conflict. https://github.com/torproject/tor/pull/830
• chutney_ci_v2_040 -- no merge conflict. https://github.com/torproject/tor/pull/831
• chutney_ci_v2_041 -- no merge conflict. https://github.com/torproject/tor/pull/832

Trac:
Status: needs_revision to needs_review

Trac:
Summary: Use Chutney for CI to Use Chutney in Tor's CI

I've just finished #29729 (moved), and discovered a lot of interesting things about chutney, tor, and Travis in the process.

I'll have some feedback on these pull requests tomorrow (~18 hours).

From a commit message:

Previously we had "make check" launched whenever DISTCHECK was false. Now we'd like to turn it off in a few other circumstances, like running chutney. Maybe stem too?

Yes, let's turn off "make check" for stem.

The default network in chutney is bridges+hs-v2, but I discovered in #29729 (moved) that it is really unstable in chutney's CI. (Sometimes all the jobs work, and sometimes multiple jobs fail. I think it might be due to network or CPU load issues on machines.)

So let's change to a subset of the CI tests that work in chutney?

So for 0.2.9, let's use:

• bridges-min
• hs-v2-min And for 0.3.4 and later, let's add:
• hs-v3-min Each test should only take 30-60 seconds, so I think it's worth running these 3 tests. (That's about 2 extra minutes per branch, or an 18 extra minutes for a full merge-forward from 0.2.9.)

Here's the full list of all the tests in chutney from #29729 (moved): https://github.com/torproject/chutney/pull/25/files#diff-354f30a63fb0907d4ad57269548329e3R29

We're missing:

• basic-min
• single-onion-v23
• and all the IPv6 tests
  • We can only run IPv6 tests in macOS Travis, because Linux Travis doesn't support IPv6. I opened #30212 (moved) to add them later.

Then let's let CI run for a while to evaluate the chutney tests.

If we think chutney is too unstable in Tor's Chutney job, or Chutney's CI, we could:

• run chutney in offline mode (#30183 (moved)), or
• add a sleep after failure in test-network --forgiving (#30210 (moved)).

Trac:
Status: needs_review to needs_revision
Points: N/A to 1

Oops, I forgot to review 0.3.4 and later:

I am not so sure about this part of the merge commit to 0.3.4:

++    - env: DISTCHECK="yes" RUST_OPTIONS="--enable-rust --enable-cargo-online-mode" HARDENING_OPTIONS="" SKIP_MAKE_CHECK="yes"

I don't think we should skip make check on the only online rust build. We could move online to the dirauth build, and skip make check on distcheck?

Also, in future, let's try to do semantic changes like this in a separate commit? https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/MergePolicy?action=diff&version=11

In 0.3.5, you dropped the SKIP_MAKE_CHECK you added in 0.3.4:

 -    ## But without hardening (see above)
 -    - env: DISTCHECK="yes" RUST_OPTIONS="--enable-rust --enable-cargo-online-mode" HARDENING_OPTIONS="" SKIP_MAKE_CHECK="yes"
 +    - env: DISTCHECK="yes" RUST_OPTIONS="--enable-rust --enable-cargo-online-mode"

We should also re-run all these branches with the latest chutney.