Add a chutney job that runs on macOS, so that IPv6 chutney tests work

Travis Linux doesn't support IPv6, so we should add a macOS chutney job to Tor's CI.

If that's too slow, we can:

• add the chutney tests to the end of an existing macOS job
• change the chutney job to macOS

Remember: each build has a limit of 2 concurrent macOS jobs.

We can do this task after #29280 (moved) merges.

changed milestone to %Tor: 0.2.9.x-final

Trac:
Parent Ticket: #31851 (moved)

added 029-backport 035-backport 040-backport 041-backport 042-deferred-20190918 CI PTs actualpoints::1.5 component::core tor/tor consider-backport-after-ci-passes milestone::Tor: 0.2.9.x-final network-team-roadmap-2019-Q1Q2 owner::teor parent::31851 points::0.5 priority::medium resolution::fixed reviewer::nickm severity::normal sponsor-28-maybe sponsor::31-can status::closed teor-backlog-ci teor-merge tor-ci type::enhancement labels

Trac:
Parent: #29280 (moved) to #29267 (moved)
Sponsor: N/A to Sponsor28-can

Hi Gaba, I put this task in sponsor 28-can because it is related to #29280 (moved), PTs, and chutney.

It should be a quick fix, please put it back in if you think it is important.

Trac:
Sponsor: Sponsor28-can to N/A
Keywords: N/A deleted, sponsor-28-maybe added
Cc: cohosh to cohosh, gaba

Some of these tickets are part of the sponsor 31 CI improvements task, we will pick the specific tickets later, and assign them to people.

Trac:
Sponsor: N/A to Sponsor31-can
Owner: teor to N/A
Keywords: N/A deleted, teor-backlog-ci added

Deferring various tickets from 0.4.2 to Unspecified.

Trac:
Milestone: Tor: 0.4.2.x-final to Tor: unspecified
Keywords: N/A deleted, 042-deferred-20190918 added

I'm going to redesign the test matrix in #31851 (moved) to allow for the new module tests.

This would also be a good opportunity to do:

• macOS -> macOS chutney
• remove Linux chutney

We should not change:

• macOS rust Because we've had bugs in macOS rust compilation on some previous rustc versions.

Trac:
Parent: #29267 (moved) to #31851 (moved)
Cc: cohosh, gaba to N/A

We need this ticket to test the sponsor 27 HSv3 IPv6 code. But it can also be part of the Sponsor 31 CI improvements.

Trac:
Sponsor: Sponsor31-can to Sponsor27-can

Please review and merge these PRs:

• 0.2.9: https://github.com/torproject/tor/pull/1369
  • runs macOS clang IPv6 make check test-network-all (Sponsor 27)
  • removes 2 jobs (#31859 (moved))
• 0.3.5: https://github.com/torproject/tor/pull/1370
  • removes 3 more jobs (#31859 (moved))
• master: https://github.com/torproject/tor/pull/1373
  • merges RUST_VERSION into the modified Rust job

Clean merges - testing only:

• 0.4.0: https://github.com/torproject/tor/pull/1371
• 0.4.1: https://github.com/torproject/tor/pull/1372

Trac:
Milestone: Tor: unspecified to Tor: 0.4.2.x-final
Actualpoints: N/A to 0.4
Status: assigned to needs_review

Trac:
Status: needs_review to assigned
Owner: N/A to teor

(Wanted: ticket system that changes owners in one step)

Trac:
Status: assigned to needs_review

Trac:
Cc: N/A to catalyst

This change decreases the total time spent on the CI infrastructure, but it actually increases the wall-clock time.

I want to try these changes: 0. Re-run all the builds, so they benefit from the build caches

1. Reorder the builds so the slowest builds run first
2. De-merge the chutney/check and distcheck/rust builds

Edited to add:

Decreasing the total time spent on the CI infrastructure does speed up the next build, because we are using fewer CI builders. But that only really benefits backports and alpha+master merges.

Trac:
Status: needs_review to needs_revision

Replying to teor:

I want to try these changes: 0. Re-run all the builds, so they benefit from the build caches

The build caches aren't the issue.

1. Reorder the builds so the slowest builds run first
2. De-merge the chutney/check and distcheck/rust builds

Rather than de-merging the builds, I disabled make check on them. I also put the slowest compilers (macOS/gcc, Linux/clang) on the fastest build on each platform.

I activated fast_finish, but I don't think it will make much difference: the chutney job is the slowest job.

Here are the latest PRs:

• 0.2.9: ​https://github.com/torproject/tor/pull/1369
• 0.3.5: ​https://github.com/torproject/tor/pull/1370
• 0.4.1: ​https://github.com/torproject/tor/pull/1372
• master: ​https://github.com/torproject/tor/pull/1373

Clean merges - testing only:

• 0.4.0: ​https://github.com/torproject/tor/pull/1371

But macOS chutney is really slow, so I might need to make a separate make target that only runs the IPv6 networks.

I think the fastest solution to this issue is to:

• add a linux chutney job
• allow failures on the macOS chutney job

It would be nice to have a Travis config for "finish before this job is done, but fail the build if it fails". But maybe we should just speed up our tests :-)

Please review my PRs:

• 0.2.9: ​https://github.com/torproject/tor/pull/1369
• 0.3.5: ​https://github.com/torproject/tor/pull/1370
• 0.4.1: ​https://github.com/torproject/tor/pull/1372
• master: ​https://github.com/torproject/tor/pull/1373

Clean merges - testing only:

• 0.4.0: ​https://github.com/torproject/tor/pull/1371

macOS chutney and Rust are really slow, so I use fast_finish and allow_failure to let the job finish before they are done. Unfortunately, that means we don't see failures for those jobs in the build summary or IRC notifications. But we have enough fast tests, so I expect failures to be rare.

Trac:
Status: needs_revision to needs_review
Keywords: N/A deleted, 041-backport, consider-backport-after-ci-passes added
Actualpoints: 0.4 to 1.0

Here are the CI wallclock build time improvements, in minutes.

Release	Current Speed	PR Speed	Improvement
0.2.9	16	9	44%
0.3.5	30	11	63%
0.4.0	20	13	35%
0.4.1	32	20	38%
master	35	13	63%

These times vary depending on server load, but they are clearly better.

Trac:
Reviewer: N/A to nickm

This LGTM. Let's give it a try. One thing I don't understand, though, is the table above with different speeds. Are the columns reversed or partially reversed? 1-(9/16) == 44%, but 1-(35/13) != 63%.

Since this is CI, I think we should merge it all at once. Would you be okay doing the merges?

Trac:
Status: needs_review to merge_ready

Trac:
Keywords: N/A deleted, teor-merge added

Replying to nickm:

This LGTM. Let's give it a try. One thing I don't understand, though, is the table above with different speeds. Are the columns reversed or partially reversed? 1-(9/16) == 44%, but 1-(35/13) != 63%.

Yes, I think I made a mistake, I'll fix the table now.

Since this is CI, I think we should merge it all at once. Would you be okay doing the merges?

I can do the merges.

I can do the merges.

Great! Please do when you have the chance.

Replying to teor:

Please review my PRs:

• 0.2.9: ​https://github.com/torproject/tor/pull/1369
• 0.3.5: ​https://github.com/torproject/tor/pull/1370
• 0.4.1: ​https://github.com/torproject/tor/pull/1372
• master: ​https://github.com/torproject/tor/pull/1373

Clean merges - testing only:

• 0.4.0: ​https://github.com/torproject/tor/pull/1371

Merged to 0.2.9 - 0.4.2: the master branch now corresponds to maint-0.4.2, and merges cleanly. And maint-0.4.2 merges cleanly to master.

Trac:
Milestone: Tor: 0.4.2.x-final to Tor: 0.2.9.x-final
Status: merge_ready to closed
Actualpoints: 1.0 to 1.5
Resolution: N/A to fixed

Trac:
Sponsor: Sponsor27-can to Sponsor31-can

closed

changed time estimate to 4h

added 12h of time spent

mentioned in issue #31859 (moved)

mentioned in issue #31882 (moved)

mentioned in issue #31851 (moved)

moved to tpo/core/tor#30860 (closed)

mentioned in issue tpo/core/tor#31859 (closed)