Opened 11 months ago

Closed 5 months ago

#27912 closed enhancement (fixed)

Add travis CI for the Chutney repository

Reported by: nickm Owned by: teor
Priority: High Milestone:
Component: Core Tor/Chutney Version:
Severity: Normal Keywords:
Cc: teor Actual Points: 1.5
Parent ID: #20647 Points: 0.5
Reviewer: Sponsor: Sponsor19

Description


Child Tickets

TicketTypeStatusOwnerSummary
#16454defectclosedchutney's "tor does not support this option" is unclear
#24004defectclosedChutney should show tor logs when tor fails to launch
#24939defectclosedchutney should log warnings from tor-gencert and other startup processes
#27970defectclosedteorChutney's test-network.sh --debug has never worked

Change History (40)

comment:1 Changed 11 months ago by nickm

Priority: MediumHigh

comment:2 Changed 10 months ago by teor

This ticket involves two tasks:

  • add another stage to tor's travis config that runs chutney (eventually, tor's second stage could run RAM usage, performance, and other runtime tests)
  • add a travis config to chutney that installs whichever tor is available in the package manager, then runs a chutney verify. We should run chutney tests on python 2 and 3, and Linux and macOS

comment:3 Changed 10 months ago by teor

Component: Core Tor/TorCore Tor/Chutney
Parent ID: #20647

#20647 is a duplicate, let's use this ticket for the chutney changes

comment:4 Changed 10 months ago by nickm

Owner: set to catalyst
Points: 0.5
Status: newassigned

Taylor, could you try out the minimal version here? That is, just add a 'test-network' step and see if it works? If there are bugs, don't feel you need to chase them: you can just open tickets.

comment:5 in reply to:  2 Changed 10 months ago by catalyst

Replying to teor:

This ticket involves two tasks:

  • add another stage to tor's travis config that runs chutney (eventually, tor's second stage could run RAM usage, performance, and other runtime tests)

Stages and jobs on Travis don't share storage, so I think we'd have to find a way to persist stuff if we want to try this.

  • add a travis config to chutney that installs whichever tor is available in the package manager, then runs a chutney verify. We should run chutney tests on python 2 and 3, and Linux and macOS

Sounds good. At least on Debian OSes, the system tor package seems to start a tor daemon by default. I'm not sure we want this.

comment:6 Changed 10 months ago by catalyst

Status: assignednew

#28013 is for the tor repository parts of this work. Let's leave this ticket for the chutney repository parts.

comment:7 Changed 10 months ago by catalyst

Owner: catalyst deleted
Status: newassigned

comment:8 Changed 10 months ago by catalyst

Status: assignednew

comment:9 Changed 10 months ago by nickm

Milestone: Tor: 0.3.5.x-finalTor: 0.3.6.x-final

comment:10 Changed 10 months ago by teor

Summary: Add Chutney to travis-ci if possibleAdd travis CI for the Chutney repository

Make title clearer

comment:11 Changed 10 months ago by teor

Owner: set to teor
Status: newassigned

I have an experimental version in my chutney-travis branch at https://github.com/teor2345/chutney.git

It's still going through CI.

comment:12 Changed 10 months ago by teor

Status: assignedneeds_revision

comment:13 Changed 10 months ago by teor

I have a working travis config, and much better debugging:
https://github.com/torproject/chutney/pull/2

0.2.9, 0.3.3, and 0.3.4 fail less than 1% of the time.
Example failure on 0.2.9:
https://travis-ci.org/torproject/chutney/jobs/445982217

0.3.5 and 0.3.6 fail about 90% of the time, so I put them in allow_failures, and opened #28192 to track down the bug.

I want to copy the make test-network-all networks from each tor version into the travis config. Then I'll be done with this branch.

comment:14 Changed 10 months ago by teor

We can't fix 0.3.4 and earlier, so I want to run them twice (or however long it takes to get an acceptable error rate), and make the test pass if either succeeds.

comment:15 in reply to:  13 Changed 10 months ago by teor

Replying to teor:

...
I want to copy the make test-network-all networks from each tor version into the travis config. Then I'll be done with this branch.

Since this is chutney, I'll just run the networks from tor stable on stable, and any extra networks in master on nightly. That's enough.

Then I need to decide on a default network: either basic-min, or chutney's default bridges-etc-etc-etc.

Edit: I also want to split the debug and travis changes into separate branches for review.

Last edited 10 months ago by teor (previous) (diff)

comment:16 Changed 10 months ago by teor

Edit: I also want to split the debug and travis changes into separate branches for review.

I merged the debug branch to master, rebased the chutney-travis branch, and closed the relevant tickets.

comment:17 Changed 9 months ago by nickm

Milestone: Tor: 0.3.6.x-finalTor: 0.4.0.x-final

Tor 0.3.6.x has been renamed to 0.4.0.x.

comment:18 Changed 8 months ago by teor

Like #28922, we should think about copying the torproject key into the chutney repository, to reduce Travis network errors.

comment:19 Changed 7 months ago by teor

Milestone: Tor: 0.4.0.x-final

comment:20 Changed 6 months ago by teor

Here's what's left to do in this ticket:

  1. Update the Tor versions in .travis.yml. And maybe the python versions as well?
  1. Work out how reliable tor/chutney is now, and run jobs in a loop if needed, failing after N failures
  1. Is using a keyserver unreliable on Travis?
    • In #28922, sbws gave up on the keyserver, and just committed the key to the sbws git

Optional, but probably useful:

  1. Work out which networks to run, and if they should have separate jobs. We don't need to test all the networks, just the ones that test chutney features:
    • a bridge network from "make test-network-all"
    • an onion service v2 and v3 network from "make test-network-all"
    • an IPv6 network from "make test-network-all"
    • any others?

comment:21 Changed 6 months ago by teor

  1. We should rename "cached-stable" and "stable" to "macos-brew" and "deb-tpo-trusty"
    • Are we still building for trusty?

comment:22 in reply to:  21 Changed 6 months ago by teor

Replying to teor:

  1. We should rename "cached-stable" and "stable" to "macos-brew" and "deb-tpo-trusty"
    • Are we still building for trusty?

We might have to switch Travis to xenial images to get Tor binaries:
https://docs.travis-ci.com/user/reference/xenial/#using-xenial

(And update the deb lines in .travis.yml)

comment:23 Changed 6 months ago by nickm

Status: needs_revisionneeds_review

I've updated the branch as 'chutney-travis-v2'. It won't work without the fixes for #29618, though.

Here is a PR for a branch containing both sets of fixes: https://github.com/torproject/chutney/pull/3

comment:24 Changed 6 months ago by nickm

Actual Points: .5

comment:25 Changed 5 months ago by teor

Status: needs_reviewneeds_revision

My review on #29280 suggests moving test-network-forgiving.sh to the chutney repository, because if chutney/tor is unreliable, we'll need it for both sets of CI:
https://github.com/torproject/tor/pull/736

Last edited 5 months ago by teor (previous) (diff)

comment:26 Changed 5 months ago by teor

Owner: changed from teor to nickm
Status: needs_revisionassigned

comment:27 Changed 5 months ago by nickm

Now that #29618 is merged, https://github.com/torproject/chutney/pull/6 is now a PR containing only this ticket.

I suggest that if it works okay, we merge it before working on test-network-forgiving.sh in chutney

comment:28 Changed 5 months ago by nickm

Status: assignedneeds_review

comment:29 in reply to:  21 Changed 5 months ago by teor

I did a review, but I think I want to make these changes.

Replying to teor:

Here's what's left to do in this ticket:

  1. Update the Tor versions in .travis.yml. And maybe the python versions as well?

And add all supported tor versions as well.

  1. Work out how reliable tor/chutney is now, and run jobs in a loop if needed, failing after N failures

It looks really reliable: I haven't seen anything fail yet.
Let's see if we really need test-network-forgiving?

  1. Is using a keyserver unreliable on Travis?
    • In #28922, sbws gave up on the keyserver, and just committed the key to the sbws git

Nick fixed this issue by using deb.torproject.org.

Optional, but probably useful:

  1. Work out which networks to run, and if they should have separate jobs. We don't need to test all the networks, just the ones that test chutney features:
    • a bridge network from "make test-network-all"
    • an onion service v2 and v3 network from "make test-network-all"
    • an IPv6 network from "make test-network-all"
    • any others?

I'll open a ticket for this.

Replying to nickm:

Now that #29618 is merged, https://github.com/torproject/chutney/pull/6 is now a PR containing only this ticket.

I suggest that if it works okay, we merge it before working on test-network-forgiving.sh in chutney

Sounds good.

comment:30 Changed 5 months ago by teor

Sponsor: Sponsor19

comment:31 Changed 5 months ago by teor

I opened #29729 to add jobs for each of chutney's major feature categories.
We don't need a full matrix: that will go in Tor's CI.

comment:32 Changed 5 months ago by teor

Owner: changed from nickm to teor
Status: needs_reviewassigned

Hmm, maybe this should be mine again.

comment:33 Changed 5 months ago by teor

Status: assignedneeds_revision

comment:34 Changed 5 months ago by teor

I don't know how we can remember to update our travis config when deb.tpo repositories change. I opened #29741 as a low-priority task to work that out.

comment:35 Changed 5 months ago by teor

Actual Points: .51.5

My updates are in https://github.com/torproject/chutney/pull/7

I'm going to take a break now, and work on #29729 later today, or tomorrow (in 2-16 hours time).

I think we're going to need test-network-forgiving soon after we merge this CI, I'm seeing failures in about 1 in 30 jobs. Maybe the failure rate depends on Travis' load: I've also seen almost all jobs fail in some builds.

comment:36 Changed 5 months ago by nickm

This looks fine to me. I'm going to base my revised test-network-forgiving work on this branch, so it can see testing.

Also, I'm inclined to squash and merge this branch as you have it now, and merge more changes later. Would that be ok with you?

comment:37 Changed 5 months ago by teor

Sure. I think we'll want #29729 soon, so I'll do it today or tomorrow. But I can't see any reason to delay basic CI just so we can add a few more variants.

comment:38 Changed 5 months ago by teor

Status: needs_revisionmerge_ready

I can squash and merge today while I'm doing other merges?

comment:39 Changed 5 months ago by nickm

Yes, please feel free.

comment:40 Changed 5 months ago by teor

Resolution: fixed
Status: merge_readyclosed

Squashed to chutney-travis-v4-squashed, but without re-ordering the fixup commits (one of them did't apply when moved earlier in the branch).

CI passed, merged to master as 9273887.

Note: See TracTickets for help on using tickets.