Opened 11 months ago

Last modified 3 months ago

#28170 new enhancement

Test stem pull requests against all supported tor versions, and tor nightly builds

Reported by: teor Owned by:
Priority: Medium Milestone:
Component: Core Tor/Stem Version:
Severity: Normal Keywords: ci, travis
Cc: teor Actual Points: 0.3
Parent ID: Points: 0.3
Reviewer: Sponsor: Sponsor31-can

Description

In #27913, we want to test stem against all supported tor versions, and tor nightly.

Child Tickets

TicketTypeStatusOwnerSummary
#30593defectclosedteorFix argument passing in docs/republish
#30594defectclosedatagarstem unit tests fail on Tor 0.2.9, which is still supported until 1 January 2020
#30595defectnewatagarStem unit tests hang after doctest on macOS
#30596defectneeds_revisionatagarstem tests expect platform.linux_distribution, which does not exist in python 3.8
#30597defectclosedatagarbandwidth_file test_version_position fails on python 3
#30598defectclosedatagartest_relay_cell fails on pypy 3.5
#30650defectclosedteorRemove shellcheck from Stem's travis
#30651enhancementassignedMake stem's Travis jobs log controller messages from stem and tor
#30652taskclosedteorconfigure Stem's Travis CI to email atagar and the committer when there are failures
#30653enhancementclosedteorAdd stem CI for --target ONLINE
#30696defectclosedatagarstem's test_close_stream fails with an extra stream in --target ONLINE mode
#30697defectclosedatagartest_get_hidden_service_descriptor fails with an unexpected response code
#30698enhancementnewatagarShould we add TRACE to stem's CI?

Change History (20)

comment:1 Changed 11 months ago by atagar

Hi teor, what is the ask from me on this?

comment:2 in reply to:  1 Changed 11 months ago by teor

Replying to atagar:

Hi teor, what is the ask from me on this?

We want to make sure that changes to stem master don't break tor's CI. (I realise it's usually the other way around.)

I'm going to write a patch to chutney to test against all supported tor versions and tor nightly (see #28170). Then I'll copy the tor version matrix into stem's travis CI config, and update this ticket with the branch.

If you could review and merge my patch to stem's travis ci config, that would be great.

comment:3 Changed 11 months ago by atagar

We want to make sure that changes to stem master don't break tor's CI.

Gotcha. I should be made aware of regressions on my side by Stem's Jenkins runs. That tests against tor's tip (not against multiple tor versions) but I've found this tends to be sufficient for bugs on my end.

https://jenkins.torproject.org/job/stem-tor-ci/

Then I'll copy the tor version matrix into stem's travis CI config, and update this ticket with the branch.

Sorry, not sure I quite follow. Tor uses travis, but I don't for Stem.

comment:4 Changed 11 months ago by nickm

Parent ID: #27913

comment:5 in reply to:  3 Changed 11 months ago by teor

Replying to atagar:

We want to make sure that changes to stem master don't break tor's CI.

Gotcha. I should be made aware of regressions on my side by Stem's Jenkins runs. That tests against tor's tip (not against multiple tor versions) but I've found this tends to be sufficient for bugs on my end.

https://jenkins.torproject.org/job/stem-tor-ci/

Then I'll copy the tor version matrix into stem's travis CI config, and update this ticket with the branch.

Sorry, not sure I quite follow. Tor uses travis, but I don't for Stem.

The stem repository contains a travis config, and it runs under the torproject's travis:
https://travis-ci.org/torproject/stem

comment:6 Changed 8 months ago by nickm

Keywords: 035-deferred-20190115 041-proposed added
Milestone: Tor: 0.3.5.x-finalTor: unspecified

These tickets are all deferred from 0.3.5, since they are about updating CI and/or chutney settings.

comment:7 Changed 7 months ago by teor

Keywords: teor-unreached-2019-03-08 added
Owner: teor deleted

I'd like to do these tickets, but not in the next few months.

comment:8 Changed 6 months ago by teor

Parent ID: #29729

We might have diagnosed #29437 faster if we ran Stem CI with python 3.6 and any supported tor version.

Once I finish #29729, I can copy chutney's Travis config to a Stem PR, and then drop in the stem test commands.

comment:9 Changed 4 months ago by teor

Actual Points: 0.2
Points: 0.2
Sponsor: Sponsor31-can
Status: assignedneeds_review

This ticket can be sponsor 31, because working stem tests help us refactor tor.

I used the chutney Travis config from #30279, with some minor edits based on the old stem Travis config.

See my pull request:
https://github.com/torproject/stem/pull/14

Or the build for the original branch:
https://travis-ci.org/teor2345/stem/builds/536613169

We're seeing the hangs from #29437 here, as well as a bunch of other issues. I'll open child tickets for them.

comment:10 Changed 4 months ago by teor

Keywords: 035-deferred-20190115 041-proposed teor-unreached-2019-03-08 removed
Milestone: Tor: unspecified

comment:11 Changed 4 months ago by atagar

Thanks teor! Merged.

Before resolving this I'd like to check - does it make sense for the travis config to live in this repository? I use Jenkins for my CI. I don't use this configuration, and as such don't have an opinion about what's in it or know if it breaks.

I'd be happy to merge changes if you'd like to keep it here, but I'm unsure if there's value in requiring me to be in the loop on its maintenance.

Last edited 4 months ago by atagar (previous) (diff)

comment:12 Changed 4 months ago by atagar

Oh! And to be clear, if you'd care to suggest that I use Travis rather than Jenkins for my CI I'd be open to discussing that too. Personally I don't have any strong opinions at the moment about what kind of CI Stem uses for itself.

comment:13 in reply to:  12 Changed 4 months ago by teor

Replying to atagar:

Before resolving this I'd like to check - does it make sense for the travis config to live in this repository? I use Jenkins for my CI. I don't use this configuration, and as such don't have an opinion about what's in it or know if it breaks.

I'd be happy to merge changes if you'd like to keep it here, but I'm unsure if there's value in requiring me to be in the loop on its maintenance.

If the Travis CI config is in the Stem repository, then anyone can run Travis CI on their Stem pull requests. That's the standard config for CIs like Travis and Appveyor.

Using git for the CI config also allows different CI config for different release branches. That isn't as important for Stem, because your backports are usually pretty small.

Replying to atagar:

Oh! And to be clear, if you'd care to suggest that I use Travis rather than Jenkins for my CI I'd be open to discussing that too. Personally I don't have any strong opinions at the moment about what kind of CI Stem uses for itself.

The network team uses Travis and Appveyor to check our Tor branches. Our Travis and Appveyor tests are more comprehensive, because they run different build options, Windows, Stem, and soon Chutney. Travis and Jenkins also let us run CI on our branches before we merge them.

We use Jenkins to test rarer architectures, and build binary packages.

Tor's CI runs a Stem job. To avoid breaking Tor's CI, we need all of Stem's changes to pass Stem's CI. (And we need Stem's CI tests to be roughly equivalent to Tor's CI tests.) So we would really like to have Stem's Travis and Jenkins CIs pass all the time.

I don't know if you want to check both Travis and Jenkins after you merge, or if you want to standardise on one of them. (Checking multiple CI platforms adds more work.) But we'll check Stem's CI when Tor's Stem job fails, so we can work out if it was Tor or Stem that broke first.

We can also use Stem's Travis CI to check our Stem branches, but we don't write very many of them.

So it's totally up to you.

Let's try it for a while and see how it goes?

comment:14 Changed 4 months ago by atagar

Thanks for the detailed explanation. Can we configure our Stem CI to email me when there are failures?

Personally for me my ideal CI tests are...

  1. Run 'run_tests.py --all' with python 2.6.
  2. Run 'run_tests.py --all' with python 2.7.
  3. Run 'run_tests.py --all' with python 3.x.
  4. Run 'run_tests.py --all --target ONLINE' with python 2.7.

... and email me when any of these fail. Jenkins provides #2 but that's it, so swapping over to Travis sounds like it would be an improvement.

comment:15 Changed 4 months ago by teor

I have opened tickets #30650 through #30653 for follow up, I'll probably do them all in the same branch, because they all modify .travis.yml.

comment:16 Changed 4 months ago by teor

comment:17 Changed 4 months ago by teor

Actual Points: 0.20.3
Points: 0.20.3

comment:18 Changed 4 months ago by atagar

Thanks teor! Merged.

comment:19 Changed 4 months ago by teor

Parent ID: #29729

comment:20 Changed 3 months ago by teor

Status: needs_reviewnew

This is now a parent ticket, the follow up tickets are the children of this ticket

Note: See TracTickets for help on using tickets.