Opened 2 months ago

Last modified 2 months ago

#22635 new defect

Run tor's make test-stem in Jenkins

Reported by: catalyst Owned by: weasel
Priority: Medium Milestone:
Component: Internal Services/Service - jenkins Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Currently, our Jenkins CI runs stem's self-tests, but doesn't ensure that make test-stem works from the tor source tree. In the past make test-stem has failed for reasons described in #22301. We should run make test-stem in our CI so it can be reliable enough for developers to run regularly and know that its results are meaningful.

Child Tickets

Change History (3)

comment:1 Changed 2 months ago by atagar

Hi catalyst, not sure I follow. Tor's 'test-stem' target should simply be an alias for stem's 'run_tests.py --all --target RUN_ALL' which is presently run in jenkins for every tor or stem push...

https://jenkins.torproject.org/job/stem-tor-ci/

Issues you ran into were with peculiarities of your environment. Fine things to fix certainly, but they wouldn't have been caught by jenkins.

comment:2 Changed 2 months ago by catalyst

I think you're right that there are environmental dependencies for those issues, but I think they're not uncommon ones. I think to make this reproduce the observed failures, we would have to make sure there isn't a system tor binary installed in the Jenkins environment that runs this test.

As I recall, the issues in #22301 depended on both the lack of a tor binary in the default PATH and running make test-stem in the tor source tree (instead of running run_tests.py in the stem source tree). This is not an unusual combination, especially for a (possibly new) developer who has no reason to install a system tor. (Note at least 3 developers independently experienced those bugs while attempting to follow the guidelines in CodingStandards.md.)

Now that I think about it more, it's also useful to make sure that the stem tests don't unintentionally run the system tor (thereby producing invalid or misleading results).

comment:3 Changed 2 months ago by atagar

Gotcha. Actually, I do like the idea of adding test coverage for this but I suspect Jenkins isn't the right way to go. Maybe we should wrap stem's run_tests.py with a shell script invoking 'env -i' to avoid any accidental dependencies on environment variables?

Note: See TracTickets for help on using tickets.