Opened 2 years ago

Closed 2 years ago

#22301 closed defect (fixed)

make test-stem doesn't work

Reported by: catalyst Owned by: atagar
Priority: Medium Milestone: Tor: 0.3.2.x-final
Component: Core Tor/Stem Version:
Severity: Normal Keywords:
Cc: chelseakomlo Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by catalyst)

Running make test-stem doesn't seem to work on master (tor master with stem master). This probably prevents people from running make test-full, which CodingStandards.md recommends.

There are probably multiple bugs involved. (I can open child tickets for them if people think that's bettter.) Some of them involve invoking run_tests.py with a working directory not at the top of the stem source tree. At least one other involves the lack of a tor binary in the default $PATH. (I'm not sure whether the test in question expects to run the system tor.)

Running

STEM_SOURCE_DIR=`pwd`/../stem CHUTNEY_PATH=`pwd`/../chutney make test-stem

produces many errors like

Starting ./src/or/tor...

  failed to start tor: './src/or/tor' doesn't exist

This suggests that run_tests.py doesn't cope well with a relative path to the tor binary, so as a workaround I tried setting TESTING_TOR_BINARY like

STEM_SOURCE_DIR=`pwd`/../stem CHUTNEY_PATH=`pwd`/../chutney make TESTING_TOR_BINARY=`pwd`/src/or/tor test-stem

which produced

  installation...                                      failed (0.04s)
test_install                                          37 ms  [SUCCESS]
test_sdist                                                   [FAILURE]

======================================================================
FAIL: test_sdist
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tlyu/src/stem/test/util.py", line 245, in wrapped
    return func(self, *args, **kwargs)
  File "/home/tlyu/src/stem/test/integ/installation.py", line 144, in test_sdist
    raise SDIST_FAILURE
AssertionError: /home/tlyu/src/stem/dist already exists, maybe you manually ran 'python setup.py sdist'?

----------------------------------------------------------------------
Ran 2 tests in 0.038s

FAILED (failures=1)

and

  process...                                           failed (1.03s)
test_can_run_multithreaded (already ran)                     [SKIPPED]
test_dump_config_argument                            182 ms  [SUCCESS]
test_hash_password                                    42 ms  [SUCCESS]
test_hash_password_requires_argument                  33 ms  [SUCCESS]
test_help_argument                                    70 ms  [SUCCESS]
test_hush_argument                                    70 ms  [SUCCESS]
test_launch_tor_with_config_via_file (already ran)           [SKIPPED]
test_launch_tor_with_config_via_stdin (already ran)          [SKIPPED]
test_launch_tor_with_timeout (already ran)                   [SKIPPED]
test_list_fingerprint_argument                       230 ms  [SUCCESS]
test_list_torrc_options_argument                      34 ms  [SUCCESS]
test_no_orphaned_process                                     [FAILURE]
test_quiet_argument                                   37 ms  [SUCCESS]
test_take_ownership_via_controller (already ran)             [SKIPPED]
test_take_ownership_via_pid (already ran)                    [SKIPPED]
test_torrc_arguments                                  46 ms  [SUCCESS]
test_torrc_arguments_via_stdin                        45 ms  [SUCCESS]
test_validate_config_argument                         95 ms  [SUCCESS]
test_version_argument                                 39 ms  [SUCCESS]
test_with_invalid_config (already ran)                       [SKIPPED]
test_with_missing_torrc                               97 ms  [SUCCESS]

======================================================================
ERROR: test_no_orphaned_process
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tlyu/src/stem/test/util.py", line 260, in wrapped
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/tlyu/src/stem/test/integ/process.py", line 217, in test_no_orphaned_process
    stem.process.launch_tor()
  File "/home/tlyu/src/stem/stem/process.py", line 98, in launch_tor
    raise OSError("'%s' isn't available on your system. Maybe it's not in your PATH?" % tor_cmd)
OSError: 'tor' isn't available on your system. Maybe it's not in your PATH?

----------------------------------------------------------------------
Ran 21 tests in 1.033s

FAILED (errors=1, skipped=7)

The summary says

TESTING FAILED (72 seconds)
  [RUN_NONE] test_sdist (test.integ.installation.TestInstallation) ... FAIL
  [RUN_NONE] test_no_orphaned_process (test.integ.process.TestProcess) ... ERROR
  [RUN_OPEN] test_no_orphaned_process (test.integ.process.TestProcess) ... ERROR
  [RUN_PASSWORD] test_no_orphaned_process (test.integ.process.TestProcess) ... ERROR
  [RUN_COOKIE] test_no_orphaned_process (test.integ.process.TestProcess) ... ERROR
  [RUN_MULTIPLE] test_no_orphaned_process (test.integ.process.TestProcess) ... ERROR
  [RUN_SOCKET] test_no_orphaned_process (test.integ.process.TestProcess) ... ERROR
  [RUN_SCOOKIE] test_no_orphaned_process (test.integ.process.TestProcess) ... ERROR
  [RUN_PTRACE] test_no_orphaned_process (test.integ.process.TestProcess) ... ERROR

You can re-run just these tests with:

  /home/tlyu/src/tor/../stem/run_tests.py --tor /home/tlyu/src/tor/src/or/tor --all --log notice --target RUN_ALL --test test.integ.installation
  /home/tlyu/src/tor/../stem/run_tests.py --tor /home/tlyu/src/tor/src/or/tor --all --log notice --target RUN_ALL --test test.integ.process

Running from the top of the stem source tree

./run_tests.py --tor `pwd`/../tor/src/or/tor --all --log notice --target RUN_ALL --test test.integ.installation

succeeds

14 TESTS WERE SKIPPED
TESTING PASSED (10 seconds)

but

./run_tests.py --tor `pwd`/../tor/src/or/tor --all --log notice --target RUN_ALL --test test.integ.process

still fails in the same way. I don't have a system tor binary installed in this VM, but I don't see any documentation that such is required for this test. Also it would be good to not accidentally test the system tor binary when we intend to test a specific other one.

Child Tickets

TicketStatusOwnerSummaryComponent
#22365closedatagartest.integ.installation expects cwd to be the top of the stem source treeCore Tor/Stem
#22366closedatagarrun_tests.py can't find tor via relative pathname if cwd not the top of the stem treeCore Tor/Stem
#22367closedatagartest.integ.process seems to expect a tor binary in $PATHCore Tor/Stem

Change History (7)

comment:1 Changed 2 years ago by catalyst

Description: modified (diff)

comment:2 Changed 2 years ago by atagar

Thanks catalyst! Presently at a conference (PyCon) but I'll take a look at these when I get back.

comment:3 Changed 2 years ago by catalyst

Also thanks to toby_ for initially pointing this out on IRC and chelseakomlo for confirming.

comment:4 Changed 2 years ago by atagar

Hi catalyst, amended the tests so they now pass when I run them with 'env -i'. Mind giving 'em another whirl?

On a side note generally speaking the tests are not broken. I get the sense from your network team meeting status that you feel these are major issues. They're durability bugs and definitely worth correcting, but they're cropping up because your setup is a bit unusual. We run these tests in Jenkins so you can generally expect them to be highly reliable (I'm quick to update the tests or file a tor ticket when they fail).

Thanks for pointing this out!

comment:5 Changed 2 years ago by catalyst

Thanks for looking into these and working on patches.

I tried your latest changes and unfortunately still see all the failures that I reported above. The tests indeed almost all work when I run from the top of the stem source tree, but that wasn't the problem that toby_ initially reported.

  • make test-stem (and thus make test-full) still fails from the top of the tor source tree, failing to find ./src/or/tor. This should not be an unusual situation because CodingStandards.md recommends that people run make test-full to test their changes. (If we don't expect make test-full to work, we should update CodingStandards.md.)
  • Applying the workaround of specifying an absolute path to the tor binary in the source tree when running make test-stem still produces the same errors in test.integ.install and test.integ.process. I infer that test.integ.install expects its working directory to be the top of the stem source tree and not anywhere else, but this can't work with the current make test-stem rules in the tor source tree.
  • Running ./run_tests.py from the top of the stem source tree (and giving it an absolute path to the tor binary in the tor source tree) clears the install problem but not the process problem. Not having a system tor installed might be unusual but I think we should support that use case. (Also we shouldn't be testing the system tor unless we specifically intend to. Do we actually intend to test the system tor here?)

I'm sorry if I misspoke about the $PATH issue. I meant that I don't have a tor binary installed on this system, and somehow the tests were trying to look for a tor binary in my $PATH, not that I had an empty or missing $PATH environment variable.

I saw that Jenkins runs automated tests in the stem source tree, but I didn't see anything that runs make test-stem or make test-full from the tor source tree, which is something that we should make sure works if we're recommending that developers run that when testing proposed changes to tor.

I think I will make separate child tickets because these issues are partially separable.

comment:6 Changed 2 years ago by atagar

make test-stem (and thus make test-full) still fails from the top of the tor source tree

Gotcha. I tried running the tests from various relative paths when making these patches but couldn't repro the issue...

% env -i ./stem/run_tests.py --all --tor tor/tor/src/or/tor
% env -i ../run_tests.py --all --tor ../../tor/tor/src/or/tor

Maybe a bug with the make target rather than stem?

... expects its working directory to be the top of the stem source tree and not anywhere else

Hmmm, the test calls os.chdir() in an effort to avoid these issues...

https://gitweb.torproject.org/stem.git/tree/test/integ/installation.py#n34

Are you able to come up with a repro invoking run_tests.py directly rather than tor's make target?

Not having a system tor installed might be unusual but I think we should support that use case.

Absolutely agreed, and this is a miss on my side. Puzzled though. I'm expecting the 'env -i' to allow me to repro this...

I'm sorry if I misspoke about the $PATH issue. I meant that I don't have a tor binary installed on this system, and somehow the tests were trying to look for a tor binary in my $PATH, not that I had an empty or missing $PATH environment variable.

Oh no, you didn't misspeak. I'm dropping my PATH in an effort to repro the issue you mentioned. That's all.

I think I will make separate child tickets because these issues are partially separable.

Sounds good!

comment:7 Changed 2 years ago by atagar

Resolution: fixed
Status: newclosed

Hi catalyst, think I'm gonna resolve this in favor of the child tickets you filed.

Note: See TracTickets for help on using tickets.