Opened 3 years ago

Closed 2 years ago

#22902 closed defect (fixed)

run_tests.py hangs in process or control.controller

Reported by: catalyst Owned by: atagar
Priority: Medium Milestone:
Component: Archived/Stem Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Oddly enough it seems to happen less often with -v.

Excerpt from one of the hangs when using -v:

======================================================================
               test.integ.installation.TestInstallation               
======================================================================

test_install                                           0 ms  [SUCCESS]
test_sdist                                           627 ms  [SUCCESS]

----------------------------------------------------------------------
Ran 2 tests in 0.628s

OK

======================================================================
                    test.integ.process.TestProcess                    
======================================================================

During the hang I see the following tor and python processes:

tlyu      96853  1.4  9.0 21474964912 365204 pts/0 S+ 15:39   0:01 /home/tlyu/src/stem/../tor/src/or/tor -f /home/tlyu/src/stem/test/data/torrc __OwningControllerProcess 96673
tlyu      96673  4.4  1.0 394728 41408 pts/0    S+   15:39   0:05 python ./run_tests.py --tor ../tor/src/or/tor --all --log notice --target RUN_AL  -v
tlyu      96680  1.2  0.0      0     0 pts/0    ZN+  15:39   0:01 [python] <defunct>
tlyu      96686  0.0  0.0      0     0 pts/0    ZN+  15:39   0:00 [python] <defunct>
tlyu      96687  0.0  0.0      0     0 pts/0    ZN+  15:39   0:00 [python] <defunct>
tlyu      96689  0.0  0.0      0     0 pts/0    ZN+  15:39   0:00 [python] <defunct>
tlyu      96690  0.0  0.0      0     0 pts/0    ZN+  15:39   0:00 [python] <defunct>
tlyu      96692  0.0  0.0      0     0 pts/0    ZN+  15:39   0:00 [python] <defunct>
tlyu      96693  0.0  0.0      0     0 pts/0    ZN+  15:39   0:00 [python] <defunct>
tlyu      96701  0.0  0.0      0     0 pts/0    ZN+  15:39   0:00 [python] <defunct>
tlyu      96702  0.0  0.0      0     0 pts/0    ZN+  15:39   0:00 [python] <defunct>

Child Tickets

Change History (5)

comment:1 Changed 3 years ago by atagar

Hi catalyst, think I know why you're getting a deadlock. Seems unexpected exceptions cause aynchronously run tests to never join. Fixing that. That said, there's likely a stacktrace somewhere earlier in your testing output and we need to sort out that too (what I'm working on now will simply turn the deadlock into a failing test).

comment:2 Changed 3 years ago by atagar

Status: newneeds_information

Fix pushed, you should no longer see deadlocks: https://gitweb.torproject.org/stem.git/commit/?id=7901ff1

Ball's back in your court. What's the process test failing with?

comment:3 Changed 3 years ago by catalyst

Thanks. I think I haven't seen a deadlock since updating to that.

Here's one failure (after I had been running run_tests.py in a loop for a while:

======================================================================
                    test.integ.process.TestProcess                    
======================================================================

test_can_run_multithreaded                             0 ms  [SUCCESS]
test_dump_config_argument                              0 ms  [SUCCESS]
test_hash_password                                     0 ms  [SUCCESS]
test_hash_password_requires_argument                   0 ms  [SUCCESS]
test_help_argument                                     0 ms  [SUCCESS]
test_hush_argument                                     0 ms  [SUCCESS]
test_launch_tor_with_config_via_file                   0 ms  [SUCCESS]
test_launch_tor_with_config_via_stdin                  0 ms  [SUCCESS]
test_launch_tor_with_timeout                           0 ms  [SUCCESS]
test_list_fingerprint_argument                         0 ms  [SUCCESS]
test_list_torrc_options_argument                       0 ms  [SUCCESS]
test_no_orphaned_process                               0 ms  [SUCCESS]
test_quiet_argument                                    0 ms  [SUCCESS]
test_take_ownership_via_controller                           [FAILURE]
test_take_ownership_via_pid                            0 ms  [SUCCESS]
test_torrc_arguments                                   0 ms  [SUCCESS]
test_torrc_arguments_via_stdin                         0 ms  [SUCCESS]
test_unanonymous_hidden_service_config_must_match      0 ms  [SUCCESS]
test_validate_config_argument                          0 ms  [SUCCESS]
test_version_argument                                  0 ms  [SUCCESS]
test_with_invalid_config                               0 ms  [SUCCESS]
test_with_missing_torrc                                0 ms  [SUCCESS]

======================================================================
FAIL: test_take_ownership_via_controller
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tlyu/src/stem/stem/util/test_tools.py", line 150, in <lambda>
    self.method = lambda test: self.result(test)  # method that can be mixed into TestCases
  File "/home/tlyu/src/stem/stem/util/test_tools.py", line 214, in result
    test.fail(self._result.msg)
AssertionError: Traceback (most recent call last):
  File "/home/tlyu/src/stem/stem/util/test_tools.py", line 165, in _wrapper
    runner(*args) if args else runner()
  File "/home/tlyu/src/stem/test/integ/process.py", line 625, in test_take_ownership_via_controller
    take_ownership = True,
  File "/home/tlyu/src/stem/stem/process.py", line 274, in launch_tor_with_config
    return launch_tor(tor_cmd, ['-f', '-'], None, completion_percent, init_msg_handler, timeout, take_ownership, stdin = config_str)
  File "/home/tlyu/src/stem/stem/process.py", line 153, in launch_tor
    raise OSError('Process terminated: %s' % last_problem)
OSError: Process terminated: Failed to bind one of the listener ports.


----------------------------------------------------------------------
Ran 22 tests in 0.001s

FAILED (failures=1)


Shutting down tor... done

Style checks require pycodestyle version 1.4.2 or later. Please install it from...
  http://pypi.python.org/pypi/pycodestyle

TESTING FAILED (35 seconds)
  [RUN_NONE] test_take_ownership_via_controller (test.integ.process.TestProcess) ... FAIL
  [RUN_OPEN] test_take_ownership_via_controller (test.integ.process.TestProcess) ... FAIL
  [RUN_PASSWORD] test_take_ownership_via_controller (test.integ.process.TestProcess) ... FAIL
  [RUN_COOKIE] test_take_ownership_via_controller (test.integ.process.TestProcess) ... FAIL
  [RUN_MULTIPLE] test_take_ownership_via_controller (test.integ.process.TestProcess) ... FAIL
  [RUN_SOCKET] test_take_ownership_via_controller (test.integ.process.TestProcess) ... FAIL
  [RUN_SCOOKIE] test_take_ownership_via_controller (test.integ.process.TestProcess) ... FAIL
  [RUN_PTRACE] test_take_ownership_via_controller (test.integ.process.TestProcess) ... FAIL

You can re-run just these tests with:

  ./run_tests.py --tor ../tor/src/or/tor --all --log notice --target RUN_ALL -v --test test.integ.process

and another one:

======================================================================
                    test.integ.process.TestProcess                    
======================================================================

test_can_run_multithreaded                             0 ms  [SUCCESS]
test_dump_config_argument                              0 ms  [SUCCESS]
test_hash_password                                     0 ms  [SUCCESS]
test_hash_password_requires_argument                   0 ms  [SUCCESS]
test_help_argument                                     0 ms  [SUCCESS]
test_hush_argument                                     0 ms  [SUCCESS]
test_launch_tor_with_config_via_file                         [FAILURE]
test_launch_tor_with_config_via_stdin                  0 ms  [SUCCESS]
test_launch_tor_with_timeout                           0 ms  [SUCCESS]
test_list_fingerprint_argument                         0 ms  [SUCCESS]
test_list_torrc_options_argument                       0 ms  [SUCCESS]
test_no_orphaned_process                               0 ms  [SUCCESS]
test_quiet_argument                                    0 ms  [SUCCESS]
test_take_ownership_via_controller                     0 ms  [SUCCESS]
test_take_ownership_via_pid                            0 ms  [SUCCESS]
test_torrc_arguments                                   0 ms  [SUCCESS]
test_torrc_arguments_via_stdin                         0 ms  [SUCCESS]
test_unanonymous_hidden_service_config_must_match      0 ms  [SUCCESS]
test_validate_config_argument                          0 ms  [SUCCESS]
test_version_argument                                  0 ms  [SUCCESS]
test_with_invalid_config                               0 ms  [SUCCESS]
test_with_missing_torrc                                0 ms  [SUCCESS]

======================================================================
FAIL: test_launch_tor_with_config_via_file
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tlyu/src/stem/stem/util/test_tools.py", line 150, in <lambda>
    self.method = lambda test: self.result(test)  # method that can be mixed into TestCases
  File "/home/tlyu/src/stem/stem/util/test_tools.py", line 214, in result
    test.fail(self._result.msg)
AssertionError: Traceback (most recent call last):
  File "/home/tlyu/src/stem/stem/util/test_tools.py", line 165, in _wrapper
    runner(*args) if args else runner()
  File "/home/tlyu/src/stem/test/integ/process.py", line 445, in test_launch_tor_with_config_via_file
    completion_percent = 5
  File "/home/tlyu/src/stem/stem/process.py", line 285, in launch_tor_with_config
    return launch_tor(tor_cmd, args, torrc_path, completion_percent, init_msg_handler, timeout, take_ownership)
  File "/home/tlyu/src/stem/stem/process.py", line 153, in launch_tor
    raise OSError('Process terminated: %s' % last_problem)
OSError: Process terminated: Failed to bind one of the listener ports.


----------------------------------------------------------------------
Ran 22 tests in 0.001s

FAILED (failures=1)


Shutting down tor... done

Style checks require pycodestyle version 1.4.2 or later. Please install it from...
  http://pypi.python.org/pypi/pycodestyle

TESTING FAILED (36 seconds)
  [RUN_NONE] test_launch_tor_with_config_via_file (test.integ.process.TestProcess) ... FAIL
  [RUN_OPEN] test_launch_tor_with_config_via_file (test.integ.process.TestProcess) ... FAIL
  [RUN_PASSWORD] test_launch_tor_with_config_via_file (test.integ.process.TestProcess) ... FAIL
  [RUN_COOKIE] test_launch_tor_with_config_via_file (test.integ.process.TestProcess) ... FAIL
  [RUN_MULTIPLE] test_launch_tor_with_config_via_file (test.integ.process.TestProcess) ... FAIL
  [RUN_SOCKET] test_launch_tor_with_config_via_file (test.integ.process.TestProcess) ... FAIL
  [RUN_SCOOKIE] test_launch_tor_with_config_via_file (test.integ.process.TestProcess) ... FAIL
  [RUN_PTRACE] test_launch_tor_with_config_via_file (test.integ.process.TestProcess) ... FAIL

You can re-run just these tests with:

  ./run_tests.py --tor ../tor/src/or/tor --all --log notice --target RUN_ALL -v --test test.integ.process

comment:4 Changed 3 years ago by atagar

Interesting. Thanks catalyst, pushed a tweak. Mind giving this a try?

https://gitweb.torproject.org/stem.git/commit/?id=d378100

comment:5 Changed 2 years ago by atagar

Resolution: fixed
Status: needs_informationclosed
Note: See TracTickets for help on using tickets.