Skip to content
Snippets Groups Projects
New look: Off

Stem hangs waiting for control port data

    • Closed Issue created by teor @teor 
        self.assertRaisesWith(stem.InvalidArguments, "DisableAllSwap, User cannot be changed while tor's running", controller.set_options, {'User': 'atagar', 'DisableAllSwap': '1'})

    or

      File "/home/travis/build/tlyu/tor/stem/test/integ/control/controller.py", line 793, in test_set_conf_when_immutable
    self.assertRaisesWith(stem.InvalidArguments, "DisableAllSwap cannot be changed while tor's running", controller.set_conf, 'DisableAllSwap', '1')

    We don't know why, because we don't have the tor logs or backtrace (#30234 (moved)).

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    • Trac
      Trac @tracbot ·

      Trac:
      Parent Ticket: #29437 (moved)

    • teor added actualpoints::0.5 component::archived/stem parent::29437 points::1 priority::medium resolution::user disappeared severity::normal sponsor::31-can status::closed tor-ci type::defect labels

      added actualpoints::0.5 component::archived/stem parent::29437 points::1 priority::medium resolution::user disappeared severity::normal sponsor::31-can status::closed tor-ci type::defect labels

    • A
      Damian Johnson @atagar ·

      Hi teor. Does this assertion failure consistently repro? If so then please tell me the tor commit id I should test against and I'll give this a whirl.

      Trac:
      Status: new to needs_information

    • T
      teor @teor ·
      Author

      Replying to atagar:

      Hi teor. Does this assertion failure consistently repro? If so then please tell me the tor commit id I should test against and I'll give this a whirl.

      No, it doesn't, unfortunately.

      So far, we've only been able to reproduce it in Travis, in about 1/10 builds: https://travis-ci.org/torproject/tor/jobs/521109187#L3568

      But I think it's probably a Tor bug, so I'll move it to that component.

      Trac:
      Status: needs_information to assigned
      Owner: atagar to N/A
      Component: Core Tor/Stem to Core Tor/Tor

    • T
      teor @teor ·
      Author

      The failure doesn't happen all the time, but when it does, it's in the same place:

          self.assertRaisesWith(stem.InvalidArguments, "DisableAllSwap, User cannot be changed while tor's running", controller.set_options, {'User': 'atagar', 'DisableAllSwap': '1'})

      https://travis-ci.org/torproject/tor/jobs/521109187#L3568https://travis-ci.org/torproject/tor/jobs/521109187#L3568

      I wonder if:

      • Tor successfully does DisableAllSwap?
      • Switching to User atagar succeeds? (It shouldn't: that user doesn't exist in Travis)

      I also wonder if there's some race condition between Tor and stem.

    • T
      teor @teor ·
      Author

      DisableAllSwap causes this error by itself:

        File "/home/travis/build/tlyu/tor/stem/test/integ/control/controller.py", line 793, in test_set_conf_when_immutable
    self.assertRaisesWith(stem.InvalidArguments, "DisableAllSwap cannot be changed while tor's running", controller.set_conf, 'DisableAllSwap', '1')

      https://travis-ci.org/tlyu/tor/jobs/521994718#L3565

      Trac:
      Actualpoints: N/A to 0.2
      Summary: Tor hangs when asked to change User and DisableAllSwap over the control port to Tor hangs when asked to change DisableAllSwap over the control port
      Description: ``` self.assertRaisesWith(stem.InvalidArguments, "DisableAllSwap, User cannot be changed while tor's running", controller.set_options, {'User': 'atagar', 'DisableAllSwap': '1'})

      
We don't know why, because we don't have the tor logs or backtrace (#30234). 

**to**

      self.assertRaisesWith(stem.InvalidArguments, "DisableAllSwap, User cannot be changed while tor's running", controller.set_options, {'User': 'atagar', 'DisableAllSwap': '1'})
      or

      File "/home/travis/build/tlyu/tor/stem/test/integ/control/controller.py", line 793, in test_set_conf_when_immutable self.assertRaisesWith(stem.InvalidArguments, "DisableAllSwap cannot be changed while tor's running", controller.set_conf, 'DisableAllSwap', '1')

      
We don't know why, because we don't have the tor logs or backtrace (#30234).
    • T
      teor @teor ·
      Author

      Nick might have some idea of what is going wrong here?

      Trac:
      Cc: N/A to nickm

    • C
      Taylor Yu @catalyst ·

      Trac:
      Cc: nickm to nickm, catalyst

    • T
      teor @teor ·
      Author

      Do you think we should change tor so it rejects DisableAllSwap over the control port?

    • T
      teor @teor ·
      Author

      Replying to teor:

      Do you think we should change tor so it rejects DisableAllSwap over the control port?

      Tor already rejects DisableAllSwap over the control port.

    • Nick Mathewson
      Nick Mathewson @nickm ·

      To be specific, Tor rejects all changes to DisableAllSwap while it is running.

    • T
      teor @teor ·
      Author

      Trac:
      Actualpoints: 0.2 to 0.3

    • T
      teor @teor ·
      Author

      Setting as sponsor 31 can, because we use these jobs to make sure our refactoring works.

      Trac:
      Sponsor: N/A to Sponsor31-can

    • T
      teor @teor ·
      Author

      It looks like stem hangs waiting for a controller reply, but stem never actually sends a command to trigger the reply:

      05/30/2019 10:20:56 [TRACE] Sent to tor: Blarg!
05/30/2019 10:20:56 [TRACE] Received from tor: 650 STATUS_SERVER NOTICE EXTERNAL_ADDRESS ADDRESS=35.202.145.110 METHOD=DIRSERV
05/30/2019 10:20:56 [TRACE] Received from tor: 514 Authentication required.
05/30/2019 10:20:56 [TRACE] Sent to tor: blarg
05/30/2019 10:20:56 [INFO] Error while receiving a control message (SocketClosed): empty socket content
05/30/2019 10:20:56 [TRACE] Received from tor: 650 STATUS_SERVER NOTICE EXTERNAL_ADDRESS ADDRESS=35.184.96.71 METHOD=DIRSERV
  control.controller... 

================================================================================
Signal SIGUSR1 received by thread MainThread in process 18617
--------------------------------------------------------------------------------
MainThread thread stacktrace
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 456, in <module>
    main()
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 297, in main
    run_result = _run_test(args, test_class, output_filters)
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 437, in _run_test
    run_result = stem.util.test_tools.TimedTestRunner(test_results, verbosity = 2).run(suite)
  File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 308, in run
    return super(TimedTestRunner, self).run(test)
  File "/usr/lib/python3.4/unittest/runner.py", line 168, in run
    test(result)
  File "/usr/lib/python3.4/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.4/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.4/unittest/case.py", line 625, in __call__
    return self.run(*args, **kwds)
  File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 258, in run
    result = super(type(self), self).run(result)
  File "/usr/lib/python3.4/unittest/case.py", line 577, in run
    testMethod()
  File "/home/travis/build/torproject/tor/stem/test/require.py", line 57, in wrapped
    return func(self, *args, **kwargs)
  File "/home/travis/build/torproject/tor/stem/test/integ/control/controller.py", line 782, in test_set_conf
    ), reset = True)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 2541, in set_options
    response = self.msg(query)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 661, in msg
    response = self._reply_queue.get()
  File "/usr/lib/python3.4/queue.py", line 167, in get
    self.not_empty.wait()
  File "/usr/lib/python3.4/threading.py", line 290, in wait
    waiter.acquire()
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 91, in log_traceback
    for thread_name, stacktrace in test.output.thread_stacktraces().items():
  File "/home/travis/build/torproject/tor/stem/test/output.py", line 110, in thread_stacktraces
    stacktraces[thread.name] = ''.join(traceback.format_stack(frame))
--------------------------------------------------------------------------------
Tor listener thread stacktrace
  File "/usr/lib/python3.4/threading.py", line 888, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 946, in _reader_loop
    control_message = self._socket.recv()
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 474, in recv
    return self._recv(lambda s, sf: recv_message(sf))
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 274, in _recv
    return handler(my_socket, my_socket_file)
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 474, in <lambda>
    return self._recv(lambda s, sf: recv_message(sf))
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 676, in recv_message
    line = control_file.readline()
  File "/usr/lib/python3.4/socket.py", line 374, in readinto
    return self._sock.recv_into(b)
--------------------------------------------------------------------------------
Event notifier thread stacktrace
  File "/usr/lib/python3.4/threading.py", line 888, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 991, in _event_loop
    self._event_notice.wait(0.05)
  File "/usr/lib/python3.4/threading.py", line 553, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.4/threading.py", line 294, in wait
    gotit = waiter.acquire(True, timeout)
================================================================================

      https://travis-ci.org/torproject/tor/jobs/539146230#L4372

      The tor logs aren't much help, because they are from 10 minutes later than the hang.

      Trac:
      Summary: Tor hangs when asked to change DisableAllSwap over the control port to Stem hangs waiting for control port data
      Milestone: Tor: 0.2.9.x-final to N/A
      Version: Tor: 0.2.4.8-alpha to N/A
      Component: Core Tor/Tor to Core Tor/Stem
      Actualpoints: 0.3 to 0.5

    • A
      Damian Johnson @atagar ·

      Hi teor, is this still occurring? If so, could I get a fresh instance with the '--log-file' output we recently added with trace logs?

      Trac:
      Status: assigned to needs_information

    • T
      teor @teor ·
      Author

      Hopefully one of the pull requests in #30591 (moved) will fail, if they do, I'll paste the logs here: https://trac.torproject.org/projects/tor/ticket/30591#comment:20

      (We don't want the logs from the deliberate failure pull request.)

    • T
      teor @teor ·
      Author

      Hmm, it seems that something goes wrong a few lines in, where stem says:

      06/14/2019 03:10:42 [TRACE] Sent to tor: GETINFO version

      But it never gets a response back from tor.

      Here's the latest output:

      06/14/2019 03:10:42 [TRACE] Sent to tor: GETINFO version
06/14/2019 03:10:42 [TRACE] Received from tor: 250-version=0.4.2.0-alpha-dev (git-aadad2c643a34ea5)\n250 OK
06/14/2019 03:10:42 [TRACE] Sent to tor: GETINFO blarg
06/14/2019 03:10:42 [TRACE] Received from tor: 552 Unrecognized key "blarg"
06/14/2019 03:10:42 [TRACE] Sent to tor: blarg
06/14/2019 03:10:42 [TRACE] Received from tor: 510 Unrecognized command "blarg"
06/14/2019 03:10:42 [TRACE] Sent to tor: GETINFO version
06/14/2019 03:10:42 [TRACE] Received from tor: 650 STATUS_SERVER NOTICE EXTERNAL_ADDRESS ADDRESS=35.188.1.99 METHOD=DIRSERV
06/14/2019 03:10:42 [TRACE] Received from tor: 650 STATUS_SERVER NOTICE EXTERNAL_ADDRESS ADDRESS=35.224.112.202 METHOD=DIRSERV
06/14/2019 03:10:42 [TRACE] Received from tor: 650 STATUS_SERVER NOTICE EXTERNAL_ADDRESS ADDRESS=35.192.85.2 METHOD=DIRSERV
06/14/2019 03:10:42 [TRACE] Sent to tor: Blarg!
06/14/2019 03:10:42 [TRACE] Received from tor: 650 STATUS_SERVER NOTICE EXTERNAL_ADDRESS ADDRESS=35.184.226.236 METHOD=DIRSERV
06/14/2019 03:10:42 [TRACE] Received from tor: 650 STATUS_SERVER NOTICE EXTERNAL_ADDRESS ADDRESS=104.154.120.187 METHOD=DIRSERV
...
6/14/2019 03:10:46 [TRACE] Received from tor: 650 STATUS_SERVER NOTICE EXTERNAL_ADDRESS ADDRESS=35.193.7.13 METHOD=DIRSERV
06/14/2019 03:10:46 [TRACE] Received from tor: 650 STATUS_SERVER NOTICE EXTERNAL_ADDRESS ADDRESS=35.184.96.71 METHOD=DIRSERV
06/14/2019 03:10:46 [TRACE] Received from tor: 514 Authentication required.
06/14/2019 03:10:46 [TRACE] Sent to tor: blarg
timelimit: sending warning signal 10
================================================================================
Signal SIGUSR1 received by thread MainThread in process 20776
--------------------------------------------------------------------------------
Tor listener thread stacktrace
  File "/usr/lib/python3.4/threading.py", line 888, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 946, in _reader_loop
    control_message = self._socket.recv()
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 474, in recv
    return self._recv(lambda s, sf: recv_message(sf))
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 274, in _recv
    return handler(my_socket, my_socket_file)
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 474, in <lambda>
    return self._recv(lambda s, sf: recv_message(sf))
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 676, in recv_message
    line = control_file.readline()
  File "/usr/lib/python3.4/socket.py", line 374, in readinto
    return self._sock.recv_into(b)
--------------------------------------------------------------------------------
Event notifier thread stacktrace
  File "/usr/lib/python3.4/threading.py", line 888, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 991, in _event_loop
    self._event_notice.wait(0.05)
  File "/usr/lib/python3.4/threading.py", line 553, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.4/threading.py", line 294, in wait
    gotit = waiter.acquire(True, timeout)
--------------------------------------------------------------------------------
MainThread thread stacktrace
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 479, in <module>
    main()
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 311, in main
    run_result = _run_test(args, test_class, output_filters)
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 457, in _run_test
    run_result = stem.util.test_tools.TimedTestRunner(test_results, verbosity = 2).run(suite)
  File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 308, in run
    return super(TimedTestRunner, self).run(test)
  File "/usr/lib/python3.4/unittest/runner.py", line 168, in run
    test(result)
  File "/usr/lib/python3.4/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.4/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.4/unittest/case.py", line 625, in __call__
    return self.run(*args, **kwds)
  File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 258, in run
    result = super(type(self), self).run(result)
  File "/usr/lib/python3.4/unittest/case.py", line 577, in run
    testMethod()
  File "/home/travis/build/torproject/tor/stem/test/require.py", line 57, in wrapped
    return func(self, *args, **kwargs)
  File "/home/travis/build/torproject/tor/stem/test/integ/control/controller.py", line 732, in test_set_conf
    controller.set_conf('ExitPolicy', exit_policy)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 2471, in set_conf
    self.set_options({param: value}, False)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 2541, in set_options
    response = self.msg(query)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 661, in msg
    response = self._reply_queue.get()
  File "/usr/lib/python3.4/queue.py", line 167, in get
    self.not_empty.wait()
  File "/usr/lib/python3.4/threading.py", line 290, in wait
    waiter.acquire()
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 92, in log_traceback
    for thread_name, stacktrace in test.output.thread_stacktraces().items():
  File "/home/travis/build/torproject/tor/stem/test/output.py", line 110, in thread_stacktraces
    stacktraces[thread.name] = ''.join(traceback.format_stack(frame))
================================================================================
================================================================================
Signal SIGUSR1 received by thread MainThread in process 20813
--------------------------------------------------------------------------------
MainThread thread stacktrace
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 479, in <module>
    main()
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 285, in main
    test_classes[0].run_tests(async_args)
  File "/home/travis/build/torproject/tor/stem/test/integ/installation.py", line 64, in run_tests
    stem.util.test_tools.ASYNC_TESTS['test.integ.installation.test_sdist'].run(test_install.pid())
  File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 201, in run
    self._process.start()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.4/multiprocessing/context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.4/multiprocessing/context.py", line 267, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.4/multiprocessing/popen_fork.py", line 21, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.4/multiprocessing/popen_fork.py", line 77, in _launch
    code = process_obj._bootstrap()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
    self.run()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 169, in _wrapper
    runner(*args) if args else runner()
  File "/home/travis/build/torproject/tor/stem/test/integ/installation.py", line 108, in test_sdist
    time.sleep(0.1)  # we need to run these tests serially
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 92, in log_traceback
    for thread_name, stacktrace in test.output.thread_stacktraces().items():
  File "/home/travis/build/torproject/tor/stem/test/output.py", line 110, in thread_stacktraces
    stacktraces[thread.name] = ''.join(traceback.format_stack(frame))
================================================================================
timelimit: sending kill signal 6
================================================================================
Signal SIGABRT received by thread MainThread in process 20776
--------------------------------------------------------------------------------
Tor listener thread stacktrace
  File "/usr/lib/python3.4/threading.py", line 888, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 946, in _reader_loop
    control_message = self._socket.recv()
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 474, in recv
    return self._recv(lambda s, sf: recv_message(sf))
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 274, in _recv
    return handler(my_socket, my_socket_file)
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 474, in <lambda>
    return self._recv(lambda s, sf: recv_message(sf))
  File "/home/travis/build/torproject/tor/stem/stem/socket.py", line 676, in recv_message
    line = control_file.readline()
  File "/usr/lib/python3.4/socket.py", line 374, in readinto
    return self._sock.recv_into(b)
--------------------------------------------------------------------------------
Event notifier thread stacktrace
  File "/usr/lib/python3.4/threading.py", line 888, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 991, in _event_loop
    self._event_notice.wait(0.05)
  File "/usr/lib/python3.4/threading.py", line 553, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.4/threading.py", line 294, in wait
    gotit = waiter.acquire(True, timeout)
--------------------------------------------------------------------------------
MainThread thread stacktrace
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 479, in <module>
    main()
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 311, in main
    run_result = _run_test(args, test_class, output_filters)
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 457, in _run_test
    run_result = stem.util.test_tools.TimedTestRunner(test_results, verbosity = 2).run(suite)
  File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 308, in run
    return super(TimedTestRunner, self).run(test)
  File "/usr/lib/python3.4/unittest/runner.py", line 168, in run
    test(result)
  File "/usr/lib/python3.4/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.4/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.4/unittest/case.py", line 625, in __call__
    return self.run(*args, **kwds)
  File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 258, in run
    result = super(type(self), self).run(result)
  File "/usr/lib/python3.4/unittest/case.py", line 577, in run
    testMethod()
  File "/home/travis/build/torproject/tor/stem/test/require.py", line 57, in wrapped
    return func(self, *args, **kwargs)
  File "/home/travis/build/torproject/tor/stem/test/integ/control/controller.py", line 732, in test_set_conf
    controller.set_conf('ExitPolicy', exit_policy)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 2471, in set_conf
    self.set_options({param: value}, False)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 2541, in set_options
    response = self.msg(query)
  File "/home/travis/build/torproject/tor/stem/stem/control.py", line 661, in msg
    response = self._reply_queue.get()
  File "/usr/lib/python3.4/queue.py", line 167, in get
    self.not_empty.wait()
  File "/usr/lib/python3.4/threading.py", line 290, in wait
    waiter.acquire()
  File "/home/travis/build/torproject/tor/stem/run_tests.py", line 92, in log_traceback
    for thread_name, stacktrace in test.output.thread_stacktraces().items():
  File "/home/travis/build/torproject/tor/stem/test/output.py", line 110, in thread_stacktraces
    stacktraces[thread.name] = ''.join(traceback.format_stack(frame))
================================================================================
  control.controller...

      https://travis-ci.org/torproject/tor/jobs/545543377#L4049

      Trac:
      Status: needs_information to new

    • A
      Damian Johnson @atagar ·

      Hi teor, yup - agreed. That's my read of those logs as well. Unfortunately at this point I suspect the next step is to compile additional logging into tor to see what's happening to that request.

      I'd also suggest adjusting the '--test' argument to '--test control.controller.test_set_conf' and see if these hangs continue to repro. If they do this will considerably cut down the haystack we need to look within.

      Anything I can be of assistance with on my end?

    • T
      teor @teor ·
      Author

      Replying to atagar:

      Hi teor, yup - agreed. That's my read of those logs as well. Unfortunately at this point I suspect the next step is to compile additional logging into tor to see what's happening to that request.

      I opened #30901 (moved) for this change to tor.

      I'd also suggest adjusting the '--test' argument to '--test control.controller.test_set_conf' and see if these hangs continue to repro. If they do this will considerably cut down the haystack we need to look within.

      I'm not sure if we want to do this on master, because we need these tests. But we could do it on another branch, if the extra tor logging doesn't help.

      Anything I can be of assistance with on my end?

      No, we need more info from tor. And then we'll know whether to fix stem or tor.

      Trac:
      Status: new to needs_information

    • T
      teor @teor ·
      Author

      Trac:
      Keywords: tor-ci-fail-sometimes deleted, tor-ci added

    • A
      Damian Johnson @atagar ·

      Trac:
      Resolution: N/A to user disappeared
      Status: needs_information to closed

    • Trac closed

      closed

    • Trac changed time estimate to 8h

      changed time estimate to 8h

    • Trac added 4h of time spent

      added 4h of time spent

    • teor mentioned in issue #30243 (closed)

      mentioned in issue #30243 (closed)

    • teor mentioned in issue #30901 (moved)

      mentioned in issue #30901 (moved)

    • Trac mentioned in issue #29437 (moved)

      mentioned in issue #29437 (moved)

    • Trac mentioned in issue tpo/core/tor#30901 (closed)

      mentioned in issue tpo/core/tor#30901 (closed)

    Please register or sign in to reply