Opened 5 years ago

Closed 16 months ago

#11978 closed defect (wontfix)

Recursion limit problem in TaskManager

Reported by: cypherpunks Owned by: hellais
Priority: Medium Milestone:
Component: Archived/Ooni Version:
Severity: Normal Keywords: archived-closed-2018-07-04
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

This issue was automatically migrated from github issue https://github.com/TheTorProject/ooni-probe/issues/296.

The task manager is currently designed to recursively call _fillSlots that will then call _run that then again will call _fillSlots on success or failure. This means that when there are a lot of tasks failing very quickly it is very likely that the default python recursion limit will be overcome (1000).

To reproduce this bug you can try and run a test with a long invalid input for example http_requests:

ooniprobe blocking/http_requests -f data/complete.deck

Note that the fact that this test fails is correct, however it fails in a surprising manner:

`
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):

File "/ooni-probe/ooni/managers.py", line 153, in _failed

super(LinkedTaskManager, self)._failed(result, task)

File "/ooni-probe/ooni/managers.py", line 44, in _failed

task.done.errback(failure)

File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/internet/defer.py", line 423, in errback

self._startRunCallbacks(fail)

File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks

self._runCallbacks()

--- <exception caught here> ---

File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks

current.result = callback(current.result, *args, kw)

File "/ooni-probe/ooni/director.py", line 188, in measurementFailed

log.msg("Failed doing measurement: %s" % measurement)

File "/ooni-probe/ooni/utils/log.py", line 62, in msg

print "%s" % msg

File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/python/log.py", line 505, in write

msg(message, printed=1, isError=self.isError)

File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/python/threadable.py", line 53, in sync

return function(self, *args, kwargs)

File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/python/log.py", line 185, in msg

actualEventDict = (context.get(ILogContext) or {}).copy()

File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/python/context.py", line 121, in getContext

return self.currentContext().getContext(key, default)

exceptions.RuntimeError: maximum recursion depth exceeded
`

I think this bug is perhaps a good opportunity to discuss some possible refactoring of the task scheduler related code. It may be a good idea to draw some inspiration from: https://github.com/terrycojones/txrdq

Child Tickets

Change History (2)

comment:1 Changed 23 months ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

comment:2 Changed 16 months ago by teor

Keywords: archived-closed-2018-07-04 added
Resolution: wontfix
Status: newclosed

Close all tickets in archived components

Note: See TracTickets for help on using tickets.