Recursion limit problem in TaskManager
-
- View options
This issue was automatically migrated from github issue https://github.com/TheTorProject/ooni-probe/issues/296.
The task manager is currently designed to recursively call _fillSlots that will then call _run that then again will call _fillSlots on success or failure. This means that when there are a lot of tasks failing very quickly it is very likely that the default python recursion limit will be overcome (1000).
To reproduce this bug you can try and run a test with a long invalid input for example http_requests:
ooniprobe blocking/http_requests -f data/complete.deck
Note that the fact that this test fails is correct, however it fails in a surprising manner:
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
File "/ooni-probe/ooni/managers.py", line 153, in _failed
super(LinkedTaskManager, self)._failed(result, task)
File "/ooni-probe/ooni/managers.py", line 44, in _failed
task.done.errback(failure)
File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/internet/defer.py", line 423, in errback
self._startRunCallbacks(fail)
File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/ooni-probe/ooni/director.py", line 188, in measurementFailed
log.msg("Failed doing measurement: %s" % measurement)
File "/ooni-probe/ooni/utils/log.py", line 62, in msg
print "%s" % msg
File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/python/log.py", line 505, in write
msg(message, printed=1, isError=self.isError)
File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/python/threadable.py", line 53, in sync
return function(self, *args, **kwargs)
File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/python/log.py", line 185, in msg
actualEventDict = (context.get(ILogContext) or {}).copy()
File "/.virtualenvs/ooni-probe/lib/python2.7/site-packages/twisted/python/context.py", line 121, in getContext
return self.currentContext().getContext(key, default)
exceptions.RuntimeError: maximum recursion depth exceededI think this bug is perhaps a good opportunity to discuss some possible refactoring of the task scheduler related code. It may be a good idea to draw some inspiration from: https://github.com/terrycojones/txrdq
Attributes
Assignees
Labels
Milestone
Dates
Start: None
Due: None