Opened 3 months ago

Closed 12 days ago

#28865 closed defect (fixed)

sbws keeps the number of AsyncResults less than the number of threads

Reported by: teor Owned by: juga
Priority: Medium Milestone: sbws: 1.1.x-final
Component: Core Tor/sbws Version: sbws: 1.0.2
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: #28663 Points: 1
Reviewer: Sponsor:

Description

This ticket is a follow-up to #28864.

multiprocessing.Pool can accept any number of queued AsyncResults:
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool

But sbws waits for 5 seconds when there are max_pending_results queued AsyncResults:
https://github.com/torproject/sbws/blob/ee64d76df54ceb3a3c9e1e2a797fd70d68bb0035/sbws/core/scanner.py#L359-L361

This sbws code is unnecessary, because Pool manages its own queue of AsyncResults.

There are two different ways that this code blocks execution:

  • when a result finishes, the time.sleep(5) call blocks the thread from getting a new AsyncResult for up to 5 seconds
  • if max_pending_results AsyncResults ever block, the process hangs

Child Tickets

Change History (5)

comment:1 in reply to:  description ; Changed 3 months ago by juga

Replying to teor:
[...]

There are two different ways that this code blocks execution:

  • when a result finishes, the time.sleep(5) call blocks the thread from getting a new AsyncResult for up to 5 seconds

the main thread blocks, but not the workers.

  • if max_pending_results AsyncResults ever block, the process hangs

and that would mean there's a deadlock or a bug somewhere else. Probably the cause (or at least the main one) is #28897.
What i would to detect future bugs, is give a maximum timeout to both while loops, then print a backtrace and ask the user to open a ticket with the backtrace.

comment:2 in reply to:  1 Changed 2 weeks ago by juga

Cc: juga teor removed
Milestone: sbws: 1.0.x-finalsbws: 1.1.x-final
Owner: set to juga
Points: 1
Status: newassigned

Replying to juga:

What i would to detect future bugs, is give a maximum timeout to both while loops, then print a backtrace and ask the user to open a ticket with the backtrace.

This was implemented in #28932, so this can wait to milestone 1.1.
https://github.com/torproject/sbws/pull/340

comment:3 Changed 2 weeks ago by juga

Status: assignedneeds_review

comment:4 Changed 2 weeks ago by juga

Status: needs_reviewneeds_revision

I realize this can not work with the timeout of 10 secs now.

comment:5 Changed 12 days ago by juga

Resolution: fixed
Status: needs_revisionclosed

Implemented in #28864

Note: See TracTickets for help on using tickets.