Opened 4 years ago

Last modified 3 years ago

#20617 new enhancement

Write a unit test abstraction for tests that need to be repeated multiple times to prove failure

Reported by: teor Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: test, unit-test, backend
Cc: Actual Points:
Parent ID: Points: 2
Reviewer: Sponsor:

Description (last modified by teor)

Some unit tests succeed at random with a certain probability, so we need to repeat them a number of times until the failure rate is acceptable.

Typically, we make sure the failure rate is less than the hardware bit error rate.

But we could standardise this, by taking the following inputs:

  • test function with success/fail return value,
  • the probability of success regardless of whether the code actually works,
  • the desired overall probability of failure by chance (for example, the bit error rate).

The abstraction would then repeat the test enough times to make sure the failure rate is less than the bit error rate.

We might also want to check that the distribution of returned values is uniform, but I think that's a separate (and more complex) ticket.

Child Tickets

Change History (3)

comment:1 Changed 4 years ago by teor

For current examples of code like this, see test_crypto_rng_range and crypto_rand_check_failure_mode_zero.

comment:2 Changed 4 years ago by teor

Description: modified (diff)

comment:3 Changed 3 years ago by nickm

Keywords: unit-test backend added
Note: See TracTickets for help on using tickets.