Opened 8 months ago

Closed 7 months ago

#25908 closed project (fixed)

make test suite coverage more deterministic

Reported by: catalyst Owned by:
Priority: Medium Milestone: Tor: 0.3.4.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: tor-ci, tor-tests-coverage, tor-tests-unit
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor: Sponsor3-can

Description

One thing that appears to be happening in the coveralls integration is some small blocks of code seem to be changing coverage without obvious corresponding changes in source code or tests. This can cause false positive indications of decreased coverage reported in things like pull requests. We should eliminate as much of this indeterminacy as possible.

One example is https://coveralls.io/builds/16676967/source?filename=src%2For%2Fcircuitstats.c#L879

This ticket should probably have child tickets for specific instances of this problem.

Child Tickets

TicketStatusOwnerSummaryComponent
#25993closednickmImprove deliberate test coverage for addressmap_get_virtual_address()Core Tor/Tor
#25994closednickmtest_channel: keep time constant when running channel/outbound_cellCore Tor/Tor
#25995closednickmUse a deterministic PRNG in test_circuit_timeout() for predictable coverage.Core Tor/Tor
#25996closednickmProduce consistent coverage from test_client_pick_intro()Core Tor/Tor
#25997closednickmSolve nondeterminism in testing of hs_get_responsible_hsdirsCore Tor/Tor
#26008closednickmMake workqueue cancel code always get covered by tests.Core Tor/Tor
#26014closednickmFix two cases of nondeterminism in voting_schedule.c coverageCore Tor/Tor

Attachments (3)

tor-cov-diffs.tar.gz (14.9 KB) - added by nickm 8 months ago.
Differences between different coverage runs (as of 8664f76dd202ec6514c06207c37fe9ae340f2f0f)
tor-cov-diffs.tar.2.gz (6.7 KB) - added by nickm 8 months ago.
tor-cov-diffs.tar.3.gz (3.3 KB) - added by nickm 8 months ago.

Download all attachments as: .zip

Change History (10)

Changed 8 months ago by nickm

Attachment: tor-cov-diffs.tar.gz added

Differences between different coverage runs (as of 8664f76dd202ec6514c06207c37fe9ae340f2f0f)

comment:1 Changed 8 months ago by nickm

I spent a couple of hours running "make check" on tor-0.3.4.0-alpha-dev-590-g8664f76dd202ec with coverage enabled under GCC, and taking the output of each run. I then picked the first run as a reference run, and computed the diff from that run's coverage to each of the others. Finally, I took the unique diffs.

Out of 203 runs, I got 164 65 distinct diffs from the reference run. (Probably, some are combinations of others). I'm attaching a tarball full of those diffs for further analysis.

Last edited 8 months ago by nickm (previous) (diff)

Changed 8 months ago by nickm

Attachment: tor-cov-diffs.tar.2.gz added

comment:2 Changed 8 months ago by nickm

Please see the more recent version of the file -- I post-processed the previous one incorrectly.

comment:3 Changed 8 months ago by catalyst

Thanks! It looks like the affected files are fewer than I feared:

$ sed -ne 's/^--- \(.*\.gcov\).*/\1/p' *|sort -u
a/addressmap.c.gcov
a/circuitmux_ewma.c.gcov
a/circuitstats.c.gcov
a/hs_client.c.gcov
a/hs_common.c.gcov

Also, skimming the diffs still shows a fair amount of duplication, so fixing these might not be too difficult.

Changed 8 months ago by nickm

Attachment: tor-cov-diffs.tar.3.gz added

comment:4 Changed 8 months ago by catalyst

I set the coverage decrease threshold on coveralls.io to 0.1%, which is hopefully enough to quiet things a bit for now. We should still work on this ticket though.

comment:5 Changed 8 months ago by nickm

Milestone: Tor: 0.3.5.x-finalTor: 0.3.4.x-final

comment:6 Changed 7 months ago by nickm

I ran the tests another 450 times, with all of the deterministic-coverage fixes so far applied to it. I found only one case in which the coverage was different, and opened #26008 to track it.

To reproduce at home, build Tor with coverage enabled; create a "coverage-raw" directory, and then run:

while true; do 
  make reset-gcov
  CD=coverage-raw/coverage-`date +%s`
  mkdir -p $CD
  make -j5 check
  ./scripts/test/coverage $CD
done

Then, once you have enough runs in coverage-raw, use the cov-diff tool to see if they have any coverage differences.

comment:7 Changed 7 months ago by nickm

Resolution: fixed
Status: newclosed

I spent the last 24 hours running the tests in a loop again -- this time, I got no differences in line coverage, though I did find #26101 and #26105. Calling this "done for 0.3.4": we should open up more tickets for other cases as we find them.

Note: See TracTickets for help on using tickets.