Opened 3 years ago

Last modified 11 months ago

#16710 new enhancement

Graph or plot test coverage over time?

Reported by: arma Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: testing, jenkins, SponsorS-deferred
Cc: weasel Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

In reading our Sponsor S quarterly report draft, I found a section about how our unit test coverage is up to 37%, and our stem+chutney test coverage is up to 62%.

Great. What were they before, and what is their rate of change?

Is this the sort of thing where we have the numbers somewhere? Is it easy to automate over time?

Ideally we should even arrange things so the graphs can help *us* in some way too -- first by e.g. giving somebody a sense of satisfaction when a big branch gets merged and the coverage line shoots up a lot, but then next by helping us notice when some *other* big branch gets merged that suddenly undermines coverage in a way we didn't expect? Let's not do this to satisfy a funder -- let's figure out a way where tracking and visualizing our progress helps us make progress better.

Child Tickets

Change History (13)

comment:1 Changed 3 years ago by nickm

Keywords: testing added

comment:2 Changed 3 years ago by nickm

This could use a quick script or two. Specifically, let's have a tool that generates a directory full of lcov html, possibly dual-colored based on "make check" and "make check-full" output. And let's have it also generate overall coverage, stick that into a database, and graph it over time.

comment:3 Changed 3 years ago by rl1987

I've seen this being done with Jenkins. Apparently you need to install some Jenkins plugins and XML'ify the coverage data to make it readable by e.g. Cobertura plugin. Maybe weasel knows more and is able to set it up on jenkins.tpo?

Last edited 3 years ago by rl1987 (previous) (diff)

comment:4 Changed 3 years ago by nickm

interesting. If weasel is interested, that would be a convenient way to do it.

comment:5 Changed 3 years ago by nickm

Cc: weasel added
Keywords: jenkins added

comment:6 Changed 3 years ago by nickm

Keywords: SponsorS removed
Sponsor: SponsorS

Bulk-replace SponsorS keyword with SponsorS sponsor field in Tor component.

comment:7 Changed 3 years ago by teor

Milestone: Tor: 0.2.???

comment:8 Changed 3 years ago by isabela

Sponsor: SponsorSSponsorS-can

comment:9 Changed 2 years ago by nickm

Keywords: SponsorS-deferred added
Sponsor: SponsorS-can

Remove the SponsorS status from these items, which we already decided to defer from 0.2.9. add the SponsorS-deferred tag instead in case we ever want to remember which ones these were.

comment:10 Changed 2 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:11 Changed 22 months ago by nickm

Keywords: tor-03-unspecified-201612 added
Milestone: Tor: 0.3.???Tor: unspecified

Finally admitting that 0.3.??? was a euphemism for Tor: unspecified all along.

comment:12 Changed 17 months ago by nickm

Keywords: tor-03-unspecified-201612 removed

Remove an old triaging keyword.

comment:13 Changed 11 months ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

Note: See TracTickets for help on using tickets.