Establish policy for writing regression tests

Ideally, every non-trivial bug fix will have a regression test. Consider making and documenting a policy that requires that bug fixes will have accompanying regression tests, or some searchable justification (Trac keyword?) for why it's infeasible to add one. (Often this is a symptom that the underlying code is too complex and needs refactoring.)

This documentation should probably be somewhere under doc/HACKING.

changed milestone to %Tor: unspecified

added 034-removed-20180328 034-triage-20180328 component::core tor/tor docs milestone::Tor: unspecified policy priority::medium severity::normal status::new type::task labels

After discussion on IRC, it seems like [query:keywords=~test-waiver test-waiver] might be a reasonable keyword to use to mark tickets where we decide that it's infeasible to add a regression test for a bug.

Such a ticket should contain a written justification for why it's infeasible to add a regression test covering the bug. This can include excessive code complexity or the requirement for external test fixtures that need to be manually set up.

Being able to easily search Trac for such tickets will make it easier to locate places where we could improve the testability of our code.

Also patch reviewers should make sure the patch contains a regression test, and if one isn't present, make sure that there's a test-waiver keyword along with a comment explaining why that patch should be an exception from this rule. The reviewer should also review the justification and decide whether it's reasonable.

Do we also have a policy requiring any new features to have unit tests?

I think we should make it consistent across all tickets, and use the test-waiver keyword when we merge any code without tests. (Then we can search for "defect" or "regression" and "test-waiver" to find defect waivers, or regression waivers.)

Also, let's start trialing this for (some?) new code/tickets straight away, so we work out the bugs in the process.

Trac:
Type: defect to task

I think CodingStandards.md already requires unit tests for new functionality. Also, I think we should be really strict about allowing waivers for testing of new functionality. I guess there are some features where pre-existing technical debt makes it really hard to add tests for the new stuff, but in those cases I would argue strongly for paying down the technical debt prior to implementing the new features.

Trac:
Cc: N/A to mcs

Trac:
Keywords: N/A deleted, policy docs added

add the "docs" keyword to tickets that are documentation-only, and can enter Tor post-freeze.

Trac:
Keywords: policy docs deleted, docs, policy added

Trac:
Milestone: Tor: 0.3.2.x-final to Tor: 0.3.3.x-final

Moving a bunch of tickets from 033 to 034.

Trac:
Milestone: Tor: 0.3.3.x-final to Tor: 0.3.4.x-final

Trac:
Keywords: N/A deleted, 034-triage-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

Trac:
Keywords: N/A deleted, 034-removed-20180328 added

These tickets, tagged with 034-removed-*, are no longer in-scope for 0.3.4. We can reconsider any of them, if time permits.

Trac:
Milestone: Tor: 0.3.4.x-final to Tor: unspecified

moved to tpo/core/tor#22745 (closed)