Merge original WTF-PAD branch

This is a roadmap item for merging Mike's original WTF-PAD branch upstream. It's the first step to completing the WTF-PAD project.

changed milestone to %Tor: 0.4.0.x-final

added component::core tor/tor milestone::Tor: 0.4.0.x-final owner::mikeperry padding parent::28631 priority::medium resolution::fixed reviewer::asn severity::normal sponsor::2 status::closed tor-cell tor-relay type::enhancement wtf-pad labels

Currently in needs_revision as Mike fixes the branch after the Mexico review.

Trac:
Status: new to needs_revision

https://github.com/torproject/tor/pull/439

I rebased everything onto 0.3.6 and organized the commits in a similar order to how we reviewed in Mexico. Strongly recommend reviewing it in commit order.

WARNING: The github PR UI is showing the commits in timestamp or some other random ordering. Reviewing in commit hash order is best. I have no idea how to change the github ordering. :/

Trac:
Status: needs_revision to needs_review

Trac:
Keywords: N/A deleted, padding, tor-relay, tor-cell added
Owner: N/A to mikeperry
Type: defect to enhancement
Reviewer: N/A to asn
Status: needs_review to assigned

I started doing the review, and I put some general comments in github. I'm far from done.

I also started testing it with chutney and it seems to work pretty nicely. I have some local changes that add some extra logs, but I'm gonna push them after I do more testing.

Mike, you have any tips for more effective testing of this feature?

Trac:
Status: assigned to needs_review

Ok, fresh squashed PR for nickm: https://github.com/torproject/tor/pull/461

Nickm -- the branch's commits are in recommended review order. Note that github reorders them by timestamp in its UI, which is not the right order to look at it. The right order is in the git hash-parent history.

Here's the latest proposal, updated to cover this branch. Section 3 and onward is the relevant bit. Section 2 is about netflow negotiation: https://github.com/asn-d6/torspec/blob/circuitpadding-proposal-updates/proposals/254-padding-negotiation.txt#L78

I would read that first, then the PR in commit order (starting with the header).

Trac:
Cc: N/A to nickm

I have continued the review of this branch. I think I have reviewed about 75% of the actual code.

The main parts I haven't reviewed are the various token removal logic functions which are quite complicated.

Tor 0.3.6.x has been renamed to 0.4.0.x.

Trac:
Milestone: Tor: 0.3.6.x-final to Tor: 0.4.0.x-final

The last few days I've been focusing on using Riastradh's better probability distribution formulas as part of circuitpadding. Current work can be found in https://github.com/torproject/tor/pull/533.

Next steps in this area:

• Fix the remaining compiler warnings.
• ~~Figure out if we want to keep the stochastic tests or not. Need to understand more about their failure rates, and also they are quite slow. Maybe I can put them into the slow test category which AFAIK is not run by the CI. We still get to run the numerical tests. ~~Moved stochastic warnings to test-slow. I will open a ticket to figure out what to do about them.
• ~Address various XXXs (e.g. geometric distr needs to conform to the API as well). ~~Resolved a few of them, the rest we can open tickets for.
• Do further documentation.
• ~~Ensure that the API consumer circpad_distribution_sample() works well in terms of using the prob distributions correctly. ~~Added a basic test for uniform distribution circuit padding delay sampling.
• Fix coding style of prob distr code because it's different from Tor's and make check-spaces is freaking out.
• The new files were ported to Tor quite hastily. Ensure that they are proper wrt header file etc. Also fix the copyrights etc.

All remaining tasks above have been resolved. I pushed the code in !adaptive_padding-prob-distr-tests and pending review from Riastradh. After review has been done, I will squash into something presentable along with the rest of the #28142 (moved) branch. I will also open some tickets for leftover optional work in this area.

Trac:
float-warn.patch

patch to commit a9f511ddd5bf5e57c159a21b3bbe0203412bc04a on asn-d6/adaptive_padding-prob-distr-tests to work around -Wfloat-equal, -Wfloat-conversion

I attached a patch to address all the -Wfloat-equal and -Wfloat-conversion warnings.

I didn't address -Wbad-function-casts; if you really want to appease the warning at the expense of code clarity, you can just replace each case of

integer-type y = (integer-type)floor(f(x));

by the clumsier

double yd = floor(f(x));
integer-type y = (integer-type)yd;

FYI: a part of this work, namely the uniform ^^[0,1^^] sampler used throughout, is echoed in #23061 (moved). Just letting you know in case you want to avoid the duplication sooner rather than later.

Trac:
patches.mbox

miscellaneous patches on top of de7b56e0d0ecf44f291c6159d2ff30defafea616

I attached some patches to:

• reindent the code according to my understanding of tor's indentation style, since it had been getting a little inconsistent;
• address one of the questions in https://github.com/torproject/tor/pull/533, and re-enable some related tests that were mistakenly disabled;
• use separate functions for floor and ceiling instead of one with a boolean parameter; and
• convert assert to tor_assert.

With these patches, the only remaining complaints from make check are:

• lines that are too long because they contain URLs that are too long; and
• includes that are not allowed because of missing .may_include or something.

I'm not sure what the best way to address these is -- break the URLs, or add an override; update .may_include, or move files around -- so I'll leave that up to you folks to decide.

Trac:
epsilon-note.patch

minor clarification of what epsilon means here and in general