prop278: Add support for LZMA2 and/or Zstandard

Add support for the compression schemes needed to implement prop#278.

See: http://facebook.github.io/zstd/ and http://7-zip.org/sdk.html for the respective libraries.

changed milestone to %Tor: 0.3.1.x-final

added TorCoreTeam201703 component::core tor/tor milestone::Tor: 0.3.1.x-final owner::ahf points::2 priority::medium prop278 resolution::implemented severity::normal sponsor::4 status::closed type::task labels

Trac:
Owner: N/A to ahf
Status: new to accepted

Trac:
Type: defect to task

Add prop278 keyword.

Trac:
Keywords: N/A deleted, prop278 added

Starting to track the cleaned up commits in: https://gitlab.com/ahf/tor/merge_requests/2

Currently LZMA support is available - Zstandard coming up.

Note that this code depends upon the code that is currently being reviewed as part of #21663 (moved).

Ready for review: https://gitlab.com/ahf/tor/merge_requests/2/

Trac:
Status: accepted to needs_review

Looks good! I've made some initial comments and asked some questions. Also, I should take another pass or two over this code when I'm a little less tired, and think about the pointer arithmetic very carefully.

Trac:
Status: needs_review to needs_revision

I believe I've been over everything you have commented on in the reviews now. We've postponed merging the test functions into a simplified function that tests all backends and we have postponed doing coverage "hints" in the code.

I've added a clean-up patch of some of the Zstandard code with regards to flushing the internal buffer in a more eager manner.

Let me know if this looks acceptable for merge and I'll happily do the squashing as well (whatever you find easiest!)

Note that this is build on the comments already made in both https://gitlab.com/ahf/tor/merge_requests/1 and https://gitlab.com/ahf/tor/merge_requests/2 - Gitlab seems to hide that a bit.

Once this code lands we can close both #21662 (moved), #21663 (moved) and #21664 (moved).

Trac:
Status: needs_revision to needs_review

Merging!

Trac:
Status: needs_review to closed
Resolution: N/A to implemented

Neat! Friendly reminder that this needs to be added to the dir-spec or it doesn't exist. ;)

atagar: It isn't actually used yet; this ticket is just for the backend code.

Ahhh, gotcha. So just to be clear: it's not actually being exposed on the DirPort yet?

That's correct.

closed

changed time estimate to 16h

mentioned in issue #21663 (moved)

mentioned in issue #21758 (moved)

moved to tpo/core/tor#21662 (closed)