Opened 12 months ago

Last modified 12 months ago

#32515 new enhancement

Keep Good Summaries From Reliable Sources. Example: Tor's CODEC Strategy, zlib, lzma, zstd

Reported by: werd Owned by:
Priority: Medium Milestone:
Component: Webpages Version:
Severity: Normal Keywords: zlib, lzma, zstd
Cc: ggus Actual Points:
Parent ID: Points:
Reviewer: Sponsor:


Ironically, about a week after I spent a bunch of time reading up on zlib, lzma, and zstd, on 2019-09-13 someone (Steve Snyder) on the tor-dev list asked:

Given the multiple compression types supported (none, lzma, zlib, zstd), what is the order of preference for runtime use?

Put another way, which compression method(s) should be supported to get optimal runtime performance from a Tor node?

To which Nick Mathewson replied:

For big objects like consensuses or consensus diffs that are sent over and over, relays prefer to use whichever compression method has the highest compression -- that's lzma2, then zstd, then zlib, then none. Lzma2 (aka xz) is more expensive to calculate, but the relays only need to calculate it once per compressed object, and then they can send it over and over.

For smaller objects that are compressed in a stream (descriptors and microdescriptors), relays will not use xz, since it would be to expensive to recompute it for every stream. They'll prefer zstd, then zlib, then none.

So if you want to save bandwidth above all, you should enable all compression algorithms.

If you want to save CPU above all, you should enable all compression algorithms except xz.

If you want to save bandwidth and CPU, I _think_ enabling all the compression algorithms will result in Tor making good choices (as described above). But I'd appreciate benchmarks if anybody has tried it both ways to find out.

This awesome summary, which I'd not gleaned in all my reading, should be preserved some place in a hierarchical structure of a builder/developer FAQ.

As you know, by default Tor reports via standard output and logging, the compression types compiled into Tor. I wanted to know if it was worth the effort to add more than what the current default is.

There didn't seem a place to put it in the existing structure. Both and are places where it makes sense, but best I can tell this raises a few issues: duplicating info is not great, those pages are official, and from other pages on I've read, those pages are output from some other software. And looking at the trac index page is overwhelming, I'm not sure this awesome tidbit should be yet another page on that list.

Please excuse me if I placed this under the wrong Type, and also I wasn't sure what Subcomponent should be chosen.

Child Tickets

Change History (2)

comment:1 Changed 12 months ago by pili

This looks like something that should go in the new dev or docs portal

comment:2 Changed 12 months ago by cypherpunks

further, compression algo is unused if compiled with older version since libs upgrade.

Nov 17 02:10:36.000 [notice] Tor opening new log file.
Nov 17 02:10:36.669 [warn]

Tor was compiled with zstd 1.4.2, but is running with zstd 1.4.3. For safety, we'll avoid using advanced zstd functionality.

Note: See TracTickets for help on using tickets.