Opened 7 years ago

Closed 2 years ago

#7044 closed enhancement (duplicate)

Use a better compression algorithm for directory stuff

Reported by: nickm Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: tor-relay needs-proposal
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Have a look at these results:

[554]$ cat cached-microdesc-consensus |wc -c
  595929
[555]$ gzip -c -9 cached-microdesc-consensus |wc -c
  255689
[557]$ xz -c -9 cached-microdesc-consensus |wc -c
  238396
[556]$ bzip2 -c -9 cached-microdesc-consensus |wc -c
  222433

Now, we might still want to stick with zlib for general-purpose compression of descriptors and microdescriptors, since we need to compress those streams on the fly. But for consensus documents, which change infrequently, we don't much care about the CPU load required to compress them.

Sending 13% less bytes on consensus traffic would be a clear and win.

Child Tickets

Change History (4)

comment:1 Changed 7 years ago by rransom

Is there a spec for xz or bzip2?

Have you audited libxz or libbzip2? (The consensus must be decompressed and parsed before the signatures on it can be verified.)

comment:2 Changed 7 years ago by nickm

xz has its specifications here, allegedly: http://tukaani.org/xz/format.html

bzip2 seems to be less well-specified, but the format is stable and well-understood enough to allow over a dozen independent implementations, fwict.

I have audited neither one's implementation; that would need to be part of any attempt to come up with a proposal for this. But keep in that they're both fairly widely used for uncompressing fairly untrusted stuff already, and seem to have gotten a fair amount of attention.

comment:3 in reply to:  2 Changed 7 years ago by rransom

Replying to nickm:

xz has its specifications here, allegedly: http://tukaani.org/xz/format.html

The ‘specification’ linked to on that page contains no information about the LZMA compression format. It does provide quite a few ways to make an .xz stream incompatible with some .xz readers, though.

comment:4 Changed 2 years ago by nickm

Resolution: duplicate
Severity: Normal
Status: newclosed

Closing as duplicate of #21662

Note: See TracTickets for help on using tickets.