LZMA/ZSTD descriptor compression support

changed milestone to %Tor: 0.3.3.x-final

added 033-must component::core tor/tor milestone::Tor: 0.3.3.x-final needs-analysis priority::medium regression? resolution::worksforme severity::normal status::closed type::defect labels

We should figure out if we broke compression. If not, this might not be 033-must, but it would be a shame to ship with a bug like this.

First off -- just because Moria is 0.3.4, doesn't mean it will necessarily support zstd and/or lzma2. It only supports those if it detects them at compile time, and I think Roger builds his own tors from source. So that's something to check on.

One way to learn more about what's going on here might be to use the -v option with curl, so it will also log all the sent/received headers to stderr.

Trac:
Milestone: N/A to Tor: 0.3.3.x-final
Keywords: N/A deleted, regression? 033-must needs-analysis added

Thanks Nick!

If not, this might not be 033-must, but it would be a shame to ship with a bug like this.

Agreed. To be clear I don't think this is particularly important. The introduction of lzma and zstd support are nice additions but if they don't work then not the end of the world. :)

First off -- just because Moria is 0.3.4, doesn't mean it will necessarily support zstd and/or lzma2. It only supports those if it detects them at compile time

Ahhh. My first thought when I read this was "how is this a useful feature?", but in reading proposal 278 I see now that the caller is supposed to provide all compression schemes they're willing to accept.

I think we need to change the spec in a few ways...

The spec should say that relays 'MUST' support plaintext and deflate/gzip (ie. callers can rely on those) but that relays only 'SHOULD' support lzma and zstd.
Callers should advertise all compression schemes they support in 'Accept-Encoding'. The relay will then pick the best it supports from among those (falling back to identity if none are supported). The compression scheme used is the response is indicated in the reply's 'Content-Encoding' header.
The spec should say how we pick among the compression. That is to say, lzma > zstd > gzip > identity (or whatever the actual behavior is).

If this sounds good and you can tell me how tor actually picks for #3 (closed) I'll send ya a spec patch.

Cheers! -Damian

Hi Nick. Just tried requesting lzma and zstd compressed descriptors from all present dirauths but they all fell back to plaintext responses as moria1 did. Do we know if this feature is exercised anywhere in practice?

The spec should say how we pick among the compression. That is to say, lzma > zstd > gzip > identity (or whatever the actual behavior is).

This is the required behaviour::

Relays MUST compress all directory documents with gzip. If lzma or zstd are available, each compression method MAY be used to compress some types of directory documents.

This is how the current implementation works, but clients shouldn't rely on the exact details:

Typically, compression methods are used for the documents that provide the best compression/CPU/RAM tradeoffs. Some compression methods are used to compress long-lived documents, then those documents are cached. Other methods are used for streaming compression when documents are requested.

This is the required behaviour:

When a directory mirror receives a request for compressed data, it MUST serve a format that is available and supported by the client.

This is how the current implementation works, but clients shouldn't rely on the exact details:

If multiple methods are supported, tor chooses the compression method in this order:

Client requests contain all supported methods in this order: lzma > zstd > zlib > gzip > identity https://gitweb.torproject.org/tor.git/tree/src/or/directory.c#n3627

Directory responses choose a common, supported method in this order: Precompressed, cached data: lzma > zstd > zlib > gzip > identity Streamed data: zstd > zlib > gzip > identity Anonymous connections (e.g. HSDirs): zlib > gzip > identity https://gitweb.torproject.org/tor.git/tree/src/or/directory.c#n3575

Thanks Tim! I went ahead and pushed Stem support for compression headers. Once I have an example of lzma and zstd working in the live network I'll add integ coverage and whip up a spec patch.

Replying to atagar:

Hi Nick. Just tried requesting lzma and zstd compressed descriptors from all present dirauths but they all fell back to plaintext responses as moria1 did. Do we know if this feature is exercised anywhere in practice?

You can test lzma against any of these relays: https://metrics.torproject.org/rs.html#search/radia

Unfortunately, they don't have zstd support, because it's not in the Debian release they are using. If you ask on tor-relays, you can probably find a few operators who have only enabled zstd, or who have enabled both lzma and zstd. The relevant configure options are --enable-lzma and --enable-zstd.

Tor also logs the supported compression methods on startup: Tor %s running on %s with Libevent %s, OpenSSL %s, Zlib %s, Liblzma %s, and Libzstd %s. https://gitweb.torproject.org/tor.git/tree/src/or/main.c#n3323

Hmm, I am having pkg-config issues on radia. No lzma support yet.

You can now test lzma and zstd against any of these relays: https://metrics.torproject.org/rs.html#search/radia

[notice] Tor 0.3.2.10-dev (git-57b69160a4a50f38) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2l, Zlib 1.2.8, Liblzma 5.1.0alpha, Libzstd 1.3.2, and PrivCount 3.0.0.

(The log line is slightly different because they are running a research version of Tor. That shouldn't matter for compression.)

Thanks Tim, but no luck. Perhaps I'm doing something wrong? The following should fetch a lzma compressed descriptor from radia0...

curl --verbose --header "Accept-Encoding: x-tor-lzma" 91.121.230.208:9030/tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020CE31 > /tmp/desc
*   Trying 91.121.230.208...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 91.121.230.208 (91.121.230.208) port 9030 (#0)
> GET /tor/server/fp/9695DFC35FFEB861329B9F1AB04C46397020CE31 HTTP/1.1
> Host: 91.121.230.208:9030
> User-Agent: curl/7.47.0
> Accept: */*
> Accept-Encoding: x-tor-lzma
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 29 Mar 2018 20:10:25 GMT
< Content-Type: text/plain
< X-Your-Address-Is: 208.113.130.116
< Content-Encoding: identity
< Expires: Sat, 31 Mar 2018 20:10:25 GMT

Replying to atagar:

Thanks Tim, but no luck. Perhaps I'm doing something wrong? The following should fetch a lzma compressed descriptor from radia0...

Descriptors are not available in the lzma encoding, because they are streamed:

Replying to teor:

Directory responses choose a common, supported method in this order: Precompressed, cached data: lzma > zstd > zlib > gzip > identity Streamed data: zstd > zlib > gzip > identity Anonymous connections (e.g. HSDirs): zlib > gzip > identity https://gitweb.torproject.org/tor.git/tree/src/or/directory.c#n3575

This is what I get:

$ for encoding in x-tor-lzma x-zstd deflate gzip identity; do
    for doc in server/authority status-vote/current/consensus; do
        echo "Accept-Encoding: $encoding"
        echo "Requested: $doc"
        curl -s -O --header "Accept-Encoding: $encoding" \
            91.121.230.208:9030/tor/$doc && \
        file authority
    done
done
Accept-Encoding: x-tor-lzma
Requested: server/authority
authority: ASCII text
Accept-Encoding: x-tor-lzma
Requested: status-vote/current/consensus
authority: ASCII text
Accept-Encoding: x-zstd
Requested: server/authority
authority: Zstandard compressed data (v0.8+), Dictionary ID: None
Accept-Encoding: x-zstd
Requested: status-vote/current/consensus
authority: Zstandard compressed data (v0.8+), Dictionary ID: None
Accept-Encoding: deflate
Requested: server/authority
authority: zlib compressed data
Accept-Encoding: deflate
Requested: status-vote/current/consensus
authority: zlib compressed data
Accept-Encoding: gzip
Requested: server/authority
authority: gzip compressed data, max compression, from Unix
Accept-Encoding: gzip
Requested: status-vote/current/consensus
authority: gzip compressed data, max compression, from Unix
Accept-Encoding: identity
Requested: server/authority
authority: ASCII text
Accept-Encoding: identity
Requested: status-vote/current/consensus
authority: ASCII text

I think that the consensus might not be available in lzma because it was cached on disk by a previous version of tor. Maybe wait an hour or two for a new consensus to arrive, and try lzma again?

Oh, it helps if the script checks the consensus file after downloading consensuses. Now lzma works:

$
for encoding in x-tor-lzma x-zstd deflate gzip identity; do
    for doc in server/authority status-vote/current/consensus; do
        echo
        echo "Accept-Encoding: $encoding"
        echo "Requested: $doc"
        curl -s -O --header "Accept-Encoding: $encoding" \
            91.121.230.208:9030/tor/$doc && \
            file `basename $doc`
    done
done

Accept-Encoding: x-tor-lzma
Requested: server/authority
basename $doc
authority: ASCII text

Accept-Encoding: x-tor-lzma
Requested: status-vote/current/consensus
basename $doc
consensus: LZMA compressed data, streamed

Accept-Encoding: x-zstd
Requested: server/authority
basename $doc
authority: Zstandard compressed data (v0.8+), Dictionary ID: None

Accept-Encoding: x-zstd
Requested: status-vote/current/consensus
basename $doc
consensus: Zstandard compressed data (v0.8+), Dictionary ID: None

Accept-Encoding: deflate
Requested: server/authority
basename $doc
authority: zlib compressed data

Accept-Encoding: deflate
Requested: status-vote/current/consensus
basename $doc
consensus: zlib compressed data

Accept-Encoding: gzip
Requested: server/authority
basename $doc
authority: gzip compressed data, max compression, from Unix

Accept-Encoding: gzip
Requested: status-vote/current/consensus
basename $doc
consensus: zlib compressed data

Accept-Encoding: identity
Requested: server/authority
basename $doc
authority: ASCII text

Accept-Encoding: identity
Requested: status-vote/current/consensus
basename $doc
consensus: ASCII text, with very long lines

I opened #25676 (moved) for zlib consensuses in response to gzip requests:

Accept-Encoding: gzip
Requested: status-vote/current/consensus
basename $doc
consensus: zlib compressed data

Great, thanks Tim! I'll give this a whirl tomorrow.

Stem now has working support for zstd and lzma. Thanks for the help, Tim!

Trac:
Status: new to closed
Resolution: N/A to worksforme

closed

moved to tpo/core/tor#25667 (closed)

LZMA/ZSTD descriptor compression support

Child items ...

Activity