Opened 3 years ago

Last modified 22 months ago

#22706 new enhancement

Compress cached descriptors and consensus

Reported by: nickm Owned by:
Priority: Low Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: disk-usage, 034-triage-20180328, 034-removed-20180328, 035-removed-20180711
Cc: Actual Points:
Parent ID: #25503 Points: 4
Reviewer: nickm Sponsor:


We can save disk space on clients (perhaps?) if we store more of these objects compressed.

Child Tickets

Attachments (1)

002-gzipped_cache.patch (32.7 KB) - added by nickm 3 years ago.

Download all attachments as: .zip

Change History (11)

comment:1 Changed 3 years ago by nickm

Component: - Select a componentCore Tor/Tor
Status: newneeds_review

Changed 3 years ago by nickm

Attachment: 002-gzipped_cache.patch added

comment:2 Changed 3 years ago by nickm

Keywords: review-group-18 removed
Status: needs_reviewnew


  • I think this presumes #22704.
  • This patch seems to have the same issue with anonymous mmaps as I noted in #22704: is it really saving RAM? I can believe it saves disk when Tor isn't running.
  • This patch should IMO be off-by-default, and I don't think it's safe to turn it on for directory caches.
  • The compression parts need to be rewritten to use the new compress.c APIs.
  • There are bigger memory savings available on clients: once we have the (micro)descriptors parsed, we can just leave them on disk, parse them, and not keep their text in memory at all. (After parsing we only use the text to answer controller requests and directory requests.)
  • We shouldn't add our own functions in the gz* namespace.
  • Cache compression won't work very well: it only handles one (micro)descriptor at a time.

If we decide to take this approach, we should start with a fresh patch, using parts of the design here as needed.

comment:3 Changed 3 years ago by nickm

Sponsor: Sponsor8-can

comment:4 Changed 3 years ago by nickm

Milestone: Tor: 0.3.2.x-finalTor: 0.3.3.x-final

comment:5 Changed 3 years ago by nickm

Milestone: Tor: 0.3.3.x-finalTor: 0.3.4.x-final
Priority: MediumLow
Type: defectenhancement

comment:6 Changed 3 years ago by nickm

Keywords: 034-triage-20180328 added

comment:7 Changed 3 years ago by nickm

Keywords: 034-removed-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

comment:8 Changed 3 years ago by nickm

Milestone: Tor: 0.3.4.x-finalTor: 0.3.5.x-final
Parent ID: #7176#25503

comment:9 Changed 2 years ago by nickm

Keywords: 035-removed-20180711 added
Milestone: Tor: 0.3.5.x-finalTor: unspecified

These tickets are being triaged out of 0.3.5. The ones marked "035-roadmap-proposed" may return.

comment:10 Changed 22 months ago by gaba

Keywords: sponsor8-maybe removed
Sponsor: Sponsor8-can
Note: See TracTickets for help on using tickets.