Opened 3 years ago

Last modified 17 months ago

#22704 new enhancement

Use mmaps to handle journalled microdesc/routerdesc files.

Reported by: nickm Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: mmap, 034-triage-20180328, 034-removed-20180328, 035-removed-20180711
Cc: Actual Points:
Parent ID: #25503 Points: 3
Reviewer: nickm Sponsor:


We waste RAM by keeping extrainfos, microdescs, and descriptors allocated locally after we have saved them to the journal. Instead, we could mmap the journal, and use mremap to grow it as we expand.

#7176 has a patch for this that could use some review and revision.

Child Tickets

Attachments (1)

001-anon_mmap.patch (32.9 KB) - added by nickm 3 years ago.

Download all attachments as: .zip

Change History (12)

Changed 3 years ago by nickm

Attachment: 001-anon_mmap.patch added

comment:1 Changed 3 years ago by nickm

Keywords: review-group-18 added
Reviewer: nickm
Status: newneeds_review

comment:2 Changed 3 years ago by nickm

Status: needs_reviewnew

Notes for revisions:

  • This patch kills off the "write to X.tmp, then rename to X" logic globally. That's not okay to do. If it turns out to be necessary on low-disk systems, then we should do it on those systems only, and we should do it for the cached-* files only.
  • The mmap-logic is unix-only. We need to support Windows too.
  • I'm missing something, but I don't understand how anonymous memory maps actually help here. Is it a memory fragmentation issue or something? An anonymous mmap is RAM-backed and swappable, right?
  • load_string_into_mmap should have a name like tor_mmap_append(); it should work on file mmaps too (maybe it does?), and it should do at most one increment and copy the data all at once.
  • likewise for load_file_into_mmap -- and why would you even do that when you can just mmap the file?
  • There should be a wrapper function that does a nul-terminated memdup for a single microdesc/descriptor.
  • It may be a lower priority to do this for routerdescs and extrainfos: clients don't use those, after all.

I think we can reuse some of the code from the original patch, but we should really write a completely new branch here if we decide to do this one.

comment:3 Changed 3 years ago by nickm

Keywords: review-group-18 removed

whoa, that keyword was a mistake.

comment:4 Changed 3 years ago by nickm

Sponsor: Sponsor8-can

comment:5 Changed 3 years ago by nickm

Milestone: Tor: 0.3.2.x-finalTor: 0.3.3.x-final

comment:6 Changed 2 years ago by nickm

Milestone: Tor: 0.3.3.x-finalTor: 0.3.4.x-final
Type: defectenhancement

comment:7 Changed 2 years ago by nickm

Keywords: 034-triage-20180328 added

comment:8 Changed 2 years ago by nickm

Keywords: 034-removed-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

comment:9 Changed 2 years ago by nickm

Milestone: Tor: 0.3.4.x-finalTor: 0.3.5.x-final
Parent ID: #7176#25503

comment:10 Changed 23 months ago by nickm

Keywords: 035-removed-20180711 added
Milestone: Tor: 0.3.5.x-finalTor: unspecified

These tickets are being triaged out of 0.3.5. The ones marked "035-roadmap-proposed" may return.

comment:11 Changed 17 months ago by gaba

Keywords: sponsor8-maybe removed
Sponsor: Sponsor8-can
Note: See TracTickets for help on using tickets.