Use mmaps to handle journalled microdesc/routerdesc files.

We waste RAM by keeping extrainfos, microdescs, and descriptors allocated locally after we have saved them to the journal. Instead, we could mmap the journal, and use mremap to grow it as we expand.

#7176 (moved) has a patch for this that could use some review and revision.

changed milestone to %Tor: unspecified

added 034-removed-20180328 034-triage-20180328 035-removed-20180711 component::core tor/tor milestone::Tor: unspecified mmap parent::25503 points::3 priority::medium reviewer::nickm severity::normal status::new type::enhancement labels

Trac:
001-anon_mmap.patch

Trac:
Keywords: sponsor8-maybe mmap deleted, sponsor8-maybe mmap review-group-18 added
Reviewer: N/A to nickm
Status: new to needs_review

Notes for revisions:

• This patch kills off the "write to X.tmp, then rename to X" logic globally. That's not okay to do. If it turns out to be necessary on low-disk systems, then we should do it on those systems only, and we should do it for the cached-* files only.
• The mmap-logic is unix-only. We need to support Windows too.
• I'm missing something, but I don't understand how anonymous memory maps actually help here. Is it a memory fragmentation issue or something? An anonymous mmap is RAM-backed and swappable, right?
• load_string_into_mmap should have a name like tor_mmap_append(); it should work on file mmaps too (maybe it does?), and it should do at most one increment and copy the data all at once.
• likewise for load_file_into_mmap -- and why would you even do that when you can just mmap the file?
• There should be a wrapper function that does a nul-terminated memdup for a single microdesc/descriptor.
• It may be a lower priority to do this for routerdescs and extrainfos: clients don't use those, after all.

I think we can reuse some of the code from the original patch, but we should really write a completely new branch here if we decide to do this one.

Trac:
Status: needs_review to new

whoa, that keyword was a mistake.

Trac:
Keywords: sponsor8-maybe mmap review-group-18 deleted, sponsor8-maybe mmap added

Trac:
Sponsor: N/A to Sponsor8-can

Trac:
Milestone: Tor: 0.3.2.x-final to Tor: 0.3.3.x-final

Trac:
Milestone: Tor: 0.3.3.x-final to Tor: 0.3.4.x-final
Type: defect to enhancement

Trac:
Keywords: sponsor8-maybe mmap deleted, mmap, sponsor8-maybe, 034-triage-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

Trac:
Keywords: N/A deleted, 034-removed-20180328 added

Trac:
Milestone: Tor: 0.3.4.x-final to Tor: 0.3.5.x-final
Parent: #7176 (moved) to #25503 (moved)

These tickets are being triaged out of 0.3.5. The ones marked "035-roadmap-proposed" may return.

Trac:
Milestone: Tor: 0.3.5.x-final to Tor: unspecified
Keywords: N/A deleted, 035-removed-20180711 added

Trac:
Sponsor: Sponsor8-can to N/A
Keywords: sponsor8-maybe deleted, N/A added

changed time estimate to 24h

mentioned in issue #22706 (moved)

moved to tpo/core/tor#22704 (closed)