Opened 7 years ago

Closed 17 months ago

#7176 closed enhancement (not a bug)

Adapt and update AccessLabs' patches for reduced Tor memory consumption for embedded devices

Reported by: cypherpunks Owned by: nickm
Priority: Medium Milestone: Tor: 0.3.4.x-final
Component: Core Tor/Tor Version: Tor: unspecified
Severity: Normal Keywords: tor-client, embedded, low-memory, sponsor8-maybe, 034-triage-20180328, 034-removed-20180328
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor: Sponsor8-can

Description

These are a set of patches written by Access Labs. They claim that these patches reduce memory usage on small low memory devices.

The patches are in several different areas:

  • using anonymous mmap() for storing large chunks of data (instead of tempfiles)
  • persisting the keys into /etc/tor/keys, rather than putting them in /var (tmpfs on OpenWRT)
  • using fixed values for should_rebuild_md_cache() && router_should_rebuild_store()
  • reducing the age of microdesc (changing the compile time defines)
  • using gzipped files for storing cache

I am including the files here as one ticket item rather than 5, but it might make sense to break them up into separate issues.

The source for these changes is the TorWRT package from Access Labs: https://accesslabs.org/wiki/TorWRT

Child Tickets

TicketTypeStatusOwnerSummary
#22703enhancementclosednickmAdd a KeyDirectory option to override location of $datadir/keys, and/or a cachedir option to override location of cached files.

Attachments (5)

001-anon_mmap.patch (32.9 KB) - added by cypherpunks 7 years ago.
anonymous mmap() patch
001-persistent_keys_in_etc.patch (1.7 KB) - added by cypherpunks 7 years ago.
persist keys to /etc/tor/
001-rebuild_when_big_dropped.patch (1.7 KB) - added by cypherpunks 7 years ago.
rebuild big when dropped
001-shorter_desc_lifetime.patch (1.5 KB) - added by cypherpunks 7 years ago.
shorter lifetimes
002-gzipped_cache.patch (32.7 KB) - added by cypherpunks 7 years ago.
gzipped cache

Download all attachments as: .zip

Change History (27)

Changed 7 years ago by cypherpunks

Attachment: 001-anon_mmap.patch added

anonymous mmap() patch

Changed 7 years ago by cypherpunks

persist keys to /etc/tor/

Changed 7 years ago by cypherpunks

rebuild big when dropped

Changed 7 years ago by cypherpunks

shorter lifetimes

Changed 7 years ago by cypherpunks

Attachment: 002-gzipped_cache.patch added

gzipped cache

comment:1 Changed 7 years ago by ioerror

I think Accesslabs should probably comment on this bug.

comment:2 Changed 7 years ago by ioerror

Can you give us some feedback on the device you're trying to use here?

comment:3 Changed 7 years ago by nickm

Keywords: tor-client low-memory added; low memory removed
Milestone: Tor: 0.2.4.x-final
Status: newneeds_review

Definitely worth looking into. (Are we sure the license on these is cool?)

Some of these look like they could never go into Tor per se -- for instance, the one about sticking keys in /etc -- if we wanted a separate key directory, it would need to be configurable, not a hardwired default.

The shorter desc lifetime patch seems like it should also be configurable.

Still gotta read the more complex ones.

comment:4 Changed 7 years ago by cypherpunks

There is a trac at AccessLabs as well, but it looks like it hasn't been updated in a few months (after they closed "support 2.3.19"). Might be able to contact someone via there to get them to comment on these.

Having a specific key directory different from the TorDataDirectory is useful for an OpenWRT router (as I mentioned above) as it allows the keys to survive a reboot. Assuming that preserving the keys across reboots is actually a useful thing to do. I think it would be absolutely fine for the OpenWRT build process to pass in a couple of ./configure defines to make this happen, e.g.
$ ./configure --enable-key-dir --with-key-dir=/etc/tor/keys

The use of mmap() rather than tempfile might be a total wash, as I'm not sure how tempfile() on a tmpfs differs from mmap(-1) in terms of memory usage. I would be surprised if they were hugely different. A small savings, sure, but I would really be shocked if it was a huge improvement.

The gzipped() file seems like it might be the biggest winner for mem usage. Compressing the data stored in the cache will really help with reducing memory consumption (I would imagine).

I can test the patches in a couple groups to see if they provide huge improvements. I will assume that the time to live patches aren't too important (they take hours to have any effect), so I'll ignore them. The anon_mmap() and the gzip cache are the most likely candidates for immediate improvement, with gzip the most likely to be useful. I will test memory consumption and CPU utilization with the following combinations:

  • tor-alpha (vanilla build)
  • tor-alpha + gzip
  • tor-alpha + anon_mmap()
  • tor-alpha + gzip + anon_mmap()

I'll report back the results of those changes. It is a bit hard to measure using only busybox, so the findings might be rough. Hopefully the memory savings are so profound that even busybox will detect them.

The device I am going to be testing on is a TP-LINK MR3040, which has the following spec: 400mhz MIPS cpu, 32mb RAM, 4mb Flash. This device can load tor-stable, but after doing so it is... unstable. This is due primarily to tor-stable requiring something like 20mb of RAM during load, and using 100% of CPU. I am hoping these patches will help significantly with that and bring the memory usage down to a sub-16mb size.

comment:5 Changed 7 years ago by cypherpunks

Ok, so I've done some testing and it turns out: a) testing memory consumption is harder than I thought, and b) the patches (in whichever combination) appear to have little to no effect. I think this is due to the length of testing being too short for the benefits of these patches to appear. I only tested them for less than an hour each.

Currently I have all the patches applied and running on my latest build. The memory consumption of Tor-alpha is around 11mb. With Tor-alpha vanilla (sans patches) the memory usage was the same, 11mb. Compared this with Tor, which used 16mb. I am happy keeping the patches on while I continue testing. I don't think they'll hurt performance and if they improve the memory consumption over extended use then that works for me.

comment:6 Changed 7 years ago by cypherpunks

Also, I have found the cause of Tor's 100% CPU utilization during load time. When OpenWRT boots up it doesn't have an accurate clock, so it starts at 00000000. Tor's init script launches at slot 50, and ntpd launches at 98. Tor complains about the serious clock skew and seems to be in a busy loop somewhere. Restarting the daemon after the time has been set correctly makes Tor start up properly w/o 100% CPU use.

Should I open a bug report?

comment:7 in reply to:  6 Changed 7 years ago by nickm

Replying to cypherpunks:

Also, I have found the cause of Tor's 100% CPU utilization during load time. When OpenWRT boots up it doesn't have an accurate clock, so it starts at 00000000. Tor's init script launches at slot 50, and ntpd launches at 98. Tor complains about the serious clock skew and seems to be in a busy loop somewhere. Restarting the daemon after the time has been set correctly makes Tor start up properly w/o 100% CPU use.

Should I open a bug report?

Sounds like two bug reports are in order: one on OpenWRT about starting Tor before you know what time it is, and another about Tor wasting cycles when it starts with a very wrong clock. For the latter one, it would be really helpful to know where Tor is spending its time.

comment:8 Changed 7 years ago by nickm

Milestone: Tor: 0.2.4.x-finalTor: unspecified

comment:9 Changed 2 years ago by nickm

Keywords: sponsor8-maybe added
Milestone: Tor: unspecifiedTor: 0.3.2.x-final
Severity: Normal

comment:10 Changed 2 years ago by nickm

Keywords: review-group-18 added

comment:11 Changed 2 years ago by nickm

Status: needs_reviewneeds_information

(Are we sure the license on these is cool?)

I'm not sure we can merge these at all. The tor-openwrt_0.2.3.19-rc-3.tgz file from which they are taken is confusing: It has a GPLv2 LICENSE file in its toplevel, and I can't tell whether they wanted that license to apply to their patches or not.

I'm going to try to find out to see if anybody from Access Now Labs can make a definitive statement here before I consider them for real merge or revision. Alternatively, if we can't get clarity on the licensing terms, we may need to recreate similar work independently in the time-honored manner.

Here is what the patches seem to be trying to do:

0001-anon_mmap: instead of using a journal file and keeping journaled descs in ram, tries to use mmap stuff to append them to the mmap on disk.
0001-persistent_keys_in_etc: Changes where keys are stored. This should be replaced with a separate KeyDirectory torrc option.
001-rebuild_when_big_dropped: Changes some logic in should_rebuild_md_cache() to rebuild the cache more aggressively.
001-shorter_desc_lifetime: Lowers TOLERATE_MICRODESC_AGE. This is risky and should only be done after analysis of microdescriptor lifetime. Also lowers ROUTER_MAX_AGE and OLD_ROUTER_DESC_MAX_AGE.
002-gzipped-cache: Stores microdescriptors and routerdescs compressed with gzip, uses anonymous mmap for decompressing them. May have sync/flush issues.

Last edited 2 years ago by nickm (previous) (diff)

comment:12 Changed 2 years ago by nickm

I just got this statement from Michael Carbone at Access Now:

Looping in lawyer person who says all is good.

Access Now owns the code and we allow it to be distributed under the
same licensing terms as Tor.

So we can have the above patches under BSD3, not GPLv2.

comment:13 Changed 2 years ago by nickm

Keywords: review-group-18 removed
Owner: set to nickm
Status: needs_informationaccepted
Summary: Reduced TOR memory consumption for embedded devicesAdapt and update AccessLabs' patches for reduced TOR memory consumption for embedded devices

Based on comments above, we can't take the patches as-is, but we can at least clean them up and merge them first. I'm going to open subtickets for this.

comment:14 Changed 2 years ago by nickm

Child tickets are now open, with review and comment on the logical next steps for each patch.

comment:15 Changed 2 years ago by nickm

Sponsor: Sponsor8-can

comment:16 Changed 2 years ago by nickm

Milestone: Tor: 0.3.2.x-finalTor: 0.3.3.x-final

comment:17 Changed 2 years ago by cypherpunks

Summary: Adapt and update AccessLabs' patches for reduced TOR memory consumption for embedded devicesAdapt and update AccessLabs' patches for reduced Tor memory consumption for embedded devices

comment:18 Changed 19 months ago by nickm

Milestone: Tor: 0.3.3.x-finalTor: 0.3.4.x-final

comment:19 Changed 17 months ago by nickm

Keywords: 034-triage-20180328 added

comment:20 Changed 17 months ago by nickm

Keywords: 034-removed-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

comment:21 Changed 17 months ago by nickm

Closing this ticket, since its children have been reparented to #25503.

comment:22 Changed 17 months ago by nickm

Resolution: not a bug
Status: acceptedclosed
Note: See TracTickets for help on using tickets.