Opened 6 years ago

Closed 6 years ago

#9686 closed defect (fixed)

MaxMemInCellQueues minimum of 500MB is too large for low-RAM relays (Raspberry Pi)

Reported by: gmorehouse Owned by:
Priority: High Milestone: Tor: 0.2.4.x-final
Component: Core Tor/Tor Version:
Severity: Keywords: tor-relay, 024-backport, lowmem, raspberrypi, nickm-backport-02422, arma-backport-02422
Cc: anong Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

This is related to #9646.

Based on discussion in that ticket, I attempted to set my MaxMemInCellQueues to 384MB on my Raspberry Pi relay, which has a hardware maximum of 512MB RAM (and boots with about 485MB usable physical memory when running Raspbian).

This results in the following error:

[warn] MaxMemInCellQueues must be at least 500 MB for now. Ideally, have it as large as you can afford.

Ideally, this setting would be flexible, and allow lower values, possibly calculated as some percentage of total available physical memory (or possibly not, depending on how much time you'd like to spend mucking with it).

If Tor has 500MB in cell queues on a 512MB Raspberry Pi, though, the Pi is already guaranteed to be hitting swap if the out-of-memory killer hasn't already come around to 'fix' things. I took a complete guess at 384MB as a reasonable value for a machine that boots with 485MB free.

Under non-botnet network conditions, a Raspberry Pi can happily sustain at least 2 to 3 Mbps of traffic as a relay, and that's without extensive tuning. I think it's worth the effort to keep these low-cost, low-resource, physically small boxes in mind when working on Tor, as many of them scattered around the world would be a fairly easy accomplishment and might do a lot to beef up the network.

Also, the libgcrypt folks have been working hard in the last 30 days on ARM assembly language implementations of algorithms which Tor uses[1]; if you assume these implementations will eventually cross-pollinate into libraries used by Tor, a Raspberry Pi may one day be able to relay a lot more than 2-3Mbps.

  1. http://gnupg.10057.n7.nabble.com/template/NamlServlet.jtp?macro=search_page&node=27394&query=arm&days=30

Child Tickets

Change History (16)

comment:1 Changed 6 years ago by nickm

Keywords: 024-backport added
Milestone: Tor: 0.2.5.x-final

To be clear -- you've tried running it with the minimum lowered to 384 MB, and it seems to work okay without generating frequent messages about having run out of memory?

comment:2 Changed 6 years ago by gmorehouse

No, I *can't* run it with the minimum lowered to 384MB, it doesn't allow anything below 500MB.

comment:3 Changed 6 years ago by nickm

Status: newneeds_review

I've tried this as part of my #10169 branch, to avoid a conflict. (#10169 also renames the option.)

comment:4 Changed 6 years ago by gmorehouse

How distant is your branch from 0.2.4.18-rc? I have to compile Tor for my Raspberry Pi each upgrade anyway, so maybe I could help do some testing or something.

comment:5 Changed 6 years ago by sysrqb

Raspberry Pis also have a version with 256 MB. I don't know if it is wise to appease such a memory constrained system though...

I wonder if the validation error message should explicitly state that Tor is defaulting to the lower limit. Right now users may think it's just a suggestion. E.g. change

    log_warn(LD_CONFIG, "MaxMemInQueues must be at least 256 MB for now. "
             "Ideally, have it as large as you can afford.");

to

    log_warn(LD_CONFIG, "MaxMemInQueues was configured to be less than 256 MB, "
             "we will increase it to the lower limit of 256 MB. "
             "Ideally, have it as large as you can afford.");

I'm really not happy with that wording, though.

Otherwise I think the fix looks okay.

comment:6 Changed 6 years ago by gmorehouse

So is 256MB the new minimum?

I don't think it'd be wise to try running Tor on a 256MB machine, but surely a 512MB machine should be allowable - at $25 each (and falling), 10,000 of these could help sustain the network and reduce the percentage of compromised relays an attacker can potentially control.

I came back to check this ticket after stumbling across the Sniper Attack entry[1] on the Tor Project blog. I'm concerned that setting the minimum MaxMemInQueues to 512MB leaves all relays with <=512MB of physical RAM vulnerable to this attack. Beyond the potential usefulness of the 512MB Raspberry Pi to the Tor network, there's also a potential harm here given that a 512MB Pi relay could be DOS'd or used in deanonymizing hidden services. People are very much using 512MB Pis as relays, not only myself but many others as shown by posts I've found and the level of interest from users in my Pi-specific Tor relay project[2] (still pre-alpha).

If the lower bound on MaxMemInQueues has been reduced (say, to 384MB), consider this support for that. If not, these are the reasons I think it should be reduced. (Sorry, I don't know where to look to see what's been done in the branch mentioned above.)

  1. https://blog.torproject.org/blog/new-tor-denial-service-attacks-and-defenses
  2. https://github.com/gordon-morehouse/cipollini

comment:7 Changed 6 years ago by anong

Priority: normalmajor

This is definiteley a major issue. At the moment it seems impossible to run a stable relay on a low-mem device like the Raspberry Pi, since the Tor process gets killed by the oom-killer repeatedly. This prevents tens of thousands of devices from being stable entry guards or even stable exits.

Lowering the MaxMemInQueues minimum to 384 MB sounds like a good solution.

Last edited 6 years ago by anong (previous) (diff)

comment:8 Changed 6 years ago by gmorehouse

I can't even tell from previous comments whether the MaxMemInQueues has been changed or not. It'd probably be good to address that, and again, Raspberry Pis were capable of handling several Mbps of traffic before any of this, and before the botnet. It is a bad idea to exclude them and other 512MB devices.

comment:9 Changed 6 years ago by anong

Cc: anong added
Keywords: lowmem raspberrypi added

comment:10 Changed 6 years ago by nickm

Milestone: Tor: 0.2.5.x-finalTor: 0.2.4.x-final

Resolved in 0.2.5. Marking for possible backport.

comment:11 Changed 6 years ago by nickm

Recommendation: 0.2.4.22 backport, very simple

comment:12 Changed 6 years ago by nickm

Keywords: nickm-backport-02422 added

Adding a tag for tickets I think we should backport into 0.2.4.22. Omitting ones where I said "unsure"

comment:13 in reply to:  7 Changed 6 years ago by arma

Replying to anong:

This is definiteley a major issue. At the moment it seems impossible to run a stable relay on a low-mem device like the Raspberry Pi, since the Tor process gets killed by the oom-killer repeatedly. This prevents tens of thousands of devices from being stable entry guards or even stable exits.

Lowering the MaxMemInQueues minimum to 384 MB sounds like a good solution.

This comment worries me a bit -- I think we are still hoping to have MaxMemInQueues be so large that it never triggers in practice, not even for raspberry pi relays. Because if it ever does trigger, that means we're closing the circuit on a user. So the change in this ticket should *not* be interpreted as fixing "the Tor process gets killed by the oom-killer repeatedly", yes?

comment:14 in reply to:  11 Changed 6 years ago by arma

Replying to nickm:

Recommendation: 0.2.4.22 backport, very simple

Backport is ok by me if you're convinced there are users of 0.2.4.x who need it.

comment:15 Changed 6 years ago by nickm

Keywords: arma-backport-02422 added

Adding a keyword to both tickets arma likes for backport to 0.2.4.22 so far.

comment:16 Changed 6 years ago by nickm

Resolution: fixed
Status: needs_reviewclosed

Backported as 35699ef9f5d2814203653e16cb0cd176a0190ae0.

Note: See TracTickets for help on using tickets.