Opened 6 years ago

Closed 3 years ago

#9403 closed enhancement (worksforme)

consider using PyPy with 'arm' where appropriate

Reported by: gmorehouse Owned by: atagar
Priority: Medium Milestone:
Component: Core Tor/Nyx Version:
Severity: Normal Keywords: arm, python, pypy
Cc: gordon@… Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I'm currently doing a lot of experimentation with Tor relays on single-board computers such as the Raspberry Pi. Currently, 'arm' is *very* heavy on such machines, taking a lot of memory and CPU.

It might be worth it to release a PyPy[1] powered version if it results in significant improvements in speed and/or memory usage on resource-limited systems with ARM CPUs. Or, possibly, to build alternate packages for users of the Tor Project repos for Debian et al.

[1] http://pypy.org/

Child Tickets

Change History (7)

comment:1 Changed 6 years ago by gmorehouse

Cc: gordon@… added

comment:2 Changed 6 years ago by atagar

I suspect the vast majority of arm's cpu intensiveness is connection and resource resolution. If proc contents are unavailable then it falls back on shelling out to ps and netstat. If you're able to localize what makes arm heavy on those systems then I'd be curious to hear your results.

comment:3 Changed 6 years ago by gmorehouse

I am occasionally seeing some exception spew in between refreshes that contains the text 'lsof is unavailable.'

Got any quick URLs I could use to learn how to profile 'arm' in a quick and dirty way to see what it's shelling out to?

Also, are there common (eg POSIX) system calls that are wrapped/usable in Python that are the equivalent of querying /proc for the information you need? /proc is pretty Linux-specific. I'd think system calls would be just as fast if not faster, and maybe more portable, but I'm unsure what information you're reading.

comment:4 Changed 6 years ago by atagar

Try running 'arm --debug'. That will dump a debug log to '~/.arm/log'.

Also, are there common (eg POSIX) system calls that are wrapped/usable in Python that are the equivalent of querying /proc for the information you need?

Not that I'm aware of. PSUtils, a python package that provides this kind of information, takes the same approach with regard to proc. Connection resolution is actually pretty portable due to the number of fallbacks arm uses...

  • Linux: /proc, netstat, sockstat, lsof, ss
  • OSX: lsof
  • FreeBSD: sockstat, procstat

https://gitweb.torproject.org/arm.git/blob/HEAD:/src/util/connections.py

I'd think system calls would be just as fast if not faster, and maybe more portable, but I'm unsure what information you're reading.

I'm not really sure of the alternative you're proposing. If you have a better approach then patches welcome.

Did you have any other questions?

comment:5 Changed 6 years ago by gmorehouse

I'll run with --debug as you suggest and report back.

I'll look into the code if I can at some point and see if I can come up with coherent suggestions and/or patches. At my current rate of overcommitment, I'll probably get back to you with patches before 2030 or so. ;)

comment:6 Changed 6 years ago by gmorehouse

Apparently I don't have many alternatives to propose: http://stackoverflow.com/questions/9486786/portable-way-in-c-to-get-the-routing-table

However, I'll still try to figure out where it's bottlenecked on the Pi and maybe we can make it run much more smoothly.

comment:7 Changed 3 years ago by atagar

Resolution: worksforme
Severity: Normal
Status: newclosed

Cleaning up old tickets. Nyx is undergoing a rewrite, if this is still relevant then feel free to reopen.

Note: See TracTickets for help on using tickets.