wiki:doc/arm

Arm Development Wiki

Bugs

Easy

Interested in hacking on arm? The following is a great place to start. These are small, decently easy bugs and improvements I haven't had time to address. If you get stuck at all then please let me know and I'd be happy to help!

  • When running as a client (no ORPort) and Tor stops the header panel doesn't say so. This was reported a long while back so we first need to double check if this is still an issue.
  • The config write dialog (ie, the one for saving the config) has its a misaligned border when it's smaller than the top detail section.
  • Simple Improvements
    • Graph lacks a config option for moving right to left (feature request by ioerror and rdegraaf)
    • Tab completion for input fields that expect a filesystem path
    • Make arm's configuration values dynamically editable via a listener architecture, with an option to save the current configuration to the armrc.
    • Add arm's configuration options to its man page. Ideally these wouldn't be hardcoded, but instead added by a script that fetches options from the source.
    • The current arm configuration options were added in an ad hoc fashion using a naming scheme a little reminiscent of Firefox's about:config options. Now that we have a general idea what config options arm will use we can come up with a nicer, more user friendly naming hierarchy.
  • Prompt the user to open the control port if tor is running but it's closed at startup (either a custom prompt or offer to modify the torrc on their behalf, idea by Thomas Schreiber), a better plan for the autoconfiguration is...
    • Check the arguments of the tor process to see if it was given a -f arg.
    • Look in all of the common default torrc locations to see if there's a file found. If either none or multiple are found then abort.
    • If the above have narrowed the torrc to a single likely location then prompt the user to...
      • set controller connection for this session by reverting the change when arm quits (can this be done via a temporary copy?)
      • persist the change to the torrc
      • abort
    • Cycle input when doing tab completion in the interpretor panel (ie, "GETINFO " would cycle through results alphabetically), suggestion by Sebastian

Harder

  • Log deduplication is currently an n^2 operation. Hence it can't handle large logs (for instance, when at the DEBUG runlevel). Currently we're timing out the function if it takes too long, but a more efficient method for deduplication would be preferable.
  • Tor configuration values cached in 'util/torTools.py' don't take SETCONF events by other controllers (like Vidalia) into account. This is blocked on ticket 1692.
  • The log prepopulation fails to limit entries to just the current Tor instance if the file isn't logged to at the NOTICE level. A fix is to use the timestamps to see if it belongs to the attached Tor instance or not. But this requires Tor's uptime, which is blocked on the implementation of proposal 173.

1.4.X Plans

Next Release

  • Match application connection entries to circuts (via the Target field of STREAM events?), thanks to Sebastian and StrangeCharm
  • Wizard options...
    • run at system start
    • tor weather integration
  • Interpretor improvements
    • Provide coloring and highlighting comparable to the descriptor popup for ns/* and desc/* GETINFO queries
  • Connection Panel Family Entries
    • Figure out a method of making it obvious these aren't real connections
    • Give warning and display in red if...
      • family declarations aren't mutual
      • relay isn't in the consensus
      • check if relay is alive via a VERSION cell handshake?
      • multiple family entries resolve to the same entry (could use the fingerprint for one entry, and the nickname for another)
      • provide a warning if named family entries lack the named flag
  • Allow for hotswapping between tor instances, with prepopulation of the stats
  • Provide a config option for nickname => control port mappings
  • Associate streams to circuits (if there isn't a method for doing this then poke nickm)
  • bug: graph updates (both the display and avg) when redraw is forced
  • use circ-bw events to show activity of circuits
  • option to clear dead circ with another option to clear them out (feature request by nickm)
  • use sparklines to improve client/exit usage graph accuracy?
  • the cpu usage spikes for scrollable content when the key's held. Tried coalescing the events without success.
  • warn if there's exit connections (other than DNS) when we're strictly a non-exiting relay
  • optionally display unproxied connections in red (to help catch leaks)
  • look into CAPs to get around permission issues for the connection listing (check the 'capabilities' man page and this article)
  • fallback when unable to limit connection lookups to just Tor, instead listing anything in both the lookup and consensus
  • pick apart applications like iftop and pktstat to see how they get per-connection bandwidth usage (forum discussion)
  • ... TBD

Pending Release Notes

The following are the release notes for completed portions of the next release.

  • fix: accounting for Fedora's UsrMove feature (fix by miceliux, ticket)

Projects

Circuit Details

We can passively provide consensus and locale information on the relays being used in our circuits, but most of the more interesting information requires active data collection (for instance rdns and whois lookups for the hostname, ISP, etc). We can't do this on-demand because it would leak our circuits to resolvers and our ISP (hopefully it's needless to say why this is horribly bad).

An alternative would be to actively collect this data on every relay in the consensus, caching the results, and only displaying information form that cache. This has the obvious disadvantage of needing thousands of lookups which would both clog the user's connection and take a very long time. It's especially wasteful since the information we receive back is mostly thrown away (we just care about the distilled results presented to the user).

Having arm do these lookups should be an option, but better behavior would be if only a few arm instances fetched, distilled, and cached the lookups and these compressed results were fetched from other instances. This project would include a few components:

  1. How much information can we mine from whois lookups? Can we reliably parse out the ISP? Can we determine the jurisdiction or other information that would be of interest to arm users?
  2. Are there any other lookups (besides rdns and whois) that would be of interest? An option would be higher precision locale lookups (Vidalia used to have more precise locale information, for instance).
  3. Implement the distilling, caching, and secure transmission. Results should be updated whenever a new consensus is available (resolving new relays and dropping anything that's been removed for a few hours).
  4. How can others volunteer to be mirrors? This would require directories and a reachability test.
  5. For relays fetching can (optionally) be direct, but for clients it *must* be fetched through Tor (arm directories shouldn't now who client Tor users are).
  6. Possibility make the lookup caches available via torrenting?

Client Mode Use Cases

Thus far arm has been chiefly geared for Tor relay operators. However, this doesn't need to be the case. This project would be to expand and simplify arm to make it useful for Tor's client users too. Foremost this needs more thought (and community input) about what would be helpful, but some starting points would be:

  • Populate the header with information of more use to client users when we aren't a relay. One option is to list port information from startup: Socks, Transparent pf/netfilter, and DNS listener. (idea by velope)
  • Option for selecting exit country or other manual path selection.
  • Look at Vidalia and TorK for ideas.
  • Dialog with the status of our bridges. This is currently blocked on ticket 2068. (idea by mikeperry)
  • Provide simple options, like Vidalia, for contributing back to the network ('make me a bridge/relay/exit').
  • Setup wizard, prompting the user in a very friendly fashion for what they'd like Tor to be. This could be provided if (a) Tor isn't already running or (b) is using a completely default configuration since it lacks a torrc. (idea by ioerror)
  • Have arm act as a netinstaller if Tor isn't available (fetching the right package for the platform, verifying signatures, etc)? This treads into Thandy's territory and quite likely isn't a desired feature.
  • Option for manually closing circuits

Control Port Interpretor

The control port provides far greater capabilities than just what arm uses. We should provide a prompt that allows for raw controller access, along with usability features to make it more user friendly like:

  • An IRC style help option (for instance "/help GETINFO" could provide a summary of getinfo commands, partly using the results from "GETINFO info/names")
  • Tab completion and up/down for previous commands
  • Warn and get confirmation if command would disrupt arm (for instance 'SETEVENTS')
  • 'safe' option that restricts to read-only access (start with this)
  • Aliases for useful commands like sighup and newnym
  • User friendly formatting of Tor responses and color support
  • Both integration with arm's curses interface and option to be run as a standalone application.
  • Sometimes the best solution to a user's issue is a simple control port command which begs the obvious question from them "ok... so how do I send it?". Unfortunate answer has been either "telnet, and somehow puzzle out the authentication handshake" or "um... do you know python?". These aren't good answers. A solution would be a simple script that executes single controller commands or scripts (an initial version of this has already been done). Ideally this could run independently (no TorCtl or arm dependency). For instance...
    atagar@fenrir:~/tsh$ python tsh.py -c "GETINFO ns/all" | grep caerSidi
    r caerSidi p1aag7VwarGxqctS7/fS0y5FU+s XA8SNU99CzbLEfFNtnfvpLxpcU8 2011-02-26 05:15:36 71.35.147.132 9001 0
    

Tomás has already started his own interpretor called TorSH.

Handle Multiple Tor Instances

Large relays, like Blutmagie and Amunet, run multiple instances of Tor on the same system to saturate their connections. A single instance of arm should be able to attach to all of these, and provide a screen style method for switching between them. It would be especially useful if this included an aggregate display, that showed information from all of the instances (ie, bandwidth sum, total resource (cpu/memory) usage, all connections, combined logs with indicators of which instance they came from, etc). (feature request by ioerror)

Email Summaries

Periodic client-side reports of the relay's status. This should be runnable as either a standalone application *or* arm plugin, and could include:

  • Warnings when the relay falls out of the consensus, has error, etc (like Tor Weather but also making use of client-side information)
  • Accounting notices or warning when relay bandwidth drops to zero
  • Status reports, starting with...
    • Daily log output, bandwidth history, etc
    • Graph image of the bandwidth usage
    • Aggregated port usage for exits and country of origin for clients
  • Merge in and improve the consensus tracker functionality
  • Provide an option to also do a relaying stat dump to stdout. This would just be a snapshot of the current state, but could include things like ps / connection information, descriptor and consensus info on our relay, etc. This could also benefit from terminal coloring. (feature request by StrangeCharm)

Investigate Useful APIs

Urwid is an open source python library for terminal applications. It provides a nicer interface, ideal for terminal forms and some common UI elements.

As I recall it isn't compatible with using curses directly, but checking the library for interesting capabilities and code snippets could be very helpful. For instance, the screen shots exemplify color support beyond the 8 colors (RGB + CMYK) available with curses... *very* curious how they do this and how widely higher color support is available!

Also, newer versions of arm's current dependencies (python, curses, and the control port) might have additional capabilities that would be useful. Be sure to fall back gracefully if new functionality isn't available.

Consensus Panel

This would be a command-line counterpart for consensus listings like TorStatus. TorCtl has a ConsensusTracker class that might be helpful for this though when last investigated it boosted arm's startup time by several seconds (if this is still the case then either add lazy loading or make the feature off by default). This could also provide the total network capacity and note new additions.

Feature Ideas

  • Updater, giving the user a notice if arm's out of date. This wouldn't be useful if arm was installed via its deb package.
  • Options for more detailed control of when bandwidth's contributed as discussed in ticket 1730
  • Investigate hidden service use cases and what we could do to better support them
  • Provide an option for showing cpu usage on a per-core basis. This would be similar to top when you press 1, ie cpu0 X%, cpu1 Y%, etc and would help relay operators in checking system load (feature request by Jordan)
  • Splash screen when starting up to indicate that arm is loading, which would be especially helpful when arm's taking a while to parse the man page. However, improvements to shorten the startup time will likely make this moot. (idea by Sjon)
  • Provide a warning if Tor accesses file descriptors outside a known pattern?

Trac Tickets

Below are tickets tracking issues of interest to arm.

Tor Issues

TorCtl

Packaging

Packaging

OpenWrt

Uses the opkg packaging format which could make use of arm's current deb packages. Packaging for this platform would help with the Torouter project.

Mac

No packages are currently available and, since packaging must be done on a mac I won't be addressing this platform (volunteers welcome!).

  • Homebrew - Similar to Macports (below), arm is available to build on OS X using Homebrew.
  • macport - Build-from-source distribution method (like BSD portinstall). This has been suggested by several people.
  • dmg - Most conventional method of software distribution on mac. This is just a container (no updating/removal support), but could contain an icon for the dock that starts a terminal with arm. This might include a pkg installer.
  • mpkg - Plugin for distutils. Like most mac packaging, this can only run on a mac. It also requires setuptools. Nice instructions for making a pkg with an icon for swing apps is available here (there's probably parallels for python).

Windows

Arm, in theory, should run on Windows within Cygwin. Native Windows support, however, requires a couple things:

  • a curses counterpart (required to render the CLI)
  • ps/netstat lookup counterparts (not needed to run, but fetches a lot of the information)

Just about all of arm's curses usage is abstracted away behind its Panel class so, given a library with similar capabilities, adding Windows support to the interface shouldn't be too hard. The forerunner for this seems to be PDCurses which might, someday, be added to Python 3 as per Python ticket 2889.

Other options include Console, WConio, and wcurses which each provide toolkits with a subset of curses' functionality though none of them look particularly promising. The wcurses library, for instance, is completely inactive and carries the warning "This module does NOT yet implement the full API that the UNIX curses extension does, and what it does implement is buggy".

The ps/netstat lookup counterparts, however, is a far easier problem to solve...

  • The ResourceTracker of src/util/sysTools.py could be easily expanded to use whatever method is preferable on Windows for fetching the cpu/memory usage of the Tor process (TaskList?).
  • Fixing netstat lookups should just consist of teaching connections.py to recognize the netstat formatting used on Windows.

Release Checklist

  1. Run a sanity check over major changes:
    • pylint --indent-string=" " --disable=C,R interface/foo.py | less
    • double check init.py, setup.py, and README for added or removed files
  2. Update release notes
    • Spellcheck notes and copy them to the site (atagar.com/arm/log.php)
    • Copy rendered release notes into the ChangeLog
  3. Check into release and upload the signed tarball
    • git checkout release
    • git merge master
    • bump the version
      • release/src/version.py
      • resources/deb-prep.sh
      • resources/build/debian/changelog
    • git commit -a -m "Arm release 1.4.1"
      • note the commit number in the site log and ChangeLog
    • git tag -u 9ABBEEC6 -m "arm release 1.4.2.0" 1.4.2.0 d0bb81a
    • git checkout packaging
    • ./export
    • tar cjf arm-1.4.1.0.tar.bz2 arm
    • gpg --detach-sig --armor arm-1.4.1.0.tar.bz2
    • copy tarball and sig to the webserver (both to resources and static)
    • make the rpm and upload that too
    • update links and bump the release in download.php and index.php
      • double check that there aren't any broken links
    • request that the files be uploaded to archive.torproject.org (example ticket)
  4. Release notifications
  5. Revise this wiki, adding development plans for the next release

Package Repositories

After a week notices about the new release are sent to package maintainers. This gap is to give some time in case important problems are discovered and need hotfixes. However, many of the package maintainers monitor or-talk and tend to bump their versions ahead of time. :P

Debian

  • Contact: weasel (Peter) and dererk
  • Initial Release
  • Update Instructions:
    • git checkout packaging
    • update build/debian/changelog
    • ./deb-prep.sh
    • cd release_deb
    • ./debian/make-deb
    • copy the results to the webserver, example
    • send the dsc link to weasel

Red Hat

  • Contact: Juan
  • Update Instructions:
    • update resources/build/redHat/MANIFEST
    • ./rpm-prep.sh
    • cd release_rpm
    • ./redhat/make-rpm
    • sign the noarch rpm and copy that and sig for the download page

Gentoo

ArchLinux

  • Contact: Spider.007
  • Initial Release
  • Update Instructions:
    • go to aur.archlinux.org
    • select "Out-of-date" for the package

Slackware

  • Contact: pyllyukko
  • Initial Release
  • Update Instructions:
    • If this falls behind then email pyllyukko (this might be due to being in the SlackBuilds QA backlog)

FreeBSD

  • Contact: Carlo Strub
  • Initial release
  • Update Instructions:
    • If this falls behind then email Carlo
Last modified 8 months ago Last modified on Sep 4, 2013 11:13:31 AM