wiki:doc/arm

Version 91 (modified by karsten, 6 years ago) (diff)

Fix internal wiki link

Arm Development Wiki

Bugs

Easy

Interested in hacking on arm? The following is a great place to start. These are small, decently easy bugs and improvements I haven't had time to address. If you get stuck at all then please let me know and I'd be happy to help!

  • Use SAVECONF instead of writing the file directly when we're overwriting our current torrc
  • The default connection resolver isn't configurable
  • When running as a client (no ORPort) and Tor stops the header panel doesn't say so
  • The config write dialog (ie, the one for saving the config) has its a misaligned border when it's smaller than the top detail section.
  • Simple Improvements
    • Graph lacks a config option for moving right to left (feature request by ioerror and rdegraaf)
    • Tab completion for input fields that expect a filesystem path
    • Make arm's configuration values dynamically editable via a listener architecture, with an option to save the current configuration to the armrc.
    • Add arm's configuration options to its man page. Ideally these wouldn't be hardcoded, but instead added by a script that fetches options from the source.
    • Write unit tests for backend utilities (uiTools has some particularly good candidates)
    • Check for parity of our exitability check with Mike's implementation in TorCtl (his ExitPolicyLine and will_exit_to function match up with our ExitPolicy class and isExitingAllowed/getExitPolicy functions)
    • The current arm configuration options were added in an ad hoc fashion using a naming scheme a little reminiscent of Firefox's about:config options. Now that we have a general idea what config options arm will use we can come up with a nicer, more user friendly naming hierarchy.
  • Prompt the user to open the control port if tor is running but it's closed at startup (either a custom prompt or offer to modify the torrc on their behalf, idea by Thomas Schreiber), a better plan for the autoconfiguration is...
    • Check the arguments of the tor process to see if it was given a -f arg.
    • Look in all of the common default torrc locations to see if there's a file found. If either none or multiple are found then abort.
    • If the above have narrowed the torrc to a single likely location then prompt the user to...
      • set controller connection for this session by reverting the change when arm quits (can this be done via a temporary copy?)
      • persist the change to the torrc
      • abort

Harder

  • Log deduplication is currently an n^2 operation. Hence it can't handle large logs (for instance, when at the DEBUG runlevel). Currently we're timing out the function if it takes too long, but a more efficient method for deduplication would be preferable.
  • Tor configuration values cached in 'util/torTools.py' don't take SETCONF events by other controllers (like Vidalia) into account. This is blocked on ticket 1692.
  • The log prepopulation fails to limit entries to just the current Tor instance if the file isn't logged to at the NOTICE level. A fix is to use the timestamps to see if it belongs to the attached Tor instance or not. But this requires Tor's uptime, which is blocked on the implementation of proposal 173.
  • Evidently some terminal renderers expect Ctrl+L to redraw the screen... respect this? (requested by Thomas Schreiber, ticket)

1.4.X Plans

Next Release

  • Controller and Popups
    • Descriptor Details
    • File Descriptors
    • Help
    • Input Popups (sorting, list selection, etc)
    • Migrate page handlers out of the controller
    • Controller rewrite
  • Connection Panel Family Entries
    • Figure out a method of making it obvious these aren't real connections
    • Give warning and display in red if...
      • family declarations aren't mutual
      • relay isn't in the consensus
      • check if relay is alive via a VERSION cell handshake?
      • multiple family entries resolve to the same entry (could use the fingerprint for one entry, and the nickname for another)
      • provide a warning if named family entries lack the named flag
  • Optional autoconnection on arm startup
  • Allow arm to resume when Tor is restarted
  • Look for performance bottlenecks that can be improved (refresh rate, cpu usage, and startup time)
  • Drop the deprecated connection panel from the codebase
  • Escaping function for util/panel's formatted strings (or maybe just drop the feature entirely? is it still used?)
  • The cpu usage spikes for scrollable content when the key's held. Tried coalescing the events without success.
  • Warn if there's exit connections (other than DNS) when we're strictly a non-exiting relay
  • Drop deprecated config options (like 'group') from the listing (idea thanks to NightMonkey)
  • More descriptive controller password prompt (suggestion by weasel)
  • When extra room is available, the config panel should expand its value field
  • Fallback when unable to limit connection lookups to just Tor, instead listing anything in both the lookup and consensus
  • Config options to show or hide individual panels
  • Run Heapy to determine the memory usage of each individual panel and take care of anything wasteful. When I last checked (3/1/11) the baseline memory usage of arm was 4 MB, half of it as strings. However, Heapy doesn't tell where in the application this memory is being used so I'll need to wait until we can cut out portions of the codebase to narrow this down.
  • Optionally display unproxied connections in red (to help catch leaks)
  • Provide the client country or exiting port statistics (the former for bridges/guards, and the later for exits). GETINFO options are available for these in aggregate (feature request by waltman and ioerror).
  • Pick apart applications like iftop and pktstat to see how they get per-connection bandwidth usage (forum discussion)
  • Look into CAPs to get around permission issues for the connection listing (check the 'capabilities' man page and this article)
  • Write summaries for new config options, example
  • Config option for preconfigured log regexes
  • List nicknames on main connection page if able
  • fix: If being used as a client then don't list the nickname source as 'unnamed'
  • Option (menu only?) for quitting the tor process...
    • kill tor - immediately ends the process if able (check for permissions before providing the option?)
    • graceful shutdown (relays only) - issue a "killall -s SIGINT tor"
    • cancel
  • Make use of new Tor GETINFO options when available (like BW total)
  • Bug: Config options with long names (ex. UpdateBridgesFromAuthority) are causing he layout on the config panel to be shifted

Pending Release Notes

The following are the release notes for completed portions of the next release.

--/--/-- - version 1.4.3
This completes refactoring of the arm codebase, including the controller, handlers, and popups.

  • added: option for reconnecting to tor if it's been shut down and restarted
  • added: fetching the download and upload bandwidth totals from Tor if it's available
  • added: reintroduced the descriptor popup for the new connection panel
  • added: file descriptor usage in the header panel when running low
  • added: logging a notice when file descriptor usage is high
  • added: moved arm's codebase to git, with helper scripts to replace svn:externals and export
  • change: renaming our process from "python src/starter.py <args>" to "arm <args>" (thanks to help from ioerror)
  • change: hiding connection init latency, dropping arm's startup time from 0.84 seconds to 0.14 (83% improvement)
  • change: dropping the file descriptor popup (it was both unused and inaccurate)
  • change: dropping deprecated connection panel from the codebase
  • change: using SAVECONF rather than writing directly when able
  • change: closing all message prompts when a key is pressed
  • fix: all pid resolution was failing on macs, causing much of arm's functionality to fail
  • fix: dropping straight to lsof on macs to avoid fallback process, which was both lengthy and raised warnings
  • fix: misparsing circut paths for Tor < 0.2.2.1 (caught by asn)
  • fix: pressing enter on an empty connection page caused crash (caught by asn, ticket)
  • fix: edited config entries weren't displaying their new value
  • fix: ns/desc lookup failures could crash descriptor popup (caught by asn)
  • fix: autogenerated Nickname field caused a torrc validation false positive
  • fix: crashing issue when querying locales with an unavailable geoip file
  • fix: misparsing config option types for some older tor versions
  • fix: stacktrace from unjoined thread at shutdown
  • fix: intermediate graphing bounds were either inaccurate or missing
  • fix: man page typo (caught by tagnaq)
  • fix: --debug flag broke if config-text unavailable (caught by trogdor)
  • fix: man page had author's hard coded home rather than ~ (caught by kuhkatz)

Roadmap

  • 1.4.3 - finish rewrite
    • see above
  • 1.4.4 - menus
  • 1.4.5 - control port interpretor
    • Provide coloring and highlighting comparable to the old descriptor popup for ns/* and desc/* queries
  • 1.4.6 - display sparklines at the top of the controller for entry/exit statistics on locale and port usage

Projects

Circuit Details

We can passively provide consensus and locale information on the relays being used in our circuits, but most of the more interesting information requires active data collection (for instance rdns and whois lookups for the hostname, ISP, etc). We can't do this on-demand because it would leak our circuits to resolvers and our ISP (hopefully it's needless to say why this is horribly bad).

An alternative would be to actively collect this data on every relay in the consensus, caching the results, and only displaying information form that cache. This has the obvious disadvantage of needing thousands of lookups which would both clog the user's connection and take a very long time. It's especially wasteful since the information we receive back is mostly thrown away (we just care about the distilled results presented to the user).

Having arm do these lookups should be an option, but better behavior would be if only a few arm instances fetched, distilled, and cached the lookups and these compressed results were fetched from other instances. This project would include a few components:

  1. How much information can we mine from whois lookups? Can we reliably parse out the ISP? Can we determine the jurisdiction or other information that would be of interest to arm users?
  2. Are there any other lookups (besides rdns and whois) that would be of interest? An option would be higher precision locale lookups (Vidalia used to have more precise locale information, for instance).
  3. Implement the distilling, caching, and secure transmission. Results should be updated whenever a new consensus is available (resolving new relays and dropping anything that's been removed for a few hours).
  4. How can others volunteer to be mirrors? This would require directories and a reachability test.
  5. For relays fetching can (optionally) be direct, but for clients it *must* be fetched through Tor (arm directories shouldn't now who client Tor users are).
  6. Possibility make the lookup caches available via torrenting?

Client Mode Use Cases

Thus far arm has been chiefly geared for Tor relay operators. However, this doesn't need to be the case. This project would be to expand and simplify arm to make it useful for Tor's client users too. Foremost this needs more thought (and community input) about what would be helpful, but some starting points would be:

  • Populate the header with information of more use to client users when we aren't a relay. One option is to list port information from startup: Socks, Transparent pf/netfilter, and DNS listener. (idea by velope)
  • Provide a method for requesting a new identity (ie, send a newnym signal). This control for doing this and the current circuit dirtiness duration, could be displayed in the header panel for non-relays. Vidalia provides an indicator for when a newnym isn't an options (since it's ignored by Tor if made too frequently), so we should check the Tor code for what this duration is, and ask chiiph if Vidalia has a smarter method for detecting when a newnym will be respected.
  • Option for selecting exit country or other manual path selection.
  • Look at Vidalia and TorK for ideas.
  • Dialog with the status of our bridges. This is currently blocked on ticket 2068. (idea by mikeperry)
  • Provide simple options, like Vidalia, for contributing back to the network ('make me a bridge/relay/exit').
  • Setup wizard, prompting the user in a very friendly fashion for what they'd like Tor to be. This could be provided if (a) Tor isn't already running or (b) is using a completely default configuration since it lacks a torrc. (idea by ioerror)
  • Have arm act as a netinstaller if Tor isn't available (fetching the right package for the platform, verifying signatures, etc)? This treads into Thandy's territory and quite likely isn't a desired feature.

The current hotkey interface is clunky and very unfriendly for new users. Precious few users ever hit 'h' to see what options are available, and sometimes users never even figure out that the interface has multiple pages (whenever I hear this I die a little inside...). We should keep the hotkey interface as an option, but default to something more intuitive, like menus. These should optionally be hidden (maybe whenever the user's idle?) to avoid clutter. This would also give a method of adding even more help functionality (for instance descriptions for the relay flags). Charming python has an example for doing this in curses.

Control Port Interpretor

The control port provides far greater capabilities than just what arm uses. We should provide a prompt that allows for raw controller access, along with usability features to make it more user friendly like:

  • An IRC style help option (for instance "/help GETINFO" could provide a summary of getinfo commands, partly using the results from "GETINFO info/names")
  • Tab completion and up/down for previous commands
  • Warn and get confirmation if command would disrupt arm (for instance 'SETEVENTS')
  • 'safe' option that restricts to read-only access (start with this)
  • Aliases for useful commands like sighup and newnym
  • User friendly formatting of Tor responses and color support
  • Both integration with arm's curses interface and option to be run as a standalone application.
  • Sometimes the best solution to a user's issue is a simple control port command which begs the obvious question from them "ok... so how do I send it?". Unfortunate answer has been either "telnet, and somehow puzzle out the authentication handshake" or "um... do you know python?". These aren't good answers. A solution would be a simple script that executes single controller commands or scripts (an initial version of this has already been done). Ideally this could run independently (no TorCtl or arm dependency). For instance...
    atagar@fenrir:~/tsh$ python tsh.py -c "GETINFO ns/all" | grep caerSidi
    r caerSidi p1aag7VwarGxqctS7/fS0y5FU+s XA8SNU99CzbLEfFNtnfvpLxpcU8 2011-02-26 05:15:36 71.35.147.132 9001 0
    

Tomás has already started his own interpretor called TorSH.

Handle Multiple Tor Instances

Large relays, like Blutmagie and Amunet, run multiple instances of Tor on the same system to saturate their connections. A single instance of arm should be able to attach to all of these, and provide a screen style method for switching between them. It would be especially useful if this included an aggregate display, that showed information from all of the instances (ie, bandwidth sum, total resource (cpu/memory) usage, all connections, combined logs with indicators of which instance they came from, etc). (feature request by ioerror)

Email Summaries

Periodic client-side reports of the relay's status. This should be runnable as either a standalone application *or* arm plugin, and could include:

  • Warnings when the relay falls out of the consensus, has error, etc (like Tor Weather but also making use of client-side information)
  • Accounting notices or warning when relay bandwidth drops to zero
  • Status reports, starting with...
    • Daily log output, bandwidth history, etc
    • Graph image of the bandwidth usage
    • Aggregated port usage for exits and country of origin for clients
  • Merge in and improve the consensus tracker functionality
  • Provide an option to also do a relaying stat dump to stdout. This would just be a snapshot of the current state, but could include things like ps / connection information, descriptor and consensus info on our relay, etc. This could also benefit from terminal coloring. (feature request by StrangeCharm)

Investigate Useful APIs

Urwid is an open source python library for terminal applications. It provides a nicer interface, ideal for terminal forms and some common UI elements.

As I recall it isn't compatible with using curses directly, but checking the library for interesting capabilities and code snippets could be very helpful. For instance, the screen shots exemplify color support beyond the 8 colors (RGB + CMYK) available with curses... *very* curious how they do this and how widely higher color support is available!

Also, newer versions of arm's current dependencies (python, curses, and the control port) might have additional capabilities that would be useful. Be sure to fall back gracefully if new functionality isn't available.

Consensus Panel

This would be a command-line counterpart for consensus listings like TorStatus. TorCtl has a ConsensusTracker class that might be helpful for this though when last investigated it boosted arm's startup time by several seconds (if this is still the case then either add lazy loading or make the feature off by default). This could also provide the total network capacity and note new additions.

GUI Frontend

Arm has several unique features, some of the most interesting being its connection listing (correlating netstat results against the Tor consensus) and configuration editor (a quick method for editing Tor's config, with information pulled from the control port and man page). However, since arm is a command line controller it's of limited appeal to certain sets of users. This project would be to build a GTK or Qt frontend for the controller, providing similar features set but with a windowed interface.

The vast majority of arm's more interesting functionality lies in its backend utilities, so there should be little to no work decoupling the CLI from its backend. Instead, this project would mostly be UI hacking and experimentation, trying different interfaces to find something that's elegant and simple, but matches the information found in the current terminal application.

Feature Ideas

  • Updater, giving the user a notice if arm's out of date. This wouldn't be useful if arm was installed via its deb package.
  • Options for more detailed control of when bandwidth's contributed as discussed in ticket 1730
  • Investigate hidden service use cases and what we could do to better support them
  • Provide an option for showing cpu usage on a per-core basis. This would be similar to top when you press 1, ie cpu0 X%, cpu1 Y%, etc and would help relay operators in checking system load (feature request by Jordan)
  • Splash screen when starting up to indicate that arm is loading, which would be especially helpful when arm's taking a while to parse the man page. However, improvements to shorten the startup time will likely make this moot. (idea by Sjon)
  • Provide a warning if Tor accesses file descriptors outside a known pattern?

Trac Tickets

Below are tickets tracking issues of interest to arm.

Tor Issues

Bugs

Features

TorCtl Issues

Packaging

OpenWrt

Uses the opkg packaging format which could make use of arm's current deb packages. Packaging for this platform would help with the Torouter project.

Mac

No packages are currently available and, since packaging must be done on a mac I won't be addressing this platform (volunteers welcome!).

  • macport - Build-from-source distribution method (like BSD portinstall). This has been suggested by several people.
  • dmg - Most conventional method of software distribution on mac. This is just a container (no updating/removal support), but could contain an icon for the dock that starts a terminal with arm. This might include a pkg installer.
  • mpkg - Plugin for distutils. Like most mac packaging, this can only run on a mac. It also requires setuptools. Nice instructions for making a pkg with an icon for swing apps is available here (there's probably parallels for python).

Windows

Arm, in theory, should run on Windows within Cygwin. Native Windows support, however, requires a couple things:

  • a curses counterpart (required to render the CLI)
  • ps/netstat lookup counterparts (not needed to run, but fetches a lot of the information)

Just about all of arm's curses usage is abstracted away behind its Panel class so, given a library with similar capabilities, adding Windows support to the interface shouldn't be too hard. The forerunner for this seems to be PDCurses which might, someday, be added to Python 3 as per Python ticket 2889.

Other options include Console, WConio, and wcurses which each provide toolkits with a subset of curses' functionality though none of them look particularly promising. The wcurses library, for instance, is completely inactive and carries the warning "This module does NOT yet implement the full API that the UNIX curses extension does, and what it does implement is buggy".

The ps/netstat lookup counterparts, however, is a far easier problem to solve...

  • The ResourceTracker of src/util/sysTools.py could be easily expanded to use whatever method is preferable on Windows for fetching the cpu/memory usage of the Tor process (TaskList?).
  • Fixing netstat lookups should just consist of teaching connections.py to recognize the netstat formatting used on Windows.

Release Checklist

  1. Run a sanity check over major changes:
    • pylint --indent-string=" " --disable=C,R interface/foo.py | less
    • double check init.py, setup.py, and README for added or removed files
  2. Update release notes
    • Spellcheck notes and copy them to the site (atagar.com/arm/log.php)
    • Copy rendered release notes into the ChangeLog
  3. Check into release and upload the signed tarball
    • git checkout release
    • git merge master
    • bump the version
      • release/src/version.py
      • resources/deb-prep.sh
      • resources/build/debian/changelog
    • git commit -a -m "Arm release 1.4.1"
      • note the commit number in the site log and ChangeLog
    • git tag -u 9ABBEEC6 -m "arm release 1.4.2.0" 1.4.2.0 d0bb81a
    • git checkout packaging
    • ./export
    • tar cjf arm-1.4.1.0.tar.bz2 arm
    • gpg --detach-sig --armor arm-1.4.1.0.tar.bz2
    • copy tarball and sig to the webserver (both to resources and static)
    • update links and bump the release in download.php and index.php
      • double check that there aren't any broken links
  4. Release notifications
  5. Revise this wiki, adding development plans for the next release

Package Repositories

After a week notices about the new release are sent to package maintainers. This gap is to give some time in case important problems are discovered and need hotfixes. However, many of the package maintainers monitor or-talk and tend to bump their versions ahead of time. :P

Debian

  • Contact: weasel (Peter)
  • Initial Release
  • Update Instructions:
    • update resources/build/debian/changelog
    • ./resources/deb-prep.sh
    • cd release_deb
    • ./debian/make-deb
    • copy the results to the webserver, example
    • send the dsc link to weasel
    • sign deb and copy that and sig for the download page

Red Hat

  • Contact: None
  • Update Instructions:
    • update resources/build/redHat/MANIFEST
    • ./resources/rpm-prep.sh
    • cd release_rpm
    • ./redhat/make-rpm
    • copy the results to the webserver
    • sign rpm and copy that and sig for the download page

Gentoo

ArchLinux

  • Contact: Spider.007
  • Initial Release
  • Update Instructions:
    • go to aur.archlinux.org
    • select "Out-of-date" for the package

Slackware

  • Contact: pyllyukko
  • Initial Release
  • Update Instructions:
    • If this falls behind then email pyllyukko (this might be due to being in the SlackBuilds QA backlog)