Tor Test Networks using Chutney: A Guide
A guide to creating Tor test networks with chutney. (This page is an early draft outline of the documentation and functionality we want to have.)
Features which are not yet implemented are marked NYI, with a corresponding Trac ticket number and link (if it exists).
Chutney README
The chutney README gives detailed descriptions and command-line examples for each chutney feature:
https://gitweb.torproject.org/chutney.git/tree/README
Testing an installed Tor version
Run a Single Test
Get the latest version of chutney:
git clone !https://git.torproject.org/chutney.git
If you already have tor installed, you can run a test network:
chutney/tools/test-network.sh
Testing while Building Tor from Source
Run a Single Test
git clone !https://git.torproject.org/tor.git
git clone !https://git.torproject.org/chutney.git
cd tor
sh autogen.sh
./configure
make test-network
The ./configure step will fail if there are missing libraries, and will indicate which those are. Run ./configure again after installing all the necessary libraries.
Setting the CHUTNEY_PATH environmental variable
Recent versions of Tor (0.2.7.3-alpha (?) and later) will autodetect the $CHUTNEY_PATH if the chutney directory is next to the tor directory (#16903 (moved)).
If your tor and chutney source directories are not side-by-side, you'll neeed to set the CHUTNEY_PATH environmental variable to the path to the chutney directory (not the script itself).
You can do this for one command:
- CHUTNEY_PATH=/path/to/chutney/directory make test-network (in many shells, or see this article for a more detailed explanation)
Or while the shell is running:
- export CHUTNEY_PATH=/path/to/chutney/directory (in bash and similar shells, or see this article for tcsh / csh, or consult your shell's documentation)
Or for all future shells:
Test Functionality
Recent versions of Tor (0.2.7.3-alpha (?) and later) run a comprehensive test of most functionality, including authorities, bridges, clients, exits, and hidden services (but no IPv6 for compatibility reasons). See tickets #16945 (moved) (tor) & #16946 (moved) (chutney). Older versions of Tor run a simpler test of authorities, clients, and exits.
Resolving Test Failures
Chutney occasionally fails to bootstrap or verify. Sometimes, and on some platforms / machines, bootstrap is slower than we expect (25 seconds in 0.2.6 - 0.2.7.2-alpha), causing intermittent failures.
Recent tor versions (#16952 (moved), merge pending, 0.2.7.3-alpha?):
- Retry boostrap (configure and launch) once on failure, configurable via $CHUTNEY_RECONFIGURES or --reconfigures
- Retry verify every 5 seconds from $BOOTSTRAP_TIME (now 20s) to $MAX_BOOTSTRAP_TIME (60s), configurable via --time and --max-time
These changes make src/test/test-network.sh robust against intermittment failures.
However, sometimes chutney fails to launch due to a persistent error. The most common issue is having a previous chutney network active. Old chutney networks are typically killed by chutney/tools/bootstrap-network.sh using chutney stop, but sometimes this doesn't happen. The tor processes launched by chutney can be found using a command like ps auxwww | grep chutney | grep tor. Ensure each pid doesn't belong to a process you want, like a system tor or an unrelated process, then kill each chutney tor pid using kill. Alternately, each tor instance's pid files can be found in the chutney/net/nodes directory. (Old nodes directories are renamed using an increasing numeric timestamp as a suffix.)
NYI: Better yet, make these instructions part of the chutney failure message when some tor processes fail to launch. (#17002 (moved))
Run Comprehensive Tests
git clone !https://git.torproject.org/tor.git
git clone !https://git.torproject.org/chutney.git
cd tor
sh autogen.sh
./configure
make test-network-all
make test-network-all runs comprehensive tests. It reports results PASS / SKIP / FAIL like the existing make check output. (#16953 (moved))
Base Test Suite:
- basic-min: authorities, exit relays, exit client
- bridges-min: authorities, bridge authority, bridge, bridge-exit client
- hs-min: authorities, non-exit relays, hidden service, hs client
- single-onion: authorities, non-exit relays, single onion service, hs client
IPv6 Test Suite: only run if ping6 ::1 works.
- bridges+ipv6-min: authorities, bridge authority, ipv4/ipv6 bridge, ipv6 bridge client
- ipv6-exit-min: authorities, ipv4/ipv6 exit, ipv4 exit client asking for ipv6 address
- hs-ipv6: authorities, ipv4/ipv6 non-exit relays, ipv6 hidden service, ipv4 hs client, ipv6 hs client
- single-onion-ipv6: authorities, ipv4/ipv6 non-exit relays, ipv6 single onion service, ipv4 hs client, ipv6 hs client
NYI: #17011 (moved) - test actual IPv6 connectivity (not just IPv4 on IPv6 instances); #16971 (moved) - remove requirement for IPv6 DNS on IPv6 exits.
Mixed Version Test Suite: only run if a stable version of tor exists. NYI: try harder to find one (#17015 (moved))
- mixed: mixed authorities, mixed exit relays, mixed clients
Log (.log) and result (.trs) files are available in the test_network_log directory in the build directory.
Testing using the Tor Test Network Script
tor/src/test/test-network.sh
List script arguments and corresponding chutney environmental variables in a table
List templates in a table
Describe performance testing results, and what they do and don't mean (localhost, no network hardware involved, more parallelism than you'll get as a relay)
Testing using Chutney Commands
chutney/chutney
Manually configure a network, a minimal set of steps similar to what tor/src/test/test-network.sh does
List chutney commands in a table
Reference environmental variables and templates from previous section
Other Usability Improvements
NYI: There are some minor changes to chutney which would make it much more usable, such as sensible error messages on failure. Let's work out which of these we can knock over quickly. See the full list of proposed chutney changes in #16949 (moved).
Modifying Chutney Tests
The torrc templates and test network configurations included with chutney can be modified using any text editor.
Modifying torrc Templates
chutney/torrc_templates
The torrc template files use the tor options syntax, with the addition of a few templating commands, such as file includes. Some template expansions are performed to make configuration easier. The most important template expansion creates a list of test network authorities in every torrc file.
List templating commands
List "magic" expansion operations, including the one done in the launch script
Modifying Test Network Configurations
chutney/networks
How this currently works:
- each network is a separate Python source file
- tags identify clients?, exits and hidden services so chutney can run tests for each client and each hidden service (exits are chosen at random) (there are some subtleties I'm missing here about different kinds of tags)
- other things in current files I've forgotten
How this could work: (NYI: #16954 (moved))
Refactor the existing chutney templates so it's easier to mix and match particular authority, middle relay, client (standard, bridge, IPv6 bridge) and endpoint (exit, hidden service) nodes. Also make version mixing easier. Have a replaceable minimal default? 3 authorities, 1 client, 1 bridge client, 1 bridge, 1 middle, 1 exit, 1 hs
- each network includes 3 authorities, 1 client or bridge client, 1 exit or hs, and however many middles are required to do a fast bootstrap (assuming reachability)
- work out a way to select between client / bridge client, and exit or hs endpoints
- create a way to do a full bootstrap (no assuming reachability) by flipping one setting (change torrc options, add middle relays to some minimum of auth + exit + middle: 4? 5?)
- work out a way to add IPv6 bridges, old tor versions, and other features I'm forgetting
- work out a way to scale network size by multiplication, either of all nodes, or of particular node roles (start point, relay, end point), or of individual node configurations
- work out what syntax Python needs to include one configuration in another
Maybe it could look like this:
- core: authorities, non-exit relays?
- client: client, bridge, IPv6 bridge (choose at least one)
- endpoint: exit, hs (choose at least one)
- bootstrap: fast or full (choose one)
- version: built, system (choose at least one)
Then combined to give us all the networks that currently exist, without needing to cut and paste between network configurations.
chutney/net
You can also allow chutney to configure a network, and then modify the generated torrc files yourself. (List commands with reference to chutney command section above.)