- The Tor Browser Hacking Guide
- Ways to Build Tor Browser
- Running Multiple Tor Browsers
- Tor Browser Team Communication and Organizational Patterns
- Design and Core Components
Debugging the Tor Browser
- Enabling debug logs in extensions
- Using gdb
- QA and Testing
- Requesting testing of a custom build using the tbb-qa.yml file
- Other Resources
The Tor Browser Hacking Guide
Welcome to the Official Tor Browser Hacking Guide. This page is meant to give you an overview of how to get started with Tor Browser development.
This page covers building the browser, debugging the browser, how we communicate and use trac to organize our issues and development process, what goes into the browser itself, and also provides links to other development resources and information.
Ways to Build Tor Browser
The first thing you probably want to know is how to obtain the very latest (and sometimes greatest) Tor Browser builds, and how to build your own version of the Tor Browser from our source code. Well wait no further.
Currently, nightly builds are available courtesy of Linus Nordberg. They are available in the tbb-nightly-* subdirectories of his homedirectory on people.torproject.org.
As with all of our builds, it should be possible to reproduce byte-for-byte identical versions of all of those binaries from source. To learn how to do that, read on.
Building Official Tor Browser Release Binaries
Our build system is based on Gitian, which was initially developed by the Bitcoin community. Gitian is a wrapper around Ubuntu's python-vm-builder and associated virtualization tools that helps to provide a clean, controlled, reproducible build environment in a fully automated fashion. We further wrap Gitian with our own helper scripts that download and authenticate inputs, and automate building and assembling each component piece of the browser into a final set of output packages for Linux, MacOS, and Windows.
This system enables us to provide secure, verifiable, byte-for-byte reproducible builds to ensure the integrity of our binaries and to protect the build process from compromise. We have written a pair of blog posts that describe in more detail why this is important, and the technical details behind how this is achieved, if you are curious.
To build the Tor Browser, you need an Ubuntu machine or VM. Once that is done, check out a copy of the builder repo with:
mkdir tor-browser-build cd tor-browser-build git clone https://git.torproject.org/builders/tor-browser-bundle.git cd tor-browser-bundle/gitian
After that, you should be able to run 'make', 'make beta', 'make alpha', or 'make nightly' to build the entire bundle for all three platforms. The build scripts will detect any additional packages or configuration you need to perform on your system.
Inside that directory, you will also see a README.build file with further information, should you run into any issues. In particular, the initial VM setup process is still somewhat fragile. In some cases your initial VM setup may fail or run into other issues, and you may need to restart your build with 'killall qemu-kvm'. See that README for more details on diagnosing and correcting these issues.
We also have a wiki page specific to building with Gitian that may be useful in helping you work through certain esoteric errors and failure conditions.
Reproducing an Existing Build
Every Tor Browser build includes a pair of files that describe exactly what versions of components went into making it, to allow easy reproducibility. These two files are located in the "Docs/sources" subdirectory of the Tor Browser destination directory.
To reproduce an existing bundle, inspect bundle.inputs and obtain the tor-browser-bundle.git commit hash describing the commit to build from.
Once you have this hash, you should be able to run something like the following commands:
cp versions tor-browser-build/tor-browser-bundle/gitian/versions cd tor-browser-build/tor-browser-bundle/gitian git checkout 1929fd5db0802782066d52512d4c2a0fef144ed6 make
If your have a beta, alpha, or nightly build, you will need to place the versions file in the appropriate place (versions.beta, versions.alpha, or versions.nightly) and run the corresponding make target ('make beta', 'make alpha', or 'make nightly') instead of using just 'versions' and 'make' (which are for a stable series build).
Be aware that this process is not fully future-proof. In particular, if Ubuntu has updated their development tool chain since the bundles have been built, you may encounter differences between your resulting bundles and the original binaries. This should be rare, however, as we use only the "Long Term Support" versions of Ubuntu in our build VMs. The only reason they should change the toolchain is in the event of serious security issues in the development tools themselves.
This section is mostly of interest to developers making changes to Tor Browser or its components, and who wish to test their patches.
Building Just Firefox
If you want to just develop on the browser without messing with gitian (for example, to have incremental rebuilds of just the files you modify), the best way is to build a new browser distribution and copy the result over an existing Tor Browser directory. Clone the Tor Browser repo.
To build it, from the top directory of the tor-browser repo, do:
# Install a subset of the packages listed in # https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/descriptors/linux/gitian-firefox.yml#l8 # Mostly, we don't need `libiw-dev` or `faketime`, but we *do* need `pkg-config`. sudo torsocks apt-get install \ --no-install-suggests --no-install-recommends \ zip \ unzip \ libglib2.0-dev \ libgtk2.0-dev \ libdbus-1-dev \ libdbus-glib-1-dev \ yasm \ libasound2-dev \ libcurl4-openssl-dev \ libxt-dev \ mesa-common-dev \ autoconf \ autoconf2.13 \ libtool \ hardening-wrapper \ libgstreamer-plugins-base0.10-dev \ pkg-config \ g++ \ libpulse-dev # Generate the configure scripts: make $CONFIGURE_ARGS -f client.mk configure CONFIGURE_ARGS="--with-tor-browser-version=4.5a4 --enable-update-channel=alpha" # And... compile: make $MAKEOPTS -f client.mk build make -C obj-* package INNER_MAKE_PACKAGE=true # Point the INSTDIR at an existing TBB directory: export INSTDIR="$HOME/tbb/test/tor-browser_en-US" # Move the compiled firefox on top of the old TBB browser dir: cp -a obj-*/dist/firefox/* $INSTDIR/Browser/ # If you want a smaller binary to copy into a vm/other machine for tests: strip --strip-all $INSTDIR/Browser/* rm -f $INSTDIR/Browser/firefox-bin
If that completes successfully, then your fresh build of Firefox should exist at $INSTDIR/Browser/firefox. Free the lizard.
Building Just Tor Launcher Or Torbutton
If you are only modifying one of our support addons (such as Tor Launcher or Torbutton) then you only need to create a new XPI and then copy it into an existing bundle.
For Tor Launcher:
# Point the INSTDIR at an existing TBB directory: export INSTDIR="$HOME/tbb/test/tor-browser_en-US" git clone https://git.torproject.org/tor-launcher.git cd tor-launcher # do some work make package # should give you an xpi under ./pkg. Note the full filename and version in the output cp pkg/tor-launcher-0.2.7.6.xpi $INSTDIR/Browser/TorBrowser/Data/Browserfirstname.lastname@example.org
# Point the INSTDIR at an existing TBB directory: export INSTDIR="$HOME/tbb/test/tor-browser_en-US" git clone https://git.torproject.org/torbutton.git cd torbutton # do some work ./makexpi.sh # Will create an xpi in pkg, but you need to find the version for the next step cp pkg/torbutton-126.96.36.199.xpi $INSTDIR/Browser/TorBrowser/Data/Browseremail@example.com
Now, you should be able to start Tor Browser in $INSTDIR normally. To run multiple copies at once, see the next section.
Running Multiple Tor Browsers
Now that you have a developer build (or several builds), you probably want to know how to run more than one of them at once. There are two main ways to do this: reusing an existing Tor process, and launching Tor process on a new pair of SOCKS and Control ports.
Using an Existing Tor Process
Let's assume that Tor process is listening on port 9150 for SOCKS, and 9151 for the control port.
export TOR_SOCKS_PORT=9150 export TOR_CONTROL_PORT=9151 export TOR_CONTROL_COOKIE_AUTH_FILE=~/tbb/tor-browser_en-US/Browser/TorBrowser/Data/Tor/control_auth_cookie export TOR_SKIP_LAUNCH=1 cd $INSTDIR # from the 'Partial Builds' section above ./start-tor-browser.desktop
Launching Tor with an Alternate SOCKS and Control port
If you need to test Tor Launcher changes, you probably want to launch the Tor process too. Here's a quick hack to put that on a different port pair (9250 and 9251):
export TOR_SOCKS_PORT=9250 export TOR_CONTROL_PORT=9251 unset TOR_SKIP_LAUNCH unset TOR_CONTROL_COOKIE_AUTH_FILE cd $INSTDIR # From the 'Partial Builds' section above sed -i -e 's/Port 915/Port 925/' ./Browser/TorBrowser/Data/Tor/torrc-defaults ./start-tor-browser.desktop
Tor Browser Team Communication and Organizational Patterns
The Tor Browser development team is very geographically distributed. We use a few different written forms of communication to discuss development over the Internet: Mailing lists, IRC, and this bug tracker. We also hold weekly IRC meetings.
Communication Mechanisms and Meetings
The mailinglist for Tor Browser development discussion is tbb-dev. You can also subscribe to the list of Tor Browser code commits via tbb-commits. All updates to Tor Browser specific bugs are sent to tbb-bugs. Release tags and test builds are posted to tor-qa for community review and testing.
Our primary mode of day-to-day communication is the #tor-dev IRC channel on irc.oftc.net (port 6697 is ssl). We hold weekly meetings on this IRC channel at 18:00 UTC on Mondays. For details on our meeting format, please see the original meeting announcement post (note however that the time has changed since that posting).
How we use Trac
For historical reasons, Tor Browser tickets are spread across several Trac components: "Tor bundles/installation", "TorBrowserButton", "Firefox Patch Issues", and "Tor Launcher". We are considering consolidating many of these components and switching to keywords instead, but that hasn't happened yet.
What follows is a partial list of trac keywords we use to categorize issues, independent of the component they actually arise in.
- Tickets that represent reported crash or hang issues
- Tickets that represent usability issues. Has many subcategories as suffixes of tbb-usability.
- Tickets that represent usability issues that cause users to be unable to use TBB (this is a subset of the previous query)
- Tickets that represent frequently encountered support issues or blog/twitter commentary
- Tickets that represent a violation of our identifier unlinkability Privacy Requirement.
- Tickets that represent a violation of our fingerprinting unlinkability Privacy Requirement. There are many subcategories of this tag as suffixes.
- Tickets that represent a violation of our Disk Avoidance Security Requirement.
- Tickets for problems with "New Identity"
- Tickets for which we would like to have an automated testcase to prevent regressions.
- Tickets for hardening TBB against exploitation.
- Tickets in other Tor components that TBB wants solved.
- Tickets for our plans related to the hardened builds series.
In addition to this list, if you would like someone to review a patch, you should set the state of that ticket to "needs review" and tag it that that person's name, followed by the year, the month, and the letter 'R'. For example: MikePerry201311R.
Design and Core Components
Ok, so you've got the lay of the land now, and want to really dive in. Here's how everything is organized.
At a high level, Tor Browser development is driven by the design document: The Design and Implementation of the Tor Browser. This document specifies the Design Requirements and Philosophy that guides our development decisions and our modifications to Firefox, provides an adversary model, and provides a high level description of the actual changes we have made to Firefox.
Component Source Repositories
The Tor Browser is actually built from several components. Here are the main ones specific to the browser itself.
Firefox Branch Repo
We maintain a branch of the latest Firefox "Extended Support Release" (ESR) series, to which we have applied multiple patches to satisfy our design requirements and build security needs.
These branches live in the tor-browser git repo. The branches are named for the Firefox version they are based off of, along with an integer signifying a rebase number. To learn which branch a particular build uses, inspect its version file. The version file lists a tag for the specific commit used. This tag is derived from the branch name and an additional specific build number.
Torbutton is an addon with a long and storied history. It originated as ProxyButton (a simple Firefox addon to toggle proxy settings), which was rethemed by Scott Squires and configured specifically for Tor's proxy settings. After that initial work, Mike Perry began development to address the numerous security issues that arose when browser state from non-Tor activity leaked into Tor browsing mode, and vice-versa. This toggle mode was deprecated in favor of a standalone browser several years back.
As a result, Torbutton contains a lot of cruft code specific to the toggle behavior that has not yet been removed, but is otherwise inactive. However, it still provides a few privacy features by way of observers and other extension-level changes to Firefox to improve privacy and Tor security.
Tor Launcher is a Firefox addon that acts as a Tor Controller. It handles launching and configuring Tor for use with the browser. It also is compatible with Thunderbird, InstantBird, and XULRunner, and is used by the Tails project as well.
HTTPS-Everywhere contains tens of thousands of URL rewrite rules to enforce HTTPS for sites that support both HTTP and HTTPS URL schemes.
We don't modify NoScript directly, however we do have a number of preferences changes to it and other addons in the tor-browser-bundle preferences file in the bundle layout directories.
If you're interested in bisecting or otherwise tracking down a particular NoScript change, bug, or issue, avian2 maintains a complete git repository for NoScript, with one commit per NoScript release (including alphas, betas, and rc releases).
Tor Browser includes several pluggable transports, which are a network traffic transformation layer to help avoid censorship and filtering mechanisms. If you are interested in getting your own pluggable transport added to the bundle, the FTE transport work is a good example to follow.
Pluggable Transport Bridges are specified in the bridge_prefs.js configuration file in the Bundle Skeleton directory.
Debugging the Tor Browser
There are a few different ways to debug the Tor Browser, depending on the component involved. We'll focus on the extensions and the browser itself here.
Enabling debug logs in extensions
To enable extension debug logging, set these preferences in your about:config:
Lower levels are more verbose, but if you go much below 2, you will likely be overwhelmed and/or the browser will be extremely slow.
If you want to save your logs to a file, you need to use the --log argument to start-tor-browser.desktop. It takes an optional filename. Without a filename, it writes tor-browser.log in the current directory:
$ ./start-tor-browser.desktop --log $ tail -f tor-browser.log
Using Debug Symbols
If you're going to use GDB to debug an issue, the first thing you want is debug symbols. Tor Browser builds detached debug symbols for tor and firefox.
In the bundle release download directory there should be a tor-browser-linux64-debug.zip file. Unfortunately, the directory structure of the debug zip doesn't exactly match what you need in order for gdb to find the symbols automatically. For the tor-browser-linux symbols, unzip the file in the bundle root directory, and then relocate it to the .debug subdirectory of Browser, like so:
$ cd tor-browser_en-US $ wget https://dist.torproject.org/torbrowser/5.0a3/tor-browser-linux64-debug.zip $ unzip tor-browser-linux64-debug.zip $ mv Debug/Browser Browser/.debug $ gdb ./Browser/firefox
At this point, gdb should find all of the Firefox symbols automatically upon every invocation, and you can either attach to an existing process ID or launch firefox directly, as described in the following sections below.
Attaching an already running TBB
You can attach a gdb instance to an already running TBB firefox process. To do this, find the pid of your running TBB firefox, then tell gdb to attach the running process (note that on some systems you may need to run gdb as root for this to work):
$ for p in `pgrep firefox` ; do ps -v $p ; done [...find the pid for your TBB firefox...] $ gdb ./Browser/firefox (gdb) attach <pid>
Starting firefox from inside gdb
Starting firefox inside gdb:
$ gdb ./Browser/firefox (gdb) run -profile Browser/TorBrowser/Data/Browser/profile.default
Generating and debugging core files
If you have a condition that crashes Tor Browser in an unreliable way, one helpful step is to generate a core file for crashes that do happen. To do this, before launching Tor Browser, run:
$ ulimit -c unlimited
Then, when Tor Browser crashes, a core file should show up in Browser/core. To debug this core, run:
$ gdb ./Browser/firefox ./Browser/core
You can then use the usual gdb commands (backtrace, print, up, down, etc) to inspect the stack, variables, and program state at the time of the crash.
If you found a bug and want to start bisecting to find the exact commit which introduced the problem you have three major cases to consider.
The easy case
You found your bug is part of either Torbutton, Tor Launcher or HTTPS-Everywhere. In this case you check out the respective repository and do the usual bisecting there. To test the various revisions you can take a clean Tor Browser and load the .xpi's into it. Theoretically, this would work in the NoScript case as well but unfortunately there is no public version control system available and thus if you found a bug in NoScript you need to identify the version it showed first up manually and file a ticket.
The medium difficult case
Okay, you ruled out a bug in one of the extensions we ship and determined it got introduced by one of our Tor Browser patches or changes to the bundling/build process (testing whether the bug occurs in a corresponding vanilla Firefox is helpful here). The first thing you are doing now is finding out which Tor Browser version is the one that introduced the bug. Using the Tor Browser archive is a good starting point. If you found the culprit and the bug did not get introduced while switching to a new Firefox ESR (the next section deals with the switching case) chances are high that you don't need to bisect at all. Comparing the changelogs between the last version without the bug and the first version with the bug should be enough in this case. But if the changelogs don't ring a bell and you suspect this is a bug introduced by the build process/bundling looking at the changes introduced in the tor-browser-bundle repository between the two Tor Browser releases should reveal the bad commit. If the changelogs don't help and you suspect an issue with one or more of our Tor Browser patches you need to clone the tor-browser repository and do a normal bisect following the Partial Builds strategy outlined above between the last tor-browser patch of the good Tor Browser and the last tor-browser patch that got into the bad Tor Browser.
The tricky case
The patchset effectively did not change and you are only switching to a new ESR. There could be three reasons for the bad Tor Browser then. The first one: rebasing one or more patches went wrong. The second one: some change in the new ESR is causing the trouble. The third one: a combination of the first two. The strategy to deal with this group of cases is to rule out the first one (and thus the third one as well) and then narrow down the possible patches that could be affected by changes in the new ESR. These patches in turn are applied during bisecting the commits in the mozilla-central repository that made it into the bad Tor Browser.
QA and Testing
Some automated tests are run on nightly builds, and new releases. The main page for the results is http://test-reports.tbb.torproject.org/reports/.
The different categories of tests we are running are described below.
Tor Browser Test Suite
This test suite contains a series of tests to check a Tor Browser Bundle. It is used to check different things:
- the binaries included in the bundle are correctly linked and compiled
- the tor daemon is working, with bridges, some pluggable transports and with an http proxy
- some mozill and selenium tests
This test suite is automatically run in the following cases:
- when a new nightly build is available on https://people.torproject.org/~linus/builds/. An email of the results is sent to a list of people defined in config/linus-nightly. You can ask me if you want to be added to this list.
- when a new release is tagged on git and available on https://people.torproject.org/~user/builds/$version/. The result of the tests is emailed to tor-qa@….
- when a Tor Browser developer created a tbb-qa.yml file to request a build to be test. The result of the tests is emailed to the developer requesting the tests. See the section below for details about requesting tests on custum builds.
The Windows version of the Tor Browser .exe file is uploaded to virustotal.com to check it with different antivirus. All executable files included in the archive are also uploaded.
The uploads are automatically done in the following cases:
- when a new nightly build is available on https://people.torproject.org/~linus/builds/. The results are available on http://test-reports.tbb.torproject.org/reports/index-browserbundle_virustotal.html. No emails are sent at the moment (you can ask me if you are interested to receive them).
- when a new release is tagged on git and available on https://people.torproject.org/~user/builds/$version/. The results are sent to tor-qa@….
Browser Unit Tests
The Browser Unit Tests are the tests which are included in the Mozilla sources tree (xpcshell and mochitests). The Tor Browser patches can add new tests to be run. Before running those tests, we need to build the sources tree.
We are able to run those tests on a series of commits, and display the differences in the results between each commits and its ancestor.
It is not currently run automatically. In the future, it should be possible to specify a commit or branch to be tested, by editing the tbb-qa.yml file. Until then, you can ask me to edit the config file to test a specific branch or commit.
A prototype for a rebasing tests has been made. The goal is to try to automatically rebase all Tor Browser patches on the master branch of gecko-dev.
It is not yet run automatically.
Requesting testing of a custom build using the tbb-qa.yml file
To request a custom build to be tested, you need an account on people.torproject.org and be listed in the config/testrequests file (ask me if you want to be added).
Upload your build to perdulce in directory ~public_html/builds/x.y.z-featureA
Create a file public_html/builds/tbb-qa.yml with permission 744 and a list of build directory names using the YAML syntax:
--- - x.y.z-featureA
You should receive an email on your @torproject.org email address when the tests finished running. Make sure to include a proper sha256sums.txt file and its signature in the respective build directory.
The tbb-qa.yml file can also be used to request other types of tests. Instead of adding a string which is a directory name in your builds directory, you can add a hash table containing the options for your test requests and a 'type' key containing the type of tests you want to run.
For instance the following tbb-qa.yml file can be used to request a test of type 'browserunit' (Tor Browser unit tests) on a specific commit:
--- - type: browserunit commit: e1a8bafb44367592b
The following types of tests are available:
Tor Browser Unit Tests (xpcshell and mochitest)
type name: browserunit
- commit: the commit that you want to build and run tests on. This can be a commit hash, tag or branch.
- esr_branch: using this option means that you want all commits from the Mozilla ESR branch to the selected commit to be tested. Example: esr31.
- git_url: the URL of the git repository containing the selected commit. If unspecified, https://git.torproject.org/tor-browser.git is used by default.