wiki:doc/TorBrowser/Hacking

So, You Want to Hack on Tor Browser!

Welcome to the Official Tor Browser Hacking Intro. This page is meant to give you an overview of how to get started with Tor Browser development. It covers building the browser, how we communicate and use trac to organize our issues and development process, what goes into the browser itself, and also provides links to other development resources and information.

The Bleeding Edge

The first thing you probably want to know is how to obtain the very latest (and sometimes greatest) Tor Browser builds, and how to build your own version of the Tor Browser from our source code. Well wait no further.

Nightly Builds

Currently, nightly builds are available courtesy of Linus Nordberg. They are available in the tbb-nightly-* subdirectories of his homedirectory on people.torproject.org.

As with all of our builds, it should be possible to reproduce byte-for-byte identical versions of all of those binaries from source. To learn how to do that, read on.

Building the Tor Browser

Our build system is based on Gitian, which was initially developed by the Bitcoin community. Gitian is a wrapper around Ubuntu's python-vm-builder and associated virtualization tools that helps to provide a clean, controlled, reproducible build environment in a fully automated fashion. We further wrap Gitian with our own helper scripts that download and authenticate inputs, and automate building and assembling each component piece of the browser into a final set of output packages for Linux, MacOS, and Windows.

This system enables us to provide secure, verifiable, byte-for-byte reproducible builds to ensure the integrity of our binaries and to protect the build process from compromise. We have written a pair of blog posts that describe in more detail why this is important, and the technical details behind how this is achieved, if you are curious.

To build the Tor Browser, you need an Ubuntu machine or VM. Once that is done, check out a copy of the builder repo with:

mkdir tor-browser-build
cd tor-browser-build
git clone https://git.torproject.org/builders/tor-browser-bundle.git
cd tor-browser-bundle/gitian

After that, you should be able to run 'make', 'make beta', 'make alpha', or 'make nightly' to build the entire bundle for all three platforms. The build scripts will detect any additional packages or configuration you need to perform on your system.

Inside that directory, you will also see a README.build file with further information, should you run into any issues. In particular, the initial VM setup process is still somewhat fragile. In some cases your initial VM setup may fail or run into other issues, and you may need to restart your build with 'killall qemu-kvm'. See that README for more details on diagnosing and correcting these issues.

We also have a wiki page specific to building with Gitian that may be useful in helping you work through certain esoteric errors and failure conditions.

Reproducing an Existing Build

Every Tor Browser build includes a pair of files that describe exactly what versions of components went into making it, to allow easy reproducibility. These two files are located in the "Docs/sources" subdirectory of the Tor Browser destination directory.

To reproduce an existing bundle, inspect bundle.inputs and obtain the tor-browser-bundle.git commit hash describing the commit to build from.

Once you have this hash, you should be able to run something like the following commands:

   cp versions tor-browser-build/tor-browser-bundle/gitian/versions
   cd tor-browser-build/tor-browser-bundle/gitian
   git checkout 1929fd5db0802782066d52512d4c2a0fef144ed6
   make

If your have a beta, alpha, or nightly build, you will need to place the versions file in the appropriate place (versions.beta, versions.alpha, or versions.nightly) and run the corresponding make target ('make beta', 'make alpha', or 'make nightly') instead of using just 'versions' and 'make' (which are for a stable series build).

Be aware that this process is not fully future-proof. In particular, if Ubuntu has updated their development tool chain since the bundles have been built, you may encounter differences between your resulting bundles and the original binaries. This should be rare, however, as we use only the "Long Term Support" versions of Ubuntu in our build VMs. The only reason they should change the toolchain is in the event of serious security issues in the development tools themselves.

Partial Builds

This section is mostly of interest to developers making changes to Tor Browser or its components, and who wish to test their patches.

Building Just Firefox

If you want to just develop on the browser without messing with gitian (for example, to have incremental rebuilds of just the files you modify), the best way is to build a new browser distribution and copy the result over an existing Tor Browser directory. Clone the Tor Browser repo.

To build it, from the top directory of the tor-browser repo, do:

# Install a subset of the packages listed in
# https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/descriptors/linux/gitian-firefox.yml#l8
# Mostly, we don't need `libiw-dev` or `faketime`, but we *do* need `pkg-config`.

sudo torsocks apt-get install \
    --no-install-suggests --no-install-recommends \
    zip \
    unzip \
    libglib2.0-dev \
    libgtk2.0-dev \
    libdbus-1-dev \
    libdbus-glib-1-dev \
    yasm \
    libasound2-dev \
    libcurl4-openssl-dev \
    libxt-dev \
    mesa-common-dev \
    autoconf \
    autoconf2.13 \
    libtool \
    hardening-wrapper \
    libgstreamer-plugins-base0.10-dev \
    pkg-config \
    g++ \
    libpulse-dev

# Generate the configure scripts:
make $CONFIGURE_ARGS -f client.mk configure CONFIGURE_ARGS="--with-tor-browser-version=4.5a4 --enable-update-channel=alpha"

# And... compile:
make $MAKEOPTS -f client.mk build
make -C obj-* package INNER_MAKE_PACKAGE=true

# Point the INSTDIR at an existing TBB directory:
export INSTDIR="$HOME/tbb/test/tor-browser_en-US"

# Move the compiled firefox on top of the old TBB browser dir:
cp -a obj-*/dist/firefox/* $INSTDIR/Browser/

# If you want a smaller binary to copy into a vm/other machine for tests:
strip --strip-all $INSTDIR/Browser/*
rm -f $INSTDIR/Browser/firefox-bin

If that completes successfully, then your fresh build of Firefox should exist at $HOME/install/Browser/firefox. Free the lizard.

Tor Browser Communication and Organizational Patterns

The Tor Browser development team is very geographically distributed. We use a few different written forms of communication to discuss development over the Internet: Mailing lists, IRC, and this bug tracker. We also hold weekly IRC meetings.

Communication Mechanisms and Meetings

The mailinglist for Tor Browser development discussion is tbb-dev. You can also subscribe to the list of Tor Browser code commits via tbb-commits. All updates to Tor Browser specific bugs are sent to tbb-bugs. Release tags and test builds are posted to tor-qa for community review and testing.

Our primary mode of day-to-day communication is the #tor-dev IRC channel on irc.oftc.net (port 6697 is ssl). We hold weekly meetings on this IRC channel at 18:00 UTC on Mondays. For details on our meeting format, please see the original meeting announcement post (note however that the time has changed since that posting).

How we use Trac

For historical reasons, Tor Browser tickets are spread across several Trac components: "Tor bundles/installation", "TorBrowserButton", "Firefox Patch Issues", and "Tor Launcher". We are considering consolidating many of these components and switching to keywords instead, but that hasn't happened yet.

Trac Keywords

What follows is a partial list of trac keywords we use to categorize issues, independent of the component they actually arise in.

tbb-crash
Tickets that represent reported crash or hang issues
tbb-usability
Tickets that represent usability issues. Has many subcategories as suffixes of tbb-usability.
tbb-usability-stoppoint
Tickets that represent usability issues that cause users to be unable to use TBB (this is a subset of the previous query)
tbb-helpdesk-frequent
Tickets that represent frequently encountered support issues or blog/twitter commentary
tbb-linkability
Tickets that represent a violation of our identifier unlinkability Privacy Requirement.
tbb-fingerprinting
Tickets that represent a violation of our fingerprinting unlinkability Privacy Requirement. There are many subcategories of this tag as suffixes.
tbb-disk-leak
Tickets that represent a violation of our Disk Avoidance Security Requirement.
tbb-newnym
Tickets for problems with "New Identity"
tbb-testcase
Tickets for which we would like to have an automated testcase to prevent regressions.
tbb-security
Tickets for hardening TBB against exploitation.
tbb-needs
Tickets in other Tor components that TBB needs solved.

In addition to this list, if you would like someone to review a patch, you should set the state of that ticket to "needs review" and tag it that that person's name, followed by the year, the month, and the letter 'R'. For example: MikePerry201311R.

Design and Core Components

Ok, so you've got the lay of the land now, and want to really dive in. Here's how everything is organized.

Design Document

At a high level, Tor Browser development is driven by the design document: The Design and Implementation of the Tor Browser. This document specifies the Design Requirements and Philosophy that guides our development decisions and our modifications to Firefox, provides an adversary model, and provides a high level description of the actual changes we have made to Firefox.

Component Source Repositories

The Tor Browser is actually built from several components. Here are the main ones specific to the browser itself.

Firefox Branch Repo

We maintain a branch of the latest Firefox "Extended Support Release" (ESR) series, to which we have applied multiple patches to satisfy our design requirements and build security needs.

These branches live in the tor-browser git repo. The branches are named for the Firefox version they are based off of, along with an integer signifying a rebase number. To learn which branch a particular build uses, inspect its version file. The version file lists a tag for the specific commit used. This tag is derived from the branch name and an additional specific build number.

Torbutton

Torbutton is an addon with a long and storied history. It originated as ProxyButton (a simple Firefox addon to toggle proxy settings), which was rethemed by Scott Squires and configured specifically for Tor's proxy settings. After that initial work, Mike Perry began development to address the numerous security issues that arose when browser state from non-Tor activity leaked into Tor browsing mode, and vice-versa. This toggle mode was deprecated in favor of a standalone browser several years back.

As a result, Torbutton contains a lot of cruft code specific to the toggle behavior that has not yet been removed, but is otherwise inactive. However, it still provides a few privacy features by way of observers and other extension-level changes to Firefox to improve privacy and Tor security.

Tor Launcher

Tor Launcher is a Firefox addon that acts as a Tor Controller. It handles launching and configuring Tor for use with the browser. It also is compatible with Thunderbird, InstantBird, and XULRunner, and is used by the Tails project as well.

HTTPS-Everywhere

HTTPS-Everywhere contains tens of thousands of URL rewrite rules to enforce HTTPS for sites that support both HTTP and HTTPS URL schemes.

NoScript

We don't modify NoScript directly, however we do have a number of preferences changes to it and other addons in the tor-browser-bundle preferences file in the bundle layout directories.

Pluggable Transports

Tor Browser includes several pluggable transports, which are a network traffic transformation layer to help avoid censorship and filtering mechanisms. If you are interested in getting your own pluggable transport added to the bundle, the FTE transport work is a good example to follow.

The original ticket for FTE is a good starting point. It resulted in a clean branch that merged easily.

Pluggable Transport Bridges are specified in the bridge_prefs.js configuration file in the Bundle Skeleton directory.

Debugging the Tor Browser

There are a few different ways to debug the Tor Browser, depending on the component involved. We'll focus on the extensions and the browser itself here.

Enabling debug logs in extensions

To enable extension debugging, set these preferences in your about:config:

SETTING NAME VALUE
extensions.torbutton.loglevel 2
extensions.torbutton.logmethod 0
extensions.https_everywhere.LogLevel 2
extensions.torlauncher.loglevel 2
extensions.torlauncher.logmethod 0

Lower levels are more verbose, but if you go much below 2, you will likely be overwhelmed and/or the browser will be extremely slow.

If you want to save your logs to a file, find the line in the start-tor-browser script in your Tor Browser directory which starts firefox, and change it like this:

Tor Browser 4.x:

# Find this line:
./Browser/firefox -no-remote --class "Tor Browser" -profile Data/Browser/profile.default "${@}"
# and change it to:
./Browser/firefox -no-remote --class "Tor Browser" -profile Data/Browser/profile.default "${@}" &>> ./tbb-debug.log

Tor Browser 3.6.x:

# Find this line:
./Browser/firefox -no-remote --class "Tor Browser" -profile Data/Browser/profile.default "${@}"
# and change it to:
./Browser/firefox -no-remote --class "Tor Browser" -profile Data/Browser/profile.default "${@}" &>> ./tbb-debug.log

Using gdb

Attaching an already running TBB

You can attach a gdb a your TBB firefox process to a gdb instance. To do this, find the pid of your running TBB firefox, then tell gdb to attach the running process:

$ for p in `pgrep firefox` ; do ps -v $p ; done
[...find the pid for your TBB firefox...]
$ gdb
$ attach <pid>

For example:

∃!isisⒶwintermute:(master *+$)~/dt/tor-browser_en-US ∴ ls
Browser  Data  Debug  Docs  tor-browser-selenium.log  start-tor-browser  tbb-debug.log  tbb3.0b1.core.1342  Tor
∃!isisⒶwintermute:(master *+$)~/dt/tor-browser_en-US ∴ for p in `pgrep firefox` ; do ps -v $p ; done
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
 1515 ?        Sl    32:16    127    74 1723329 723804  9.0 ./Browser/firefox -no-remote -profile Data/Browser/profile.default
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
26120 ?        Ssl  140:30    376   112 1438855 392396  4.9 firefox
∃!isisⒶwintermute:(master *+$)~/dt/tor-browser_en-US ∴ gdb -q
(gdb) attach 1515
Attaching to process 1515
Reading symbols from /home/isis/dt/tor-browser-3.0-b1/Browser/firefox...(no debugging symbols found)...done.
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.17.so...done.
done.
[New LWP 17232]
[New LWP 12369]
[…]

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/libdl-2.17.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
[…]

(gdb) bt
#0  0x00003a87be1a824d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00003a87bb5890c3 in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#2  0x00003a87b94a3194 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00003a87b94a329c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00003a87bb588f4f in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#5  0x00003a87bb5a0b35 in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#6  0x00003a87bb5a0d2a in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#7  0x00003a87bb74de76 in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#8  0x00003a87bb7209be in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#9  0x00003a87bb6559c5 in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#10 0x00003a87bb76d7d6 in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#11 0x00003a87bb5a0e69 in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#12 0x00003a87bb45ed60 in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#13 0x00003a87babc9ab5 in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#14 0x00003a87babcd7de in ?? () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#15 0x00003a87babcda2a in XRE_main () from /home/isis/dt/tor-browser-3.0-b1/Browser/libxul.so
#16 0x00003a87bf2d4d4a in _start ()
(gdb) x/10i 0x00003a87bf2d4d4a
   0x3a87bf2d4d4a <_start+1402>:        mov    %eax,%ebx
   0x3a87bf2d4d4c <_start+1404>:        mov    0x20(%rsp),%rdi
   0x3a87bf2d4d51 <_start+1409>:        test   %rdi,%rdi
   0x3a87bf2d4d54 <_start+1412>:        je     0x3a87bf2d4d5c <_start+1420>
   0x3a87bf2d4d56 <_start+1414>:        mov    (%rdi),%rax
   0x3a87bf2d4d59 <_start+1417>:        callq  *0x10(%rax)
   0x3a87bf2d4d5c <_start+1420>:        callq  0x3a87bf2d5703 <_start+3891>
   0x3a87bf2d4d61 <_start+1425>:        callq  0x3a87bf2d5969 <_start+4505>
   0x3a87bf2d4d66 <_start+1430>:        jmp    0x3a87bf2d4d6d <_start+1437>
   0x3a87bf2d4d68 <_start+1432>:        mov    $0xff,%ebx
(gdb) detach
Detaching from program: /home/isis/dt/tor-browser-3.0-b1/Browser/firefox, process 1515
(gdb) q
∃!isisⒶwintermute:(master *+$)~/dt/tor-browser_en-US ∴

Starting firefox from inside gdb

Starting firefox inside gdb, getting stacktraces and coredumps:

1) Start firefox with

gdb ./Browser/firefox

2a) For Tor Browser 4.x, at the gdb prompt (gdb), type:

run -profile Browser/TorBrowser/Data/Browser/profile.default.

2b) For Tor Browser 3.6.x, at the gdb prompt (gdb), type:

run -no-remote -profile Data/Browser/profile.default.

3) To get a stacktrace from a gdb shell, do:

thread apply all bt full.

4) To get a stacktrace from a coredump file (e.g. 'core.1234'), do:

gdb <name-of-program> <core-filename>

then the command from step 3. Note that some distros disable coredumps by default in /etc/profile, to enable it temporarily in a shell do:

ulimit -c unlimited.

Using Debug Symbols

Tor Browser builds detached debug symbols for tor, firefox, and the pluggable transports. In the bundle release download directory there should be a tor-browser-linux64-debug.zip file. You'd want to unzip that debug.zip archive in the top level of its corresponding Tor Browser directory. (In the example above, where I did ls, you can see the Debug/ directory - that comes from unzipping the debug.zip.) Then load the TBB firefox into gdb, as above, and then (I believe) the gdb command you want is:

(gdb) thread apply all load ./Debug/Browser/libxul.so

or whichever lib*.so in ./Debug/Browser you're still missing symbols from. Everything wrong, errors errors and errors only. No way to use debug symbols, make debug builds instead, don't waste your time and traffic with those symbol files.

QA and Testing

Some automated tests are run on nightly builds, and new releases. The main page for the results is http://test-reports.tbb.torproject.org/reports/.

If you want to run the test suite yourself, read the installation instructions, and the usage instructions. You can also learn how to add a new test.

The different categories of tests we are running are described below.

Tor Browser Test Suite

This test suite contains a series of tests to check a Tor Browser Bundle. It is used to check different things:

  • the binaries included in the bundle are correctly linked and compiled
  • the tor daemon is working, with bridges, some pluggable transports and with an http proxy
  • some mozill and selenium tests

The list of tests is defined in TBBTestSuite/BrowserBundleTests.pm. Documentation for adding new tests is available.

This test suite is automatically run in the following cases:

  • when a Tor Browser developer created a tbb-qa.yml file to request a build to be test. The result of the tests is emailed to the developer requesting the tests. See the section below for details about requesting tests on custum builds.

VirusTotal uploads

The Windows version of the Tor Browser .exe file is uploaded to virustotal.com to check it with different antivirus. All executable files included in the archive are also uploaded.

The uploads are automatically done in the following cases:

Browser Unit Tests

The Browser Unit Tests are the tests which are included in the Mozilla sources tree (xpcshell and mochitests). The Tor Browser patches can add new tests to be run. Before running those tests, we need to build the sources tree.

We are able to run those tests on a series of commits, and display the differences in the results between each commits and its ancestor.

It is not currently run automatically. In the future, it should be possible to specify a commit or branch to be tested, by editing the tbb-qa.yml file. Until then, you can ask me to edit the config file to test a specific branch or commit.

Rebasing Tests

A prototype for a rebasing tests has been made. The goal is to try to automatically rebase all Tor Browser patches on the master branch of gecko-dev.

It is not yet run automatically.

Requesting testing of a custom build using the tbb-qa.yml file

To request a custom build to be tested, you need an account on people.torproject.org and be listed in the config/testrequests file (ask me if you want to be added).

Upload your build to perdulce in directory ~public_html/builds/x.y.z-featureA

Create a file public_html/builds/tbb-qa.yml with permission 744 and a list of build directory names using the YAML syntax:

 ---
 - x.y.z-featureA

You should receive an email on your @torproject.org email address when the tests finished running.

At the moment, only tests using the "Tor Browser Test Suite" can be requested. In the future, the tbb-qa.yml file will be extended to allow requesting other types of tests.

Other Resources

Last modified 5 weeks ago Last modified on Feb 17, 2015 8:53:16 PM