Building the Tor Browser Bundle (TBB) Using the Gitian Build System

This page gives guidance to novice developers building TBB for the first time and serves as reference to experienced developers when troubleshooting.

How to Use This Wiki

Use the search on page for text relating to a problem at hand or the more general search function of the top level wiki system (and hope for redirection to a solution.)

Beware that reading this collection of problems like a book may cause nausea.

Starting With the Correct Source Revisions

Once the gitian build system is bootstrapped according to the hacking document, a number of revisions and branches can be used to build TBB. A typical workflow consists of:

tor-browser-bundle$ grep TORBROWSER_VERSION gitian/versions
tor-browser-bundle$ git tag -l
tor-browser-bundle$ git checkout tbb-3.5.4-build3
tor-browser-bundle$ cd gitian && make

Please note that the match-* Makefile targets will produce correct results only if the revision built matches that of the versions-* file at hand. Also beware of troubles originating from the wrong choice of branch/tag or make(1) target.

Individual Revisions Fetched

Incidentally, several bundled subprojects are fetched to populate the tor-browser-bundle/../gitian-builder/inputs directory. To inspect what gitian is downloading and building try:

tor-browser-bundle$ cd ../gitian-builder/inputs/tor-browser
gitian-builder/inputs/tor-browser$ git branch
* (detached from tor-browser-31.4.0esr-4.5-1-build1)
gitian-builder/inputs/tor-browser$ cd -

How to Avoid Destruction of Successfully Built Components

In short: Run make build, not make, when you want to restart a partial build. make destroys all partial build products and starts the build from scratch.

gitian/Makefile is simple; take a look at it to see what commands it runs.

tor-browser-bundle/gitian$ make
# ...implicitly destroys all built components
tor-browser-bundle/gitian$ make build
# ...will preserve all previously built components
tor-browser-bundle/gitian$ ./
# ...executed manually for even finer grained control

Targets for Special Product Classes

For expediency, build engineers take liberties to make assumptions and update (or fail to update) certain YAML, version, and other build system logic according to product class. For example, alpha builds do not require a updated version file to succeed (as opposed to release builds.) The Makefile targets alpha, prep-alpha, and build-alpha tolerate such a version discrepancy. For example:

# Wrong way to build a alpha product
tor-browser-bundle$ git checkout tbb-<ver>a1-build1
cd gitian && make
./ line 189: !PACKAGE: unbound variable

# Correct way to build a alpha product (prep-|build-|alpha targets)
tor-browser-bundle$ git checkout tbb-<ver>a1-build1
cd gitian && make alpha
# Encounter a build problem, restart build...
tor-browser-bundle/gitian$ make build-alpha

# Correct way to build a release product (all/build targets)
tor-browser-bundle$ git checkout tbb-<ver>-build1
cd gitian && make
# Encounter a build problem, restart build...
tor-browser-bundle/gitian$ make build

Testing With Nightly

The nightly targets are useful for building the tip of selected projects for quality assurance. Inspect the repository in question and choose the tag describing a changeset to test. Adapt the revision entry in tor-browser-bundle/gitian/versions.nightly correspondingly and build:

tor-browser-bundle/gitian$ make nightly

Examination of Gitian Build Segments

While building, the gitian build sequence produces output of which the following is most important:

****** Starting Tor Component of Linux Bundle (1/4 for Linux) ******
****** Starting TorBrowser Component of Linux Bundle (2/4 for Linux) ******
****** Starting Pluggable Transports Component of Linux Bundle (3/4 for Linux) ******
****** Starting Bundling+Localization of Linux Bundle (4/4 for Linux) ******
****** Linux Bundle complete ******

****** Starting Tor Component of Windows Bundle (1/4 for Windows) ******
****** Starting Torbrowser Component of Windows Bundle (2/4 for Windows) ******
****** Starting Pluggable Transports Component of Windows Bundle (3/4 for Windows) ******
****** Starting Bundling+Localization of Windows Bundle (4/4 for Windows) ******
****** Windows Bundle complete ******

****** Starting Tor Component of Mac Bundle (1/4 for Mac******
****** Starting Torbrowser Component of Mac Bundle (2/4 for Mac) ******
****** Starting Pluggable Transports Component of Mac Bundle (3/4 for Mac) ******
****** Starting Bundling+Localization of Mac Bundle (4/4 for Mac) ******
****** Mac Bundle complete ******

These lines correspond to the build stages of the respective Linux, Windows, and Mac bundles.

Knowing When the Build Succeeded

The TBB build succeeds just after the Mac bundle completes. Therefore, to confirm success look for:

****** Mac Bundle complete ****** the final lines of build output. Additionally, a number of bundles are found in the gitian/<version> subdirectory.

$ ls tor-browser-bundle/gitian/3.5.4

Verification of Build Products

Once a TBB build succeeds, make(1) match returns text indicating if the bundles match the hashes of valid (from independent deterministic built) bundles.

Build Times

The duration of TBB builds varies according to hardware type and availability of resources like memory and swap. For guidance in VM guest creation and corresponding requirements please review the VM guest setup document.

Type1 Distro Version Metaconf2 Arch RAM Swap CPUs3 Rate4 Hours (!) Notes
VM Ubuntu 12.04.4 Server AMD64 3 Go 2 Go 4 Phenom X4 4GHz 9 Workstation
BM Debian Wheezy NA AMD64 16 Go ? 4 ? 12 Unknown
BM Ubuntu 14.04 Desktop AMD64 8 Go 4 Go 4 Core i5-2537M 1.4GHz 32 Notebook

1 Bare metal host (BM) using KVM or virtual machine guest (VM) using LXC
2 OS preconfigured package selections (often desktop or server)
3 Total processing units (processors, cores, hyperthreads?)
4 Processor clock rate and performance influencing factors
(!) The canonical indicator (what we're interested to measure)

Gitian Build Tips

Useful log files are produced (and then deleted as each stage succeeds) and are extremely helpful in debugging build problems. Consider opening a new window and following log output as gitian produces it:

$ tail -F gitian-builder/var/install.log
$ tail -F gitian-builder/var/build.log

Or even better, install multitail(1) and monitor each of the two main logs from start to finish:

$ sudo apt-get install multitail
$ multitail gitian-builder/var/install.log gitian-builder/var/build.log

Don't be fooled by storage measurements of completed builds. Such late stage analysis doesn't reflect temporary build content stored during the build (which is quite a bit more than the resulting bundles.)

$ df -h
Filesystem      Size    Used  Avail  Use%  Mounted on
/dev/sda1        34G     20G    14G   58%  /
$ du -hs gitian-builder tor-browser-bundle
16G	gitian-builder
1,9G	tor-browser-bundle

Troubleshooting Build Failures

The gitian build sequence is complex and uses tools not particularly designed to work together. Thus, the nature of gitian building is fragile and new TBB builds typically break at some point.

RAM Exhaustion

Rather than 'You ran out of memory', error messages indicating RAM exhaustion typically look like this:

collect2: ld terminated with signal 9 [killed]

This is quite common when shaving the bits off minimum RAM requirements and gitian starts linking, for example:

make[5]: *** [] Error 1

Obviously to solve the problem of RAM exhaustion, more memory is needed. This is trivial if building on a memory reduced VM, but if not there's still the chance of increasing swap and hoping for the best.

Gitian failures with LXC on Ubuntu 14.04

If one sees Gitian failures like

bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

then this is a sign that new containers need to get created. Related to that lxc-execute is required for building Tor Browser using LXC since 14.04. This is off by default (to support older Ubuntu versions) but can be activated by setting the proper environment variable before starting the build:

export LXC_EXECUTE=lxc-execute

#15947 has further details for interested parties.

Apt-cacher-ng(1) Strangulation

The version of apt-cacher-ng(1) distributed in Ubuntu 12.04.4 enjoys strangulating guest (KVM or LXC) gitian instances. When the window where gitian/make(1) was issued returns:

Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
./bin/gbuild:21:in `system!': failed to run on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install git-core unzip zip swig m4 python-setuptools python-dev faketime libtool > var/install.log 2>&1 (RuntimeError)
	from ./bin/gbuild:93:in `build_one_configuration'
	from ./bin/gbuild:224
	from ./bin/gbuild:219:in `each'
	from ./bin/gbuild:219
	from ./bin/gbuild:217:in `each'
	from ./bin/gbuild:217
make: *** [build] Fehler 1
Command exited with non-zero status 2

...or when the the gitian-builder/var/install.log indicates a call to apt-get(1) that returns:

E: Failed to fetch ​<pkg>.deb Size mismatch

...or when encountering hash sum mismatches when fetching and comparing input checksums:

Fetched 18 MB in 1min (300 kB/s)
, stderr: bzip2: (stdin) is not a bzip2 file.
W: Failed to fetch gzip:/var/lib/apt/lists/partial/  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

qemu-img: target-precise-amd64.qcow2: Could not open 'base-precise-amd64.qcow2': Could not open 'base-precise-amd64.qcow2': No such file or directory: No such file or directory
amd64 precise VM creation failed

...or similar, then it's time to kill and restart the apt-cacher-ng(1) process in the controlling (outer) gitian host. That means, log into the machine where gitian/make(1) build was executed and:

$ ps aux | grep apt-cacher-ng
123        991  0.6  0.0 254036  2500 ?        Ssl  07:17   1:36 /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng pidfile=/var/run/apt-cacher-ng/pid SocketPath=/var/run/apt-cacher-ng/socket foreground=0
$ kill -HUP 991  # use kill -TERM 991 if HUP doesn't work
$ sudo -u \#123 /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng pidfile=<MAKE_SURE_TO_COPY_THE_ENTIRE_PROCESS_ARGS_HERE>

A alternative solution involves installing a newer (or older?) version of the apt-cacher-ng package. You'll likely need to download and install the (dependency) init-system-helpers package as well.

Another solution involves removing resudual cache cruft:

rm -rf /var/lib/apt/lists && mkdir -p /var/lib/apt/lists/partial

A yet third option is to edit gitian-builder/bin/make-base-vm and remove from the MIRROR_HOST, directly fetching packages every time. (Obviously this may use more bandwidth/be slower.)

Flawed Software Hangs

Building TBB on some (newer) systems like Ubuntu 14.04 (LTS) AMD64 Desktop employs components affected with networking problems leading to unexpected disconnections and indirectly causing hangs in the build process (actually it's the 'prep' stage.) This is indicated by the top(1) command listing the component in question at the top of the list consuming 100% CPU cycles. For example, this can happen when downloading dependencies with svn(1) in conjunction with torsocks(1). To solve this particular problem repeat the build sequence using the svn(1) command without torsocks(1), the component actually at fault:

tor-browser-bundle/gitian$ make prep TORSOCKS=

Assembly Errors in Mismatched Architecture Code or 'failed to run on-target setarch'

A general problem that can manifest in a few ways:

x86cpuid.s Err: invalid instruction suffix for `push'
Preparing build environment
setarch: x86_64: Unrecognized architecture
./bin/gbuild:21:in `system!': failed to run on-target setarch x86_64 bash < target-bin/ (RuntimeError)
	from ./bin/gbuild:82:in `build_one_configuration'
	from ./bin/gbuild:224:in `block (2 levels) in <main>'
	from ./bin/gbuild:219:in `each'
	from ./bin/gbuild:219:in `block in <main>'
	from ./bin/gbuild:217:in `each'
	from ./bin/gbuild:217:in `<main>'

In these situations, it meant that kvm was running for the wrong architecture. It happens when the VM for gitian-utils amd64 (or i386) didn't shut down properly, and the build for gitian-tor 386 (or amd64) started up using the running VM for the wrong architecture. To fix it, kill the kvm process (in difference circumstances, running 'poweroff' inside the VM works, other times the kill must be from outside), and start another "make build". If the problem is indeed that the wrong VM is running, then installing gcc-multilib may not be the right thing to do. But if you want to attempt it...

Assembly errors have been observed with output in gitian-builder/var/build.log resembling:

x86cpuid.s Err: invalid instruction suffix for `push'
x86cpuid.s Err: invalid instruction suffix for `pop'

While this specific example refers to OpenSSL, it's a typical side effect of compiling IA32 assembly code on AMD64 platforms lacking certain assembler components, probably available in gcc-multilib or similar (in the build guest not the host!)

To investigate and work around this problem, start building as usual:

tor-browser-bundle/gitian$ make build

...but once 'Building tor-linux for lucid i386' appears, run:

tor-browser-bundle/gitian$ cd ../../gitian-builder
gitian-builder$ libexec/on-target -u root
root@ubuntu~# apt-get install gcc-multilib
root@ubuntu~# cd /home/ubuntu/build/openssl-1.0.1g

By the way, installing gcc-multilib (or possibly libc6-dev-i386) is a workaround in consideration in a partly related bug report #12391 about mingw-w64 components.

Package Management Failures

The apt-get(1) package manager is used in the prep stages to create virtual machines or linux containers. A variety of errors can cause gitian-builder to fail.

WARNING: The following packages cannot be authenticated!
E: There are problems and -y was used without --force-yes

In most cases the seemingly random failure is neither deterministic nor indicative of any real problem. Restart the build and cross fingers, toes, or press the thumbs:

tor-browser-bundle/gitian$ make prep(|-alpha|-beta|-nightly)
tor-browser-bundle/gitian$ make build(|-alpha|-beta|-nightly)

Random Build Failures

There are times during the gitian build sequence when it simply hangs or stops for reasons that seem random. Rest assured that 'Ctrl-C' and 'make build' are often the best course of action, and that repeating this results in incremental build success.

If you feel your build problems are overly chaotic please contact your representative in congress or speak with a psychic counsellor. Otherwise, the friendly folks hanging out in the #tor-dev chat room (on are ready to help answer build questions. Don't forget to update this document when solutions are found!

Other Related Resources

Other documents that aide in further understanding of using gitian-build for Tor software include:

Upstream Gitian Documentation

Additional help can be obtained by appeals to #bitcoin-dev ( or a number of online resources:

Bitcoin's Gitian Building document

Last modified 7 months ago Last modified on May 12, 2015 10:20:58 AM