- Building the Tor Browser Bundle (TBB) Using the Gitian Build System
- Starting With the Correct Source Revisions
- How to Avoid Destruction of Successfully Built Components
- Targets for Special Product Classes
- Examination of Gitian Build Segments
- Gitian Build Tips
- Troubleshooting Build Failures
- Other Related Resources
Building the Tor Browser Bundle (TBB) Using the Gitian Build System
This page gives guidance to novice developers building TBB for the first time and serves as reference to experienced developers when troubleshooting.
How to Use This Wiki
Use the search on page for text relating to a problem at hand or the more general search function of the top level wiki system (and hope for redirection to a solution.)
Beware that reading this collection of problems like a book may cause nausea.
Starting With the Correct Source Revisions
Once the gitian build system is bootstrapped according to the hacking document, a number of revisions and branches can be used to build TBB. A typical workflow consists of:
tor-browser-bundle$ grep TORBROWSER_VERSION gitian/versions 3.5.4 tor-browser-bundle$ git tag -l tbb-3.5.4-build3 [...] tor-browser-bundle$ git checkout tbb-3.5.4-build3 tor-browser-bundle$ cd gitian && make
Please note that the match-* Makefile targets will produce correct results only if the revision built matches that of the versions-* file at hand. Also beware of troubles originating from the wrong choice of branch/tag or make(1) target.
Individual Revisions Fetched
Incidentally, several bundled subprojects are fetched to populate the tor-browser-bundle/../gitian-builder/inputs directory. To inspect what gitian is downloading and building try:
tor-browser-bundle$ cd ../gitian-builder/inputs/tor-browser gitian-builder/inputs/tor-browser$ git branch * (detached from tor-browser-31.4.0esr-4.5-1-build1) gitian-builder/inputs/tor-browser$ cd - tor-browser-bundle$
How to Avoid Destruction of Successfully Built Components
In short: Run make build, not make, when you want to restart a partial build. make destroys all partial build products and starts the build from scratch.
gitian/Makefile is simple; take a look at it to see what commands it runs.
tor-browser-bundle/gitian$ make # ...implicitly destroys all built components tor-browser-bundle/gitian$ make build # ...will preserve all previously built components tor-browser-bundle/gitian$ ./mkbundle-linux.sh # ...executed manually for even finer grained control
Targets for Special Product Classes
For expediency, build engineers take liberties to make assumptions and update (or fail to update) certain YAML, version, and other build system logic according to product class. For example, alpha builds do not require a updated version file to succeed (as opposed to release builds.) The Makefile targets alpha, prep-alpha, and build-alpha tolerate such a version discrepancy. For example:
# Wrong way to build a alpha product tor-browser-bundle$ git checkout tbb-<ver>a1-build1 cd gitian && make ./fetch-inputs.sh: line 189: !PACKAGE: unbound variable tor-browser-bundle/gitian$ # Correct way to build a alpha product (prep-|build-|alpha targets) tor-browser-bundle$ git checkout tbb-<ver>a1-build1 cd gitian && make alpha # Encounter a build problem, restart build... tor-browser-bundle/gitian$ make build-alpha # Correct way to build a release product (all/build targets) tor-browser-bundle$ git checkout tbb-<ver>-build1 cd gitian && make # Encounter a build problem, restart build... tor-browser-bundle/gitian$ make build
Testing With Nightly
The nightly targets are useful for building the tip of selected projects for quality assurance. Inspect the repository in question and choose the tag describing a changeset to test. Adapt the revision entry in tor-browser-bundle/gitian/versions.nightly correspondingly and build:
tor-browser-bundle/gitian$ make nightly
Examination of Gitian Build Segments
While building, the gitian build sequence produces output of which the following is most important:
****** Starting Tor Component of Linux Bundle (1/4 for Linux) ******
****** Starting TorBrowser Component of Linux Bundle (2/4 for Linux) ******
****** Starting Pluggable Transports Component of Linux Bundle (3/4 for Linux) ******
****** Starting Bundling+Localization of Linux Bundle (4/4 for Linux) ******
****** Linux Bundle complete ******
****** Starting Tor Component of Windows Bundle (1/4 for Windows) ******
****** Starting Torbrowser Component of Windows Bundle (2/4 for Windows) ******
****** Starting Pluggable Transports Component of Windows Bundle (3/4 for Windows) ******
****** Starting Bundling+Localization of Windows Bundle (4/4 for Windows) ******
****** Windows Bundle complete ******
****** Starting Tor Component of Mac Bundle (1/4 for Mac******
****** Starting Torbrowser Component of Mac Bundle (2/4 for Mac) ******
****** Starting Pluggable Transports Component of Mac Bundle (3/4 for Mac) ******
****** Starting Bundling+Localization of Mac Bundle (4/4 for Mac) ******
****** Mac Bundle complete ******
These lines correspond to the build stages of the respective Linux, Windows, and Mac bundles.
Knowing When the Build Succeeded
The TBB build succeeds just after the Mac bundle completes. Therefore, to confirm success look for:
****** Mac Bundle complete ******
...in the final lines of build output. Additionally, a number of bundles are found in the gitian/<version> subdirectory.
$ ls tor-browser-bundle/gitian/3.5.4 pluggable-transports-linux32-debug.zip pluggable-transports-linux64-debug.zip TorBrowserBundle-3.5.4-osx32_ar.zip TorBrowserBundle-3.5.4-osx32_de.zip TorBrowserBundle-3.5.4-osx32_en-US.zip TorBrowserBundle-3.5.4-osx32_es-ES.zip [...]
Verification of Build Products
Once a TBB build succeeds, make(1) match returns text indicating if the bundles match the hashes of valid (from independent deterministic built) bundles.
The duration of TBB builds varies according to hardware type and availability of resources like memory and swap. For guidance in VM guest creation and corresponding requirements please review the VM guest setup document.
|VM||Ubuntu||12.04.4||Server||AMD64||3 Go||2 Go||4||Phenom X4 4GHz||9||Workstation|
|BM||Ubuntu||14.04||Desktop||AMD64||8 Go||4 Go||4||Core i5-2537M 1.4GHz||32||Notebook|
1 Bare metal host (BM) using KVM or virtual machine guest (VM) using LXC
2 OS preconfigured package selections (often desktop or server)
3 Total processing units (processors, cores, hyperthreads?)
4 Processor clock rate and performance influencing factors
(!) The canonical indicator (what we're interested to measure)
Gitian Build Tips
Useful log files are produced (and then deleted as each stage succeeds) and are extremely helpful in debugging build problems. Consider opening a new window and following log output as gitian produces it:
$ tail -F gitian-builder/var/install.log $ tail -F gitian-builder/var/build.log
Or even better, install multitail(1) and monitor each of the two main logs from start to finish:
$ sudo apt-get install multitail $ multitail gitian-builder/var/install.log gitian-builder/var/build.log
Don't be fooled by storage measurements of completed builds. Such late stage analysis doesn't reflect temporary build content stored during the build (which is quite a bit more than the resulting bundles.)
$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 34G 20G 14G 58% / $ du -hs gitian-builder tor-browser-bundle 16G gitian-builder 1,9G tor-browser-bundle
Troubleshooting Build Failures
The gitian build sequence is complex and uses tools not particularly designed to work together. Thus, the nature of gitian building is fragile and new TBB builds typically break at some point.
Rather than 'You ran out of memory', error messages indicating RAM exhaustion typically look like this:
collect2: ld terminated with signal 9 [killed]
This is quite common when shaving the bits off minimum RAM requirements and gitian starts linking libxul.so, for example:
make: *** [libxul.so] Error 1
Obviously to solve the problem of RAM exhaustion, more memory is needed. This is trivial if building on a memory reduced VM, but if not there's still the chance of increasing swap and hoping for the best.
Gitian failures with LXC on Ubuntu 14.04
If one sees Gitian failures like
bash: cannot set terminal process group (-1): Inappropriate ioctl for device bash: no job control in this shell
then this is a sign that new containers need to get created. Related to that lxc-execute is required for building Tor Browser using LXC since 14.04. This is off by default (to support older Ubuntu versions) but can be activated by setting the proper environment variable before starting the build:
#15947 has further details for interested parties.
The version of apt-cacher-ng(1) distributed in Ubuntu 12.04.4 enjoys strangulating guest (KVM or LXC) gitian instances. When the window where gitian/make(1) was issued returns:
Updating apt-get repository (log in var/install.log) Installing additional packages (log in var/install.log) ./bin/gbuild:21:in `system!': failed to run on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install git-core unzip zip swig m4 python-setuptools python-dev faketime libtool > var/install.log 2>&1 (RuntimeError) from ./bin/gbuild:93:in `build_one_configuration' from ./bin/gbuild:224 from ./bin/gbuild:219:in `each' from ./bin/gbuild:219 from ./bin/gbuild:217:in `each' from ./bin/gbuild:217 make: *** [build] Fehler 1 Command exited with non-zero status 2
...or when the the gitian-builder/var/install.log indicates a call to apt-get(1) that returns:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/<pkg>.deb Size mismatch
...or when encountering hash sum mismatches when fetching and comparing input checksums:
Fetched 18 MB in 1min (300 kB/s) , stderr: bzip2: (stdin) is not a bzip2 file. W: Failed to fetch gzip:/var/lib/apt/lists/partial/127.0.0.1:3142_archive.ubuntu.com_ubuntu_dists_precise_universe_binary-amd64_Packages Hash Sum mismatch E: Some index files failed to download. They have been ignored, or old ones used instead. qemu-img: target-precise-amd64.qcow2: Could not open 'base-precise-amd64.qcow2': Could not open 'base-precise-amd64.qcow2': No such file or directory: No such file or directory amd64 precise VM creation failed
...or similar, then it's time to kill and restart the apt-cacher-ng(1) process in the controlling (outer) gitian host. That means, log into the machine where gitian/make(1) build was executed and:
$ ps aux | grep apt-cacher-ng 123 991 0.6 0.0 254036 2500 ? Ssl 07:17 1:36 /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng pidfile=/var/run/apt-cacher-ng/pid SocketPath=/var/run/apt-cacher-ng/socket foreground=0 $ kill -HUP 991 # use kill -TERM 991 if HUP doesn't work $ sudo -u \#123 /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng pidfile=<MAKE_SURE_TO_COPY_THE_ENTIRE_PROCESS_ARGS_HERE>
Another solution involves removing resudual cache cruft:
rm -rf /var/lib/apt/lists && mkdir -p /var/lib/apt/lists/partial
A yet third option is to edit gitian-builder/bin/make-base-vm and remove 127.0.0.1 from the MIRROR_HOST, directly fetching packages every time. (Obviously this may use more bandwidth/be slower.)
Flawed Software Hangs
Building TBB on some (newer) systems like Ubuntu 14.04 (LTS) AMD64 Desktop employs components affected with networking problems leading to unexpected disconnections and indirectly causing hangs in the build process (actually it's the 'prep' stage.) This is indicated by the top(1) command listing the component in question at the top of the list consuming 100% CPU cycles. For example, this can happen when downloading dependencies with svn(1) in conjunction with torsocks(1). To solve this particular problem repeat the build sequence using the svn(1) command without torsocks(1), the component actually at fault:
tor-browser-bundle/gitian$ make prep TORSOCKS=
Assembly Errors in Mismatched Architecture Code or 'failed to run on-target setarch'
A general problem that can manifest in a few ways:
x86cpuid.s Err: invalid instruction suffix for `push'
Preparing build environment setarch: x86_64: Unrecognized architecture ./bin/gbuild:21:in `system!': failed to run on-target setarch x86_64 bash < target-bin/init-build.sh (RuntimeError) from ./bin/gbuild:82:in `build_one_configuration' from ./bin/gbuild:224:in `block (2 levels) in <main>' from ./bin/gbuild:219:in `each' from ./bin/gbuild:219:in `block in <main>' from ./bin/gbuild:217:in `each' from ./bin/gbuild:217:in `<main>'
In these situations, it meant that kvm was running for the wrong architecture. It happens when the VM for gitian-utils amd64 (or i386) didn't shut down properly, and the build for gitian-tor 386 (or amd64) started up using the running VM for the wrong architecture. To fix it, kill the kvm process (in difference circumstances, running 'poweroff' inside the VM works, other times the kill must be from outside), and start another "make build". If the problem is indeed that the wrong VM is running, then installing gcc-multilib may not be the right thing to do. But if you want to attempt it...
Assembly errors have been observed with output in gitian-builder/var/build.log resembling:
x86cpuid.s Err: invalid instruction suffix for `push' x86cpuid.s Err: invalid instruction suffix for `pop' [...]
While this specific example refers to OpenSSL, it's a typical side effect of compiling IA32 assembly code on AMD64 platforms lacking certain assembler components, probably available in gcc-multilib or similar (in the build guest not the host!)
To investigate and work around this problem, start building as usual:
tor-browser-bundle/gitian$ make build
...but once 'Building tor-linux for lucid i386' appears, run:
tor-browser-bundle/gitian$ cd ../../gitian-builder gitian-builder$ libexec/on-target -u root root@ubuntu~# apt-get install gcc-multilib root@ubuntu~# cd /home/ubuntu/build/openssl-1.0.1g
By the way, installing gcc-multilib (or possibly libc6-dev-i386) is a workaround in consideration in a partly related bug report #12391 about mingw-w64 components.
Package Management Failures
The apt-get(1) package manager is used in the prep stages to create virtual machines or linux containers. A variety of errors can cause gitian-builder to fail.
WARNING: The following packages cannot be authenticated! E: There are problems and -y was used without --force-yes
In most cases the seemingly random failure is neither deterministic nor indicative of any real problem. Restart the build and cross fingers, toes, or press the thumbs:
tor-browser-bundle/gitian$ make prep(|-alpha|-beta|-nightly) tor-browser-bundle/gitian$ make build(|-alpha|-beta|-nightly)
Random Build Failures
There are times during the gitian build sequence when it simply hangs or stops for reasons that seem random. Rest assured that 'Ctrl-C' and 'make build' are often the best course of action, and that repeating this results in incremental build success.
If you feel your build problems are overly chaotic please contact your representative in congress or speak with a psychic counsellor. Otherwise, the friendly folks hanging out in the #tor-dev chat room (on irc.oftc.org) are ready to help answer build questions. Don't forget to update this document when solutions are found!
Other Related Resources
Other documents that aide in further understanding of using gitian-build for Tor software include:
- So, You Want to Hack on Tor Browser!
- Mike Perry's Tor blog post that started it all
- Mike Perry and Seth Schoen at CCC 31c3
- Tor Trac 'Gitian documentation lint magnet' ticket
Upstream Gitian Documentation
Additional help can be obtained by appeals to #bitcoin-dev (irc.freenode.net) or a number of online resources: