Deterministic builds

[mikeperry]

To ensure integrity against build machine compromise, we should be able to produce identical binaries on two different identically configured machines and verify that hash is the same for each. Right now, this is not possible, primarily because of two things:

1. gcc uses entropy for symbol mangling
2. The linker inserts timestamps into libraries, especially static ones.

Issue 1 can be solved by giving gcc a specific seed in our makefiles (-frandom-seed=string). If we have no collisions, we can get away with giving the same seed to every gcc invocation.

Issue 2 can be solved for static libraries by passing the -D option to 'ar'. It is unclear if shared libraries can be produced in this way, or if this option is not needed for shared libraries.

On Windows, the problem remains entirely unsolved:
​http://stackoverflow.com/questions/1180852/deterministic-builds-under-windows

However, if we can do this for Linux and Mac OS using the same build flags, that would still be worth it.

[mikeperry]

FYI the freebsd list discussion about this ended with a patch:
​http://www.mail-archive.com/freebsd-hackers@freebsd.org/msg158761.html

However, shared libraries were not mentioned. So either it was not an issue, or they didn't care about shared libraries.

[mikeperry]

Weird. tor.git is already creating deterministic builds (same sha1sum for ./src/or/tor) for me without the -frandom-seed option. It has static symbols.. I don't get why that is. Maybe Nick knows?

[mikeperry]

Vidalia also seems to be reproducible (same sha1sum after a make clean && make). I wonder if this is due to a custom patch my distro has done, or if these changes got propagated through the toolchain defaults.

Erinn, is your build process fully automated? I get errors when trying to follow the instructions in build-scripts/DEPLOYMENT

[erinn]

Yikes, that file is out of date. You should be able to just use the makefiles, eg, make -f linux.mk build-tor, and within that makefile there are various build options associated with all of the targets.

[erinn]

Gonna attempt this for the next release.

[ioerror]

I don't find that make -f linux.mk works by default - only some of the targets seem to work and not all of them. I'm going to try to produce a linux tbb build and hack up the Makefile until it works.

[ioerror]

I've added a build-dep step:

apt-get build-dep vidalia openssl zlib firefox libpng libevent

[ioerror]

I've found that the generic-bundle target does not function:

cp ../changelog.linux-2.2 generic-bundle/Docs/changelog
# This should be updated to be more generic (version-wise) and more Linux specific
cp ../README.LINUX-2.2 generic-bundle/Docs/README-TorBrowserBundle
cp -R /tmp/srv/build-trees/build-alpha-x86_64/Firefox generic-bundle/App
cp: cannot stat `/tmp/srv/build-trees/build-alpha-x86_64/Firefox': No such file or directory
make: *** [install-firefox] Error 1

[ioerror]

Erinn - what targets are you using for the Linux TBB builds? I must be doing something wrong here, right?

[ioerror]

Ok, I did this to pack it all up after building:

make -f linux.mk copy-firefox bundle clean compressed-bundle_en-US

It resulted in a 29MB tbbl-alpha-dist/tor-browser-gnu-linux-x86_64-2.2.33-3-dev-en-US.tar.gz file! Huzzah!

[mikeperry]

Ok, so the next step is to have Erinn publish the distro + version of the build system she uses, so we can try to replicate the setup and produce an identical binary.

I'll commit to trying a TBB alpha build once Erinn tells us this info + provides some good build instructions so that we can replicate her steps *exactly*, and see if we get the same sha1sum. It's about time I learn how to build the damn thing anyways.

[mikeperry]

Ooh, Seth Schoen just pointed me at this: ​http://gitian.org/. It appears to be a collection of Ruby scripts for facilitating distributed, reproducible builds in qemu vms.

[mikeperry]

I just found out we use mingw for Windows builds a few days ago. That thing might actually be able to produce the same binary twice in a row (assuming *their* build servers are not infected with malware by now..).

[proper]

Any progress on this one?

Replying to mikeperry:

Ok, so the next step is to have Erinn publish the distro + version of the build system she uses, so we can try to replicate the setup and produce an identical binary.

I'll commit to trying a TBB alpha build once Erinn tells us this info + provides some good build instructions so that we can replicate her steps *exactly*, and see if we get the same sha1sum. It's about time I learn how to build the damn thing anyways.

After waiting for 4 months I'd ping her.

[cypherpunks]

what about the debian packages?

[cypherpunks]

Made two builds of 2.2.37-2, one Linux, one Windows.
They can be found in:
​https://people.torproject.org/~shondoit/builds/

So, if anyone's interested in doing a bindiff between these and the official builds...
Might be a good start to get a feel for what we're up against.

• Shondoit

[hiviah]

I did a test building 0.2.3.20 Tor rpms (just the daemon, not whole TBB) on two Fedora 17 and one Scientific Linux 6 machines using mock (mock is a Redhat tool builds in a chroot environment with cleanly downloaded dependencies and toolchain; it can cross-compile as well).

Suprisingly the resulting tor binaries ended up identical accross build machines as long as the target configuration was identical, e.g. every binary built for 'epel-6-x86_64' configuration was bit-identical. Around 10 combinations [build_machine, target_configuration] were tried.

Then I ran a TBB build and tried to build some of the components deterministically. Qt 4.8.2 from src.rpm gave me almost identical builds (compared to distro libs), libQtCore.so and libQtGui.so differing only in 24 and 27 bytes, respectively. The differences appear in strings like 'qt_instdate=2012-08-11' and the ELF NOTE program header.

[naif]

FYI that's a topic that also APAF (anonymous python application framework) would like to evaluate for uses with Py2exe and Py2app:

​https://github.com/mmaker/APAF/issues/21

[mikeperry]

Here's how the bitcoin people use gitian:
​https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.txt
​https://github.com/bitcoin/bitcoin/tree/master/contrib/gitian-descriptors

It looks like they use a utility called 'faketime' to eliminate timestamp differences, and it appears to be a debian package. It also looks like they use it successfully for win32 under mingw, too. There is a separate pile of python scripts for Mac.

[mikeperry]

Ok, using the hacks from bitcoin's Windows gitian descriptors on Linux, I got Firefox to build almost identically a few times in a row. The only differences were in these files: libplc4.so, libplds4.so, libnspr4.so, and omni.ja.

It looks like all of the differences are due to faketime not faking the milliseconds in timestamps in those files. This might be because they didn't (properly?) hook gettimeofday(), but it looks like they at least tried to do so: ​https://github.com/wolfcw/libfaketime/blob/master/src/faketime.c#L519, so the problem may be deeper.

Here's what I did:

export CFLAGS="-frandom-seed=tor"
export CXXFLAGS="-frandom-seed=tor"
export TZ=UTC
faketime -f "1970-01-01 00:00:00" bash
make -f client.mk build && make -C obj-* package INNER_MAKE_PACKAGE=true

When I did this same thing to the full TBB, though, I got a make loop in Qt and the build never finished...

[mikeperry]

Ok, I found the culprit: nsprpub/config/now.c. It gets called during build time to set a #define, and worse, it has specific calls to GetSystemTimeAsFileTime() for Windows. That is definitely not hooked by faketime.

Luckily, it seems to have a #define to omit the build timestamp entirely. I will test a build with that and see what happens.

[mikeperry]

Ok, I just committed a Firefox patch to origin/maint-2.4 that allows me to now build Firefox deterministically using the above snippet from the previous comment:
​https://gitweb.torproject.org/torbrowser.git/blob/maint-2.4:/src/current-patches/firefox/0029-Disable-library-timestamping.patch

However, there is a library signing process for NSS where a utility called 'shlibsign' generates a temporary signing key that lives only in memory, and then signs all the NSS libs with it:
​https://www.mozilla.org/projects/security/pki/nss/tech-notes/tn6.html

One thing we can do is have the first build publish these .chk files somewhere the other builds can retrieve during their build process.

The other thing we can do is simply omit the .chk files (which would 'disable' FIPS-140 mode, whatever that means).

[mikeperry]

I have fully deterministic TBB builds with Tor Launcher working for all three platforms. Build instructions are here:
​https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/README.build

There are still some wrinkles to iron out, but we should be able to do an alpha release with this build system in the next few days.

[gk]

I played a bit with it. Great work, Mike! Some feedback/questions: 1) After I shut down my computer during the first build I wanted to continue with the build later (or restart it) but gitian (probably) complained that the VMs are already available and did not let me. I thought there was some kind of clean target for that case in the makefile, too, but eventually I had to delete the already existing VMs by hand. Not a big deal but if there were such an additional target in the makefile that would be fine. 2) Do you check the signatures for the language packs (or more precisely: the signature of the SHA512SUMS file + the contained sha512 hashes of the language packs)? That does not seem to be the case or did I miss something? 3) Do you have a result of your builds somewhere to check whether I really got the same binaries?

[gk]

Following the steps you outlined on IRC (taking the tbb-3.0alpha1-close-enough tag etc.) I got the following build error:

****** Starting Tor Component of Linux Bundle (1/3 for Linux) ******

--- Building tor-linux for lucid i386 ---
Stopping target if it is up
Making a new image copy
Formatting 'target-lucid-i386.qcow2', fmt=qcow2 size=11811160064 backing_file='base-lucid-i386.qcow2' encryption=off cluster_size=65536 
Starting target
Checking if target is up..........
Connection timed out during banner exchange
./bin/gbuild:21:in `system!': failed to run on-target true (RuntimeError)
	from ./bin/gbuild:73:in `build_one_configuration'
	from ./bin/gbuild:223
	from ./bin/gbuild:218:in `each'
	from ./bin/gbuild:218
	from ./bin/gbuild:216:in `each'
	from ./bin/gbuild:216
mv: cannot stat `var/build.log': No such file or directory
make: *** [build] Error 1

Rerunning |make| does not help.

[gk]

SHA256 hashes for builds with b28ee04d6ffbd161d6f5a68156eee1aa11c1ac51

[gk]

After running at least into issue 1, 2 and 5 mentioned in README.build I eventually succeeded. See the attached SHA256 sums. (although I am wondering why there were no 64bit Mac OS X builds done)

[mikeperry]

I think now that the builds have matched for 3.0a2, we can finally call this done.