Opened 8 years ago

Closed 6 years ago

#3688 closed enhancement (fixed)

Deterministic builds

Reported by: mikeperry Owned by: mikeperry
Priority: High Milestone: TorBrowserBundle 2.3.x-stable
Component: Applications/Tor bundles/installation Version:
Severity: Keywords: MikePerry201306
Cc: nickm, asn, mikeperry, Sebastian, tichodroma@…, g.koppen@…, drwhax@…, proper@…, Shondoit, ondrej.mikle@… Actual Points: 250
Parent ID: Points:
Reviewer: Sponsor:

Description

To ensure integrity against build machine compromise, we should be able to produce identical binaries on two different identically configured machines and verify that hash is the same for each. Right now, this is not possible, primarily because of two things:

  1. gcc uses entropy for symbol mangling
  2. The linker inserts timestamps into libraries, especially static ones.

Issue 1 can be solved by giving gcc a specific seed in our makefiles (-frandom-seed=string). If we have no collisions, we can get away with giving the same seed to every gcc invocation.

Issue 2 can be solved for static libraries by passing the -D option to 'ar'. It is unclear if shared libraries can be produced in this way, or if this option is not needed for shared libraries.

On Windows, the problem remains entirely unsolved:
http://stackoverflow.com/questions/1180852/deterministic-builds-under-windows

However, if we can do this for Linux and Mac OS using the same build flags, that would still be worth it.

Child Tickets

Attachments (1)

b28ee04d6ffbd161d6f5a68156eee1aa11c1ac51_sha256 (5.5 KB) - added by gk 6 years ago.
SHA256 hashes for builds with b28ee04d6ffbd161d6f5a68156eee1aa11c1ac51

Download all attachments as: .zip

Change History (33)

comment:1 Changed 8 years ago by mikeperry

FYI the freebsd list discussion about this ended with a patch:
http://www.mail-archive.com/freebsd-hackers@freebsd.org/msg158761.html

However, shared libraries were not mentioned. So either it was not an issue, or they didn't care about shared libraries.

comment:2 Changed 8 years ago by mikeperry

Cc: nickm asn added

Weird. tor.git is already creating deterministic builds (same sha1sum for ./src/or/tor) for me without the -frandom-seed option. It has static symbols.. I don't get why that is. Maybe Nick knows?

comment:3 Changed 8 years ago by mikeperry

Vidalia also seems to be reproducible (same sha1sum after a make clean && make). I wonder if this is due to a custom patch my distro has done, or if these changes got propagated through the toolchain defaults.

Erinn, is your build process fully automated? I get errors when trying to follow the instructions in build-scripts/DEPLOYMENT

comment:4 Changed 8 years ago by erinn

Yikes, that file is out of date. You should be able to just use the makefiles, eg, make -f linux.mk build-tor, and within that makefile there are various build options associated with all of the targets.

comment:5 Changed 8 years ago by erinn

Keywords: tbb-2.2.32-5 added

Gonna attempt this for the next release.

comment:6 Changed 8 years ago by ioerror

I don't find that make -f linux.mk works by default - only some of the targets seem to work and not all of them. I'm going to try to produce a linux tbb build and hack up the Makefile until it works.

comment:7 Changed 8 years ago by ioerror

I've added a build-dep step:

apt-get build-dep vidalia openssl zlib firefox libpng libevent

comment:8 Changed 8 years ago by ioerror

I've found that the generic-bundle target does not function:

cp ../changelog.linux-2.2 generic-bundle/Docs/changelog
# This should be updated to be more generic (version-wise) and more Linux specific
cp ../README.LINUX-2.2 generic-bundle/Docs/README-TorBrowserBundle
cp -R /tmp/srv/build-trees/build-alpha-x86_64/Firefox generic-bundle/App
cp: cannot stat `/tmp/srv/build-trees/build-alpha-x86_64/Firefox': No such file or directory
make: *** [install-firefox] Error 1

comment:9 Changed 8 years ago by ioerror

Erinn - what targets are you using for the Linux TBB builds? I must be doing something wrong here, right?

comment:10 Changed 8 years ago by ioerror

Ok, I did this to pack it all up after building:

make -f linux.mk copy-firefox bundle clean compressed-bundle_en-US

It resulted in a 29MB tbbl-alpha-dist/tor-browser-gnu-linux-x86_64-2.2.33-3-dev-en-US.tar.gz file! Huzzah!

comment:11 Changed 7 years ago by mikeperry

Cc: mikeperry added
Milestone: TorBrowserBundle 2.3.x-stable
Owner: changed from erinn to mikeperry
Status: newassigned

Ok, so the next step is to have Erinn publish the distro + version of the build system she uses, so we can try to replicate the setup and produce an identical binary.

I'll commit to trying a TBB alpha build once Erinn tells us this info + provides some good build instructions so that we can replicate her steps *exactly*, and see if we get the same sha1sum. It's about time I learn how to build the damn thing anyways.

comment:12 Changed 7 years ago by mikeperry

Cc: Sebastian added

Ooh, Seth Schoen just pointed me at this: http://gitian.org/. It appears to be a collection of Ruby scripts for facilitating distributed, reproducible builds in qemu vms.

comment:13 Changed 7 years ago by Tichodroma

Cc: tichodroma@… added

comment:14 Changed 7 years ago by gk

Cc: g.koppen@… added

comment:15 Changed 7 years ago by mikeperry

Summary: Deterministic builds for Linux and Mac OSDeterministic builds

I just found out we use mingw for Windows builds a few days ago. That thing might actually be able to produce the same binary twice in a row (assuming *their* build servers are not infected with malware by now..).

comment:16 Changed 7 years ago by DrWhax

Cc: drwhax@… added

comment:17 in reply to:  11 Changed 7 years ago by proper

Cc: proper@… added

Any progress on this one?

Replying to mikeperry:

Ok, so the next step is to have Erinn publish the distro + version of the build system she uses, so we can try to replicate the setup and produce an identical binary.

I'll commit to trying a TBB alpha build once Erinn tells us this info + provides some good build instructions so that we can replicate her steps *exactly*, and see if we get the same sha1sum. It's about time I learn how to build the damn thing anyways.

After waiting for 4 months I'd ping her.

comment:18 Changed 7 years ago by cypherpunks

what about the debian packages?

comment:19 Changed 7 years ago by Shondoit

Cc: Shondoit added

comment:20 Changed 7 years ago by cypherpunks

Made two builds of 2.2.37-2, one Linux, one Windows.
They can be found in:
https://people.torproject.org/~shondoit/builds/

So, if anyone's interested in doing a bindiff between these and the official builds...
Might be a good start to get a feel for what we're up against.

  • Shondoit

comment:21 Changed 7 years ago by hiviah

Cc: ondrej.mikle@… added

I did a test building 0.2.3.20 Tor rpms (just the daemon, not whole TBB) on two Fedora 17 and one Scientific Linux 6 machines using mock (mock is a Redhat tool builds in a chroot environment with cleanly downloaded dependencies and toolchain; it can cross-compile as well).

Suprisingly the resulting tor binaries ended up identical accross build machines as long as the target configuration was identical, e.g. every binary built for 'epel-6-x86_64' configuration was bit-identical. Around 10 combinations [build_machine, target_configuration] were tried.

Then I ran a TBB build and tried to build some of the components deterministically. Qt 4.8.2 from src.rpm gave me almost identical builds (compared to distro libs), libQtCore.so and libQtGui.so differing only in 24 and 27 bytes, respectively. The differences appear in strings like 'qt_instdate=2012-08-11' and the ELF NOTE program header.

comment:22 Changed 7 years ago by naif

FYI that's a topic that also APAF (anonymous python application framework) would like to evaluate for uses with Py2exe and Py2app:

https://github.com/mmaker/APAF/issues/21

comment:23 Changed 6 years ago by mikeperry

Here's how the bitcoin people use gitian:
https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.txt
https://github.com/bitcoin/bitcoin/tree/master/contrib/gitian-descriptors

It looks like they use a utility called 'faketime' to eliminate timestamp differences, and it appears to be a debian package. It also looks like they use it successfully for win32 under mingw, too. There is a separate pile of python scripts for Mac.

comment:24 Changed 6 years ago by mikeperry

Ok, using the hacks from bitcoin's Windows gitian descriptors on Linux, I got Firefox to build almost identically a few times in a row. The only differences were in these files: libplc4.so, libplds4.so, libnspr4.so, and omni.ja.

It looks like all of the differences are due to faketime not faking the milliseconds in timestamps in those files. This might be because they didn't (properly?) hook gettimeofday(), but it looks like they at least tried to do so: https://github.com/wolfcw/libfaketime/blob/master/src/faketime.c#L519, so the problem may be deeper.

Here's what I did:

export CFLAGS="-frandom-seed=tor"
export CXXFLAGS="-frandom-seed=tor"
export TZ=UTC
faketime -f "1970-01-01 00:00:00" bash
make -f client.mk build && make -C obj-* package INNER_MAKE_PACKAGE=true

When I did this same thing to the full TBB, though, I got a make loop in Qt and the build never finished...

comment:25 Changed 6 years ago by mikeperry

Ok, I found the culprit: nsprpub/config/now.c. It gets called during build time to set a #define, and worse, it has specific calls to GetSystemTimeAsFileTime() for Windows. That is definitely not hooked by faketime.

Luckily, it seems to have a #define to omit the build timestamp entirely. I will test a build with that and see what happens.

comment:26 Changed 6 years ago by mikeperry

Ok, I just committed a Firefox patch to origin/maint-2.4 that allows me to now build Firefox deterministically using the above snippet from the previous comment:
https://gitweb.torproject.org/torbrowser.git/blob/maint-2.4:/src/current-patches/firefox/0029-Disable-library-timestamping.patch

However, there is a library signing process for NSS where a utility called 'shlibsign' generates a temporary signing key that lives only in memory, and then signs all the NSS libs with it:
https://www.mozilla.org/projects/security/pki/nss/tech-notes/tn6.html

One thing we can do is have the first build publish these .chk files somewhere the other builds can retrieve during their build process.

The other thing we can do is simply omit the .chk files (which would 'disable' FIPS-140 mode, whatever that means).

comment:27 Changed 6 years ago by mikeperry

Keywords: MikePerry201305 added; tbb-2.2.32-5 removed

I have fully deterministic TBB builds with Tor Launcher working for all three platforms. Build instructions are here:
https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/README.build

There are still some wrinkles to iron out, but we should be able to do an alpha release with this build system in the next few days.

comment:28 Changed 6 years ago by gk

I played a bit with it. Great work, Mike! Some feedback/questions: 1) After I shut down my computer during the first build I wanted to continue with the build later (or restart it) but gitian (probably) complained that the VMs are already available and did not let me. I thought there was some kind of clean target for that case in the makefile, too, but eventually I had to delete the already existing VMs by hand. Not a big deal but if there were such an additional target in the makefile that would be fine. 2) Do you check the signatures for the language packs (or more precisely: the signature of the SHA512SUMS file + the contained sha512 hashes of the language packs)? That does not seem to be the case or did I miss something? 3) Do you have a result of your builds somewhere to check whether I really got the same binaries?

comment:29 Changed 6 years ago by gk

Following the steps you outlined on IRC (taking the tbb-3.0alpha1-close-enough tag etc.) I got the following build error:

****** Starting Tor Component of Linux Bundle (1/3 for Linux) ******

--- Building tor-linux for lucid i386 ---
Stopping target if it is up
Making a new image copy
Formatting 'target-lucid-i386.qcow2', fmt=qcow2 size=11811160064 backing_file='base-lucid-i386.qcow2' encryption=off cluster_size=65536 
Starting target
Checking if target is up..........
Connection timed out during banner exchange
./bin/gbuild:21:in `system!': failed to run on-target true (RuntimeError)
	from ./bin/gbuild:73:in `build_one_configuration'
	from ./bin/gbuild:223
	from ./bin/gbuild:218:in `each'
	from ./bin/gbuild:218
	from ./bin/gbuild:216:in `each'
	from ./bin/gbuild:216
mv: cannot stat `var/build.log': No such file or directory
make: *** [build] Error 1

Rerunning |make| does not help.

Changed 6 years ago by gk

SHA256 hashes for builds with b28ee04d6ffbd161d6f5a68156eee1aa11c1ac51

comment:30 Changed 6 years ago by gk

After running at least into issue 1, 2 and 5 mentioned in README.build I eventually succeeded. See the attached SHA256 sums. (although I am wondering why there were no 64bit Mac OS X builds done)

comment:31 Changed 6 years ago by mikeperry

Keywords: MikePerry201306 added; MikePerry201305 removed

comment:32 Changed 6 years ago by mikeperry

Actual Points: 250
Resolution: fixed
Status: assignedclosed

I think now that the builds have matched for 3.0a2, we can finally call this done.

Note: See TracTickets for help on using tickets.