Move to clang 3.8.0 for Tor Browser's clang-based macOS toolchain

Mozilla switched to clang 3.8.0 for their cross-compilation toolchain a while ago. We should do the same. This has the benefit that we can get away from using some random commit having signed tarballs instead.

added GeorgKoppen201703 TorBrowserTeam201704R component::applications/tor browser ff52-esr owner::tbb-team parent::21147 priority::medium resolution::fixed severity::normal sponsor::4 status::closed tbb-7.0-must-nightly tbb-gitian type::task labels

Just bumping the version is not enough, see https://bugzilla.mozilla.org/show_bug.cgi?id=1273981.

Just bumping the version is not enough now or for ff52-esr too?

Just for ESR52.

Trac:
Parent: N/A to #21147 (moved)

Moving our tickets to Feb 2017.

Trac:
Keywords: TorBrowserTeam201701 deleted, TorBrowserTeam201702 added

This is Sponsor4 work

Trac:
Sponsor: N/A to Sponsor4

Trac:
Keywords: N/A deleted, tbb-7.0-must added

Moving tickets to March

Trac:
Keywords: TorBrowserTeam201702 deleted, TorBrowserTeam201703 added

Adding to my plate.

Trac:
Keywords: N/A deleted, GeorgKoppen201703 added

Okay, here is my plan. clang 3.8.0 requires GCC 4.8 which our current host system does not have. So, we actually have two options to fix that:

1. We build a recent enough GCC (or clang for that matter) first and then build our clang. That could be tricky and time-consuming given a non-standard location of that GCC and its related libs (https://btorpey.github.io/blog/2015/01/02/building-clang/ has a fancy tutorial for that).

2. We could just use Debian Jessie (which comes with GCC 4.9) instead of Debian Wheezy. I tried that today and it worked as expected.

I think I'll do 2) and spend a bit time on getting rid of the old macOS toolchain which allows us to get rid of i386 VMs in the macOS build process. I am almost done with a respective patch for #10369 (moved) which is the bulk of that work. Then we only need to adapt the PT descriptor (i.e. we need to switch to using clang instead of GCC).

It seems moving to the new toolchain has reproducibility problems as a result. :( I just bumped the Debian distro to Jessie and put clang together according to https://bugzilla.mozilla.org/show_bug.cgi?id=1273981 and the resulting dmgs don't match anymore. Using the old clang compiler does not have the same problems.

#16472 (moved)?

Replying to gk:

It seems moving to the new toolchain has reproducibility problems as a result. :( I just bumped the Debian distro to Jessie and put clang together according to https://bugzilla.mozilla.org/show_bug.cgi?id=1273981 and the resulting dmgs don't match anymore. Using the old clang compiler does not have the same problems.

Okay, attached is the diff. I used https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/log/?h=bug_21328_v3. The problematic commit is: https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_21328_v3&id=868433db05b3dc57926e32b79213383483f15f59. Testing with the ones before that one gives me matching builds on the same build machine. Some things we could do to find the problem:

1. We could try to pinpoint where exactly the issue is. Is it in tor code or in openssl code, or...?
2. We could test whether the update to Jessie is causing this (by taking https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_21328_v3&id=6ff54b812b08f3b64fb277f0541dfd9562bf3d6e and just bump everything to Jessie)
3. We could just use the clang part for compiling the remaining utils and tor (and not include libcxx nor libcxxabi which should not be needed anyway) to make sure the issue is inside clang.
4. Assuming this is indeed in clang, because the investigation done in 1) - 3) shows that, we could do some bisecting finding the problematic commit.

boklm: Can you take that one from here?

Trac:
Cc: N/A to boklm

Trac:
tor.diff

We want those tickets for our first ESR52 nightlies.

Trac:
Keywords: N/A deleted, tbb-7.0-must-nightly added

Replying to gk:

boklm: Can you take that one from here? Ok, I'm currently doing some test builds to find which change is causing the diff.

In your tests, was the tor binary the only file that was not reproducible, or did it affect other parts too (such as firefox)?

Replying to boklm:

Replying to gk:

boklm: Can you take that one from here? Ok, I'm currently doing some test builds to find which change is causing the diff.

In your tests, was the tor binary the only file that was not reproducible, or did it affect other parts too (such as firefox)?

Just the tor binary.

It looks like the update to jessie is causing the diff.

I did some test rebuilds of tor, openssl and libevent using rbm (the current master branch is reproducible on OSX).

I used the following patch to update to jessie:

diff --git a/rbm.conf b/rbm.conf
index e206193..0ffdb22 100644
--- a/rbm.conf
+++ b/rbm.conf
@@ -166,7 +166,7 @@ targets:
   torbrowser-osx-x86_64:
     - osx-x86_64
   osx-x86_64:
-    distribution: Debian-7.11
+    distribution: Debian-8.7
     arch: x86_64
     var:
       osx: 1

And I built the tor compenent using this command:

./rbm/rbm build tor --target alpha --target torbrowser-osx-x86_64

After doing two builds, cleaning tor, openssl and libevent between the two builds, I get a non-matching tor binary: https://people.torproject.org/~boklm/tmp/bug_21328/move-jessie/build1/ https://people.torproject.org/~boklm/tmp/bug_21328/move-jessie/build2/

In build3 and build4, I did two builds of the tor component only (without cleaning openssl and libevent between them), and still get a non-matching tor binary: https://people.torproject.org/~boklm/tmp/bug_21328/move-jessie/build3/ https://people.torproject.org/~boklm/tmp/bug_21328/move-jessie/build4/

The reason for this diff seems to be that the path to libfaketime.so.1 changed between wheezy and jessie.

Debian wheezy: /usr/lib/faketime/libfaketime.so.1 Debian jessie: /usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1

So we need to fix the libfaketime path there: https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/tree/gitian/descriptors/mac/gitian-tor.yml?h=bug_21328_v3#n73

Fixing this path removed the diff in my test with rbm (although I did not try with the other changes yet).

This makes sense, thanks. I just pushed bug_21328_v4 (the new commit https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_21328_v4&id=b9ee73abc959b930186138ea203152d1773b24f8) with small changes and the one mentioned in comment:17.

Trac:
Keywords: TorBrowserTeam201703 deleted, TorBrowserTeam201703R added
Status: new to needs_review

Replying to gk:

This makes sense, thanks. I just pushed bug_21328_v4 (the new commit https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_21328_v4&id=b9ee73abc959b930186138ea203152d1773b24f8) with small changes and the one mentioned in comment:17.

I have been building this branch twice and got the same result.

Replying to gk:

This makes sense, thanks. I just pushed bug_21328_v4 (the new commit https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_21328_v4&id=b9ee73abc959b930186138ea203152d1773b24f8) with small changes and the one mentioned in comment:17.

The changes in this branch look good to me.

I also pushed branch bug_21328 in my tor-browser-build.git user repo, doing corresponding changes to the rbm based build: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/log/?h=bug_21328

Moving review tickets to April.

Trac:
Keywords: TorBrowserTeam201703R deleted, TorBrowserTeam201704R added

No need to use somewhat duplicated keywords.

Trac:
Keywords: tbb-7.0-must deleted, N/A added

This is commit cd4465477ac10d2b740d5a32505cdab5a197e405 on master.

Trac:
Status: needs_review to closed
Resolution: N/A to fixed

closed

mentioned in issue #21332 (moved)

moved to tpo/applications/tor-browser#21328 (closed)