Skip to content
Snippets Groups Projects
New look: Off

OS X builds are still not reproducible on some machines

    • Closed (moved) Issue created by Georg Koppen @gk

    After fixing #19856 (moved) there are still issues with the reproducibility of OS X alpha builds. While I got only a differing openssl binary compared to boklm or ln5 the tor executable is still different. My guess about what happened is that there was more than one problem and #19856 (moved) fixed only the first one showing up.

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    • Georg Koppen added GeorgKoppen201610 TorBrowserTeam201610R component::applications/tor browser owner::tbb-team priority::very high resolution::fixed severity::major status::closed tbb-gitian type::defect labels

      added GeorgKoppen201610 TorBrowserTeam201610R component::applications/tor browser owner::tbb-team priority::very high resolution::fixed severity::major status::closed tbb-gitian type::defect labels

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Okay, here is the summary of what I have so far:

      1. This happens on my machine for stable builds as well, sometimes. So far, it seems I have been "lucky" in that I got always matching builds.

      2. This is probably not faketime/timestamp related.

      After quite some amount of testing and reverting da14737254245cfa6b849dc8ac94a52cb3d9a61d it turns out that there are still differences between executables. Sometimes tor being among them and sometimes not.

      Another interesting fact us that I don't get different hash values for the tor executable every time. In all of my tests there were only three possible values.

      Attached is a typical diff (in this case between two test-ntor-cl binaries).

      Exciting!

      My current guess is this is related to the old toolchain we use for gitian-utils.yml and gitian-tor.yml for OS X. Not sure why this was not visible earlier, though. Maybe switching to Debian made this more likely to happen?

      My current plan is to use our clang cross-compiler for tor and see whether that fixes those diffs. While I expect that to be much harder for the utils we need (see: #10369 (moved)) we might get away with doing that just for tor for now.

      Trac:
      Summary: OS X alpha builds are still not reproducible on some machines (as of 6.5a3) to OS X builds are still not reproducible on some machines
      Severity: Normal to Major
      Cc: N/A to boklm
      Priority: Medium to Very High

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      test_ntor_cl_diff.txt

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Oh, I have meanwhile seen this behavior on two different machines. One running Debian stable and the other one Ubuntu 14.04.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving my tickets

      Trac:
      Keywords: GeorgKoppen201609 TorBrowserTeam201609 deleted, GeorgKoppen201610, TorBrowserTeam201609 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving tickets to October.

      Trac:
      Keywords: TorBrowserTeam201609 deleted, TorBrowserTeam201610 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Okay, cross-compiling works but tor is still crashing with

      Exception Type:  EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000
Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Dyld Error Message:
  Symbol not found: _memmem
  Referenced from: /Users/release/Desktop/TorBrowser.app/Contents/MacOS/Tor/tor.real
  Expected in: /usr/lib/libSystem.B.dylib
    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Seems using the 10.6 SDK works. \o/

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      bug_20184_v2 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_20184_v2&id=8cb911a5516dd89e99a429d2c2f5912e3a6a326c) in my public tor-browser-bundle has a patch up for review. I tested it on different machines with several builds and I got always the same .zip file. Looking at the artifacts the differences mentioned in comment:1 2) are gone. However, we got a new one only in libcurve25519_donna.a which is probably a bug in the ar tool used. There are always two bytes different, e.g.:

      @@ -5,7 +5,7 @@
 00000040: 2020 600a 5f5f 2e53 594d 4445 4620 534f    `.__.SYMDEF SO
 00000050: 5254 4544 0000 0000 0800 0000 0000 0000  RTED............
 00000060: 8000 0000 1800 0000 5f63 7572 7665 3235  ........_curve25
-00000070: 3531 395f 646f 6e6e 6100 d900 0000 0000  519_donna.......
+00000070: 3531 395f 646f 6e6e 6100 6c01 0000 0000  519_donna.l.....
 00000080: 2331 2f36 3020 2020 2020 2020 2020 2020  #1/60           
 00000090: 3934 3636 3834 3830 3020 2020 3130 3030  946684800   1000
 000000a0: 2020 3130 3030 2020 3130 3036 3434 2020    1000  100644

      It does not affect us at the moment, though.

      For some reason libfaketime is needed again which is a bit surprising and might merit further investigation as we don't seem to need it in the gitian-firefox.yml descriptor.

      I've uploaded a test bundle as I only have an old OSX 10.6 machine available (where it worked):

      https://people.torproject.org/~gk/testbuilds/TorBrowser-6.5a3-osx64_20184_en-US.dmg https://people.torproject.org/~gk/testbuilds/TorBrowser-6.5a3-osx64_20184_en-US.dmg.asc

      Trac:
      Status: new to needs_review
      Keywords: TorBrowserTeam201610 deleted, TorBrowserTeam201610R added

    • A
      Arthur Edelstein @arthuredelstein ·

      Replying to gk:

      bug_20184_v2 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_20184_v2&id=8cb911a5516dd89e99a429d2c2f5912e3a6a326c) in my public tor-browser-bundle has a patch up for review. I tested it on different machines with several builds and I got always the same .zip file. Looking at the artifacts the differences mentioned in comment:1 2) are gone. However, we got a new one only in libcurve25519_donna.a which is probably a bug in the ar tool used. There are always two bytes different, e.g.:

      @@ -5,7 +5,7 @@
 00000040: 2020 600a 5f5f 2e53 594d 4445 4620 534f    `.__.SYMDEF SO
 00000050: 5254 4544 0000 0000 0800 0000 0000 0000  RTED............
 00000060: 8000 0000 1800 0000 5f63 7572 7665 3235  ........_curve25
-00000070: 3531 395f 646f 6e6e 6100 d900 0000 0000  519_donna.......
+00000070: 3531 395f 646f 6e6e 6100 6c01 0000 0000  519_donna.l.....
 00000080: 2331 2f36 3020 2020 2020 2020 2020 2020  #1/60           
 00000090: 3934 3636 3834 3830 3020 2020 3130 3030  946684800   1000
 000000a0: 2020 3130 3030 2020 3130 3036 3434 2020    1000  100644  
}}}
It does not affect us at the moment, though.

For some reason `libfaketime` is needed again which is a bit surprising and might merit further investigation as we don't seem to need it in the gitian-firefox.yml descriptor.

I've uploaded a test bundle as I only have an old OSX 10.6 machine available (where it worked):

https://people.torproject.org/~gk/testbuilds/TorBrowser-6.5a3-osx64_20184_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-6.5a3-osx64_20184_en-US.dmg.asc

      I ran this on a 10.11.5 OS X machine and it everything seems to be working as expected.

      I happened to notice in the patch the following change: {{{

      • i686-apple-darwin11-install_name_tool -change
        INSTDIR/libevent/lib/
        LIBEVENT_FILE @executable_path/$LIBEVENT_FILE tor
      • x86_64-apple-darwin10-install_name_tool -change
        INSTDIR/libevent/lib/
        LIBEVENT_FILE @executable_path/$LIBEVENT_FILE tor
      Is there a reason this is changing from "darwin11" to "darwin10"? I'm not familiar with this tool, so sorry if this is a naive question.
    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to arthuredelstein:

      I happened to notice in the patch the following change: {{{

      • i686-apple-darwin11-install_name_tool -change
        INSTDIR/libevent/lib/
        LIBEVENT_FILE @executable_path/$LIBEVENT_FILE tor
      • x86_64-apple-darwin10-install_name_tool -change
        INSTDIR/libevent/lib/
        LIBEVENT_FILE @executable_path/$LIBEVENT_FILE tor }}} Is there a reason this is changing from "darwin11" to "darwin10"? I'm not familiar with this tool, so sorry if this is a naive question.

      The new cctools which we are using now is compiled for the darwin10 target. That's the only reason for that change. We are not doing that compilation ourselves right now, though, yet (#9711 (moved)).

    • boklm
      boklm @boklm ·

      Replying to gk:

      bug_20184_v2 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_20184_v2&id=8cb911a5516dd89e99a429d2c2f5912e3a6a326c) in my public tor-browser-bundle has a patch up for review.

      This patch looks good to me.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Cherry-picked to master with commit 04a9af0eb1e6466ce6a7007a46d73fb7a38442e8.

      Trac:
      Status: needs_review to closed
      Resolution: N/A to fixed

    • Trac closed

      closed

    • Georg Koppen mentioned in issue #20258 (moved)

      mentioned in issue #20258 (moved)

    • Trac moved to tpo/applications/tor-browser#20184 (closed)

      moved to tpo/applications/tor-browser#20184 (closed)

    Please register or sign in to reply