Opened 3 years ago

Closed 3 years ago

#20184 closed defect (fixed)

OS X builds are still not reproducible on some machines

Reported by: gk Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Major Keywords: tbb-gitian, GeorgKoppen201610, TorBrowserTeam201610R
Cc: boklm Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

After fixing #19856 there are still issues with the reproducibility of OS X alpha builds. While I got only a differing openssl binary compared to boklm or ln5 the tor executable is still different. My guess about what happened is that there was more than one problem and #19856 fixed only the first one showing up.

Child Tickets

Attachments (1)

test_ntor_cl_diff.txt (103.2 KB) - added by gk 3 years ago.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 3 years ago by gk

Cc: boklm added
Priority: MediumVery High
Severity: NormalMajor
Summary: OS X alpha builds are still not reproducible on some machines (as of 6.5a3)OS X builds are still not reproducible on some machines

Okay, here is the summary of what I have so far:

1) This happens on my machine for stable builds as well, sometimes. So far, it seems I have been "lucky" in that I got always matching builds.

2) This is probably not faketime/timestamp related.

After quite some amount of testing and reverting da14737254245cfa6b849dc8ac94a52cb3d9a61d it turns out that there are still differences between executables. Sometimes tor being among them and sometimes not.

Another interesting fact us that I don't get different hash values for the tor executable every time. In all of my tests there were only three possible values.

Attached is a typical diff (in this case between two test-ntor-cl binaries).

Exciting!

My current guess is this is related to the old toolchain we use for gitian-utils.yml and gitian-tor.yml for OS X. Not sure why this was not visible earlier, though. Maybe switching to Debian made this more likely to happen?

My current plan is to use our clang cross-compiler for tor and see whether that fixes those diffs. While I expect that to be much harder for the utils we need (see: #10369) we might get away with doing that just for tor for now.

Changed 3 years ago by gk

Attachment: test_ntor_cl_diff.txt added

comment:2 Changed 3 years ago by gk

Oh, I have meanwhile seen this behavior on two different machines. One running Debian stable and the other one Ubuntu 14.04.

comment:3 Changed 3 years ago by gk

Keywords: GeorgKoppen201610 added; GeorgKoppen201609 removed

Moving my tickets

comment:4 Changed 3 years ago by gk

Keywords: TorBrowserTeam201610 added; TorBrowserTeam201609 removed

Moving tickets to October.

comment:5 Changed 3 years ago by gk

Okay, cross-compiling works but tor is still crashing with

Exception Type:  EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000
Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Dyld Error Message:
  Symbol not found: _memmem
  Referenced from: /Users/release/Desktop/TorBrowser.app/Contents/MacOS/Tor/tor.real
  Expected in: /usr/lib/libSystem.B.dylib

comment:6 Changed 3 years ago by gk

Seems using the 10.6 SDK works. \o/

comment:7 Changed 3 years ago by gk

Keywords: TorBrowserTeam201610R added; TorBrowserTeam201610 removed
Status: newneeds_review

bug_20184_v2 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_20184_v2&id=8cb911a5516dd89e99a429d2c2f5912e3a6a326c) in my public tor-browser-bundle has a patch up for review. I tested it on different machines with several builds and I got always the same .zip file. Looking at the artifacts the differences mentioned in comment:1 2) are gone. However, we got a new one only in libcurve25519_donna.a which is probably a bug in the ar tool used. There are always two bytes different, e.g.:

@@ -5,7 +5,7 @@
 00000040: 2020 600a 5f5f 2e53 594d 4445 4620 534f    `.__.SYMDEF SO
 00000050: 5254 4544 0000 0000 0800 0000 0000 0000  RTED............
 00000060: 8000 0000 1800 0000 5f63 7572 7665 3235  ........_curve25
-00000070: 3531 395f 646f 6e6e 6100 d900 0000 0000  519_donna.......
+00000070: 3531 395f 646f 6e6e 6100 6c01 0000 0000  519_donna.l.....
 00000080: 2331 2f36 3020 2020 2020 2020 2020 2020  #1/60           
 00000090: 3934 3636 3834 3830 3020 2020 3130 3030  946684800   1000
 000000a0: 2020 3130 3030 2020 3130 3036 3434 2020    1000  100644  

It does not affect us at the moment, though.

For some reason libfaketime is needed again which is a bit surprising and might merit further investigation as we don't seem to need it in the gitian-firefox.yml descriptor.

I've uploaded a test bundle as I only have an old OSX 10.6 machine available (where it worked):

https://people.torproject.org/~gk/testbuilds/TorBrowser-6.5a3-osx64_20184_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-6.5a3-osx64_20184_en-US.dmg.asc

comment:8 in reply to:  7 ; Changed 3 years ago by arthuredelstein

Replying to gk:

bug_20184_v2 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_20184_v2&id=8cb911a5516dd89e99a429d2c2f5912e3a6a326c) in my public tor-browser-bundle has a patch up for review. I tested it on different machines with several builds and I got always the same .zip file. Looking at the artifacts the differences mentioned in comment:1 2) are gone. However, we got a new one only in libcurve25519_donna.a which is probably a bug in the ar tool used. There are always two bytes different, e.g.:

@@ -5,7 +5,7 @@
 00000040: 2020 600a 5f5f 2e53 594d 4445 4620 534f    `.__.SYMDEF SO
 00000050: 5254 4544 0000 0000 0800 0000 0000 0000  RTED............
 00000060: 8000 0000 1800 0000 5f63 7572 7665 3235  ........_curve25
-00000070: 3531 395f 646f 6e6e 6100 d900 0000 0000  519_donna.......
+00000070: 3531 395f 646f 6e6e 6100 6c01 0000 0000  519_donna.l.....
 00000080: 2331 2f36 3020 2020 2020 2020 2020 2020  #1/60           
 00000090: 3934 3636 3834 3830 3020 2020 3130 3030  946684800   1000
 000000a0: 2020 3130 3030 2020 3130 3036 3434 2020    1000  100644  

It does not affect us at the moment, though.

For some reason libfaketime is needed again which is a bit surprising and might merit further investigation as we don't seem to need it in the gitian-firefox.yml descriptor.

I've uploaded a test bundle as I only have an old OSX 10.6 machine available (where it worked):

https://people.torproject.org/~gk/testbuilds/TorBrowser-6.5a3-osx64_20184_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-6.5a3-osx64_20184_en-US.dmg.asc

I ran this on a 10.11.5 OS X machine and it everything seems to be working as expected.

I happened to notice in the patch the following change:

-  i686-apple-darwin11-install_name_tool -change $INSTDIR/libevent/lib/$LIBEVENT_FILE @executable_path/$LIBEVENT_FILE tor
+  x86_64-apple-darwin10-install_name_tool -change $INSTDIR/libevent/lib/$LIBEVENT_FILE @executable_path/$LIBEVENT_FILE tor

Is there a reason this is changing from "darwin11" to "darwin10"? I'm not familiar with this tool, so sorry if this is a naive question.

comment:9 in reply to:  8 Changed 3 years ago by gk

Replying to arthuredelstein:

I happened to notice in the patch the following change:

-  i686-apple-darwin11-install_name_tool -change $INSTDIR/libevent/lib/$LIBEVENT_FILE @executable_path/$LIBEVENT_FILE tor
+  x86_64-apple-darwin10-install_name_tool -change $INSTDIR/libevent/lib/$LIBEVENT_FILE @executable_path/$LIBEVENT_FILE tor

Is there a reason this is changing from "darwin11" to "darwin10"? I'm not familiar with this tool, so sorry if this is a naive question.

The new cctools which we are using now is compiled for the darwin10 target. That's the only reason for that change. We are not doing that compilation ourselves right now, though, yet (#9711).

comment:10 in reply to:  7 Changed 3 years ago by boklm

Replying to gk:

bug_20184_v2 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_20184_v2&id=8cb911a5516dd89e99a429d2c2f5912e3a6a326c) in my public tor-browser-bundle has a patch up for review.

This patch looks good to me.

comment:11 Changed 3 years ago by gk

Resolution: fixed
Status: needs_reviewclosed

Cherry-picked to master with commit 04a9af0eb1e6466ce6a7007a46d73fb7a38442e8.

Note: See TracTickets for help on using tickets.