Opened 5 years ago

Closed 5 years ago

#12753 closed defect (fixed)

Building pre 4.0-alpha results in non-deterministic Windows builds

Reported by: gk Owned by: gk
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Keywords: tbb-gitian, TorBrowserTeam201408
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Building 4.0-alpha bundles using a6ec17d500fadaa6ce9259ac5a7bd8150504e2a4 results in different timestamps and checksums in the PE header of libeay32.dll. The diff between https://people.torproject.org/~mikeperry/builds/4.0-alpha-1/torbrowser-install-4.0-alpha-1_de.exe and mine is:

 0000050: 6973 2070 726f 6772 616d 2063 616e 6e6f  is program canno
 0000060: 7420 6265 2072 756e 2069 6e20 444f 5320  t be run in DOS 
 0000070: 6d6f 6465 2e0d 0d0a 2400 0000 0000 0000  mode....$.......
-0000080: 5045 0000 4c01 0a00 0000 0000 0000 0000  PE..L...........
+0000080: 5045 0000 4c01 0a00 6163 6b00 0000 0000  PE..L...ack.....
 0000090: 0000 0000 e000 0e23 0b01 0218 0078 1300  .......#.....x..
 00000a0: 00e4 1900 0032 0000 2014 0000 0010 0000  .....2.. .......
 00000b0: 0090 1300 0000 0063 0010 0000 0002 0000  .......c........
 00000c0: 0400 0000 0100 0000 0400 0000 0000 0000  ................
-00000d0: 0090 1a00 0004 0000 e762 1a00 0300 0000  .........b......
+00000d0: 0090 1a00 0004 0000 b3c6 1a00 0300 0000  ................
 00000e0: 0000 2000 0010 0000 0000 1000 0010 0000  .. .............
 00000f0: 0000 0000 1000 0000 00e0 1700 20a9 0100  ............ ...
 0000100: 0090 1900 7010 0000 00d0 1900 3804 0000  ....p.......8...

This might be related to the fix for #12391.

Child Tickets

Change History (19)

comment:1 Changed 5 years ago by gk

Summary: Building pre 4.0-alpha results in non-deterministic windows buildsBuilding pre 4.0-alpha results in non-deterministic Windows builds

comment:2 Changed 5 years ago by gk

Priority: normalmajor

comment:3 Changed 5 years ago by cypherpunks

libeay32.dll

Only one?
Many files contains zero timestamp, but not all:

Time/Date        Thu Feb 11 01:47:15 1999
Time/Date        Fri Jan  2 04:44:52 1970
Time/Date        Tue Feb 17 06:57:11 1998
Time/Date        Sun Apr  1 14:26:55 2001
Time/Date        Sun May 14 05:26:34 2000
Time/Date        Sun Aug 15 21:56:21 1999
Time/Date        Thu Sep  3 06:23:38 1998
Time/Date        Sun May 14 05:26:34 2000
Time/Date        Thu Mar 18 07:52:30 2010
Time/Date        Thu Apr  8 04:56:35 1999
Time/Date        Sat Nov 13 10:06:31 1999
Time/Date        Wed Nov 29 17:16:50 2000
Time/Date        Tue May  4 18:05:53 1999
Time/Date        Thu Jan  1 04:05:20 1970
Time/Date        Sat Jul 18 05:41:57 1998
Time/Date        Mon Feb 21 13:07:56 2000
Time/Date        Mon Jan 19 04:54:31 1970
Time/Date        Thu Jan  1 00:00:00 1970
Time/Date        Thu Jan  1 00:00:20 1970
Time/Date        Wed Jul 31 19:40:46 2002
Time/Date        Tue Jun  8 15:11:49 1999
Time/Date        Mon Jan 19 04:54:31 1970
Time/Date        Sun Aug 15 21:56:21 1999
Time/Date        Thu Jan  1 00:00:00 1970
Time/Date        Thu Jan  1 00:00:00 1970
Time/Date        Thu Jan  1 00:00:00 1970
Time/Date        Thu Jan  1 00:00:00 1970
Time/Date        Sun Apr 22 19:27:02 2012
Time/Date        Thu Jan  1 00:03:04 1970
Time/Date        Thu Jan  1 00:00:00 1970
Time/Date        Sun Apr 22 19:27:50 2012
Time/Date        Sat Mar  5 02:40:00 2072
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Sun Apr 13 10:40:05 2003
Time/Date        Wed May 15 20:43:46 2013
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Wed May 15 20:43:50 2013
Time/Date        Mon Nov 10 09:40:34 2008
Time/Date        Mon Nov 10 09:40:34 2008
Time/Date        Mon Nov 10 09:40:34 2008
Time/Date        Mon Nov 10 09:40:34 2008
Time/Date        Mon Nov 10 09:40:34 2008
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Mon Nov 10 09:40:34 2008
Time/Date        Wed May 15 20:43:59 2013
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Sat Jan  1 00:00:00 2000
Time/Date        Mon Nov 10 09:40:34 2008
Time/Date        Wed May 15 20:43:48 2013
Time/Date        Wed May 15 20:43:36 2013
Time/Date        Wed May 15 20:43:48 2013

Unlikely to get deterministic output with such.

comment:4 in reply to:  3 Changed 5 years ago by gk

Replying to cypherpunks:

libeay32.dll

Only one?

Yes.

comment:5 Changed 5 years ago by cypherpunks

Then something broken. Deterministically.

comment:6 Changed 5 years ago by cypherpunks

This might be related to the fix for #12391.

Unlikely.

Time/Date		Fri Jan  2 04:44:52 1970

For firefox.exe
It's not zero timestamp and not about 2000/01/01 even. Some random bytes.

comment:7 Changed 5 years ago by cypherpunks

Just noticed, binutils 2.24 used for alpha, and binutils 2.22 for stable Tor browser. By default 2.24 omits timestamp if no option provided, seems like no timestamp field was zeroed.

comment:8 Changed 5 years ago by cypherpunks

Ok, you changed binutils version but keeps patching it

sed 's/= extern_rt_rel_d;/= extern_rt_rel_d;\n  memset (extern_rt_rel_d, 0, PE_IDATA5_SIZE);/' -i ld/pe-dll.c

does it what you want for 2.24?

comment:9 Changed 5 years ago by cypherpunks

but keeps patching it

It's unlikely reason for bug.
Bug in binutils 2.24
Lets test patch for it?

comment:10 Changed 5 years ago by cypherpunks

--- bfd/peXXigen.c.original
+++ bfd/peXXigen.c
@@ -795,6 +795,8 @@
   /* Only use a real timestamp if the option was chosen.  */
   if ((pe_data (abfd)->insert_timestamp))
     H_PUT_32 (abfd, time(0), filehdr_out->f_timdat);
+  else
+    H_PUT_32 (abfd, 0, filehdr_out->f_timdat);
 

comment:11 Changed 5 years ago by gk

Thanks, I'll give it a try shortly as I think I have found a way to reproduce the issue locally...

comment:12 Changed 5 years ago by gk

FWIW: The same issue is visible looking at our nightly builds (here ssleay32.dll is affected in the builds from August 1st and August 3rd e.g.).

comment:13 Changed 5 years ago by gk

Another interesting observation: bug_12753 in my tor-browser-bundle repo which is just building binutils and openssl gives me always (I built that one ten times at least) the same SHA 256 sum for the OpenSSL libs while that is not the case if I build the compiler and all the other utils, too.

comment:14 Changed 5 years ago by cypherpunks

bug_12753 in my tor-browser-bundle repo which is just building binutils and openssl gives me always (I built that one ten times at least) the same SHA 256 sum for the OpenSSL libs while that is not the case if I build the compiler and all the other utils, too.

it probably not using your binutils.

comment:15 Changed 5 years ago by cypherpunks

it probably not using your binutils.

To check it, verify timestamp, binutils 2.24 can't to produce binaries with 2000/01/01 timestamp.

comment:16 in reply to:  9 Changed 5 years ago by gk

Replying to cypherpunks:

Lets test patch for it?

Thanks I tested the patch with multiple builds in a setup that produced quite reliably OpenSSL libs with different timestamps before and they are now zeroed out. This is commit ad5ea06183c12e8ae18274a72cb5bbc353180400. Leaving this ticket open until it passes the alpha bundling test...

comment:17 Changed 5 years ago by gk

Keywords: TorBrowserTeam201408 added; TorBrowserTeam201407 removed

comment:18 Changed 5 years ago by mikeperry

Owner: changed from tbb-team to gk
Status: newassigned

comment:19 Changed 5 years ago by gk

Resolution: fixed
Status: assignedclosed

The alpha is out and we got matching builds, closing this ticket. There are still binaries though, that don't have the timestamp zeroed out, most notably tor.exe. I am wondering whether this has to do with the hint in gitian-firefox.yml:

# XXX: the path to ld is hardcoded in mingw.. 

This might be a limitation of an older mingw-w64 as there is no such problem in gitian-utils when compiling OpenSSL etc. (but compiling mingw-w64 itself with the new linker show the same symptoms). Opened #12926 to investigate this.

Note: See TracTickets for help on using tickets.