Skip to content
Snippets Groups Projects
New look: Off

Windows bundles based on ESR 68 are not built reproducibly

    • Closed (moved) Issue created by Georg Koppen @gk

    As mentioned in comment:20:ticket:28238 we have files that are not built reproducibly with the new mingw-w64-clang toolchain. It affects not all binaries, though, only firefox.exe, libGLESv2.dll, mozglue.dll, plugin-container.exe, plugin-hang-ui.exe, and xul.dll.

    This is still an issue after switching to clang, lld etc. 8.0.0.

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    • Trac
      Trac @tracbot ·

      Trac:
      Parent Ticket: #30322 (moved)
      Child Ticket(s): #31621 (moved)

    • Georg Koppen added GeorgKoppen201909 TorBrowserTeam201909R actualpoints::4 component::applications/tor browser ff68-esr owner::tbb-team parent::30322 points::2 priority::very high resolution::fixed severity::major status::closed tbb-9.0-must-alpha tbb-rbm type::defect labels

      added GeorgKoppen201909 TorBrowserTeam201909R actualpoints::4 component::applications/tor browser ff68-esr owner::tbb-team parent::30322 points::2 priority::very high resolution::fixed severity::major status::closed tbb-9.0-must-alpha tbb-rbm type::defect labels

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      plugin_hang_ui_diff.bz2

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      The diff is surprisingly large (see the example attached for the plugin-hang-ui.exe).

    • C
      cypherpunks @cypherpunks ·

      libtool?

    • boklm
      boklm @boklm ·

      After building the current nightly for windows x86_64 twice on the same machine, I am getting the same bundles. Is it an issue that mostly happen when building on different machines?

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to boklm:

      After building the current nightly for windows x86_64 twice on the same machine, I am getting the same bundles. Is it an issue that mostly happen when building on different machines?

      Yeah, that's part of the mystery: it only happens on different machines it seems.

    • C
      cypherpunks @cypherpunks ·

      "The builds from the same machine were identical, but different on the second machine." boklm said last time it happened...

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Move must-nightly items to must-alpha ones.

      Trac:
      Keywords: tbb-9.0-must-nightly deleted, tbb-9.0-must-alpha added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving must-alpha tickets to September.

      Trac:
      Keywords: TorBrowserTeam201908 deleted, TorBrowserTeam201909 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      First ticket move of mine.

      Trac:
      Keywords: GeorgKoppen201908 deleted, GeorgKoppen201909 added

    • Pili Guerra
      Pili Guerra @pili ·

      Trac:
      Points: N/A to 2

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      It seems at least the table information is not written reproducibly. Taking mozglue.dll as an example:

      Name: Import symbols table RVA: 802489 (0xc3eb9) Size: 380
Name: Export symbols table RVA: 792816 (0xc18f0) Size: 9673
Name: Base relocation table RVA: 937984 (0xe5000) Size: 6172
Name: Debugging information RVA: 880640 (0xd7000) Size: 28
Name: Exception table RVA: 901120 (0xdc000) Size: 26388
Name: Thread local storage table RVA: 746264 (0xb6318) Size: 40
Name: Resource table RVA: 933888 (0xe4000) Size: 1200
Name: Import address table RVA: 804752 (0xc4790) Size: 1880

      vs:

      Name: Import symbols table RVA: 802505 (0xc3ec9) Size: 380
Name: Export symbols table RVA: 792832 (0xc1900) Size: 9673
Name: Base relocation table RVA: 937984 (0xe5000) Size: 6176
Name: Debugging information RVA: 880640 (0xd7000) Size: 28
Name: Exception table RVA: 901120 (0xdc000) Size: 26388
Name: Thread local storage table RVA: 746272 (0xb6320) Size: 40
Name: Resource table RVA: 933888 (0xe4000) Size: 1200
Name: Import address table RVA: 804768 (0xc47a0) Size: 1880
    • Georg Koppen
      Georg Koppen @gk ·
      Author

      I am adding the files I got while using peanalyis for posterity.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      tpo_mozglue_analysis

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      uni_mozglue_analysis

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      mozglue_lld.log

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      mozglue_lld2.log

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Added two log files after compiling with -Wl,-verbose. It seems things start to go wrong when libc++.a is getting consulted.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      mozglue.map

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      mozglue2.map

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      mozglue.map.diff

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Added the map files we get after compiling with -Wl,-Map,map (thanks to Martin Storsjö for for both ideas) and the diff.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      The libc++.a files have the same content fwiw (even though the archive is not assembled reproducibly).

    • C
      cypherpunks @cypherpunks ·

      Broken libtool from system gcc?

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to cypherpunks:

      Broken libtool from system gcc?

      Hm, why do you think that could be relevant here?

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to gk:

      The libc++.a files have the same content fwiw (even though the archive is not assembled reproducibly).

      That's probably the issue here as the linker is taking input from libc++.a as it appears in the .a file (thanks Martin).

      Now, we do two things with libc++.a: 1) we build the archive and 2) we have the merge_libs() function. I guess as a next step we try to pin down where exactly the problem occurs (maybe it's just 2) that is problematic here).

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Okay, things go already south in 1).

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      The different order in libc++.a is already visible in the respecting link command:

      /var/tmp/dist/mingw-w64-clang/bin/llvm-ar qc libc++.a  CMakeFiles/cxx_objects.dir/__/src/shared_mutex.cpp.obj CMakeFiles/cxx_objects.dir/__/src/exception.cpp.obj CMakeFiles/cxx_objects.dir/__/src/system_error.cpp.obj CMakeFiles/cxx_objects.dir/__/src/charconv.cpp.obj CMakeFiles/cxx_objects.dir/__/src/new.cpp.obj CMakeFiles/cxx_objects.dir/__/src/variant.cpp.obj CMakeFiles/cxx_objects.dir/__/src/optional.cpp.obj CMakeFiles/cxx_objects.dir/__/src/bind.cpp.obj CMakeFiles/cxx_objects.dir/__/src/locale.cpp.obj CMakeFiles/cxx_objects.dir/__/src/mutex.cpp.obj CMakeFiles/cxx_objects.dir/__/src/iostream.cpp.obj CMakeFiles/cxx_objects.dir/__/src/stdexcept.cpp.obj CMakeFiles/cxx_objects.dir/__/src/thread.cpp.obj CMakeFiles/cxx_objects.dir/__/src/random.cpp.obj CMakeFiles/cxx_objects.dir/__/src/ios.cpp.obj CMakeFiles/cxx_objects.dir/__/src/typeinfo.cpp.obj CMakeFiles/cxx_objects.dir/__/src/future.cpp.obj CMakeFiles/cxx_objects.dir/__/src/functional.cpp.obj CMakeFiles/cxx_objects.dir/__/src/any.cpp.obj CMakeFiles/cxx_objects.dir/__/src/utility.cpp.obj CMakeFiles/cxx_objects.dir/__/src/valarray.cpp.obj CMakeFiles/cxx_objects.dir/__/src/hash.cpp.obj CMakeFiles/cxx_objects.dir/__/src/regex.cpp.obj CMakeFiles/cxx_objects.dir/__/src/condition_variable.cpp.obj CMakeFiles/cxx_objects.dir/__/src/chrono.cpp.obj CMakeFiles/cxx_objects.dir/__/src/string.cpp.obj CMakeFiles/cxx_objects.dir/__/src/debug.cpp.obj CMakeFiles/cxx_objects.dir/__/src/algorithm.cpp.obj CMakeFiles/cxx_objects.dir/__/src/memory.cpp.obj CMakeFiles/cxx_objects.dir/__/src/vector.cpp.obj CMakeFiles/cxx_objects.dir/__/src/strstream.cpp.obj CMakeFiles/cxx_objects.dir/__/src/support/win32/thread_win32.cpp.obj CMakeFiles/cxx_objects.dir/__/src/support/win32/support.cpp.obj CMakeFiles/cxx_objects.dir/__/src/support/win32/locale_win32.cpp.obj

      vs.

      /var/tmp/dist/mingw-w64-clang/bin/llvm-ar qc libc++.a  CMakeFiles/cxx_objects.dir/__/src/typeinfo.cpp.obj CMakeFiles/cxx_objects.dir/__/src/debug.cpp.obj CMakeFiles/cxx_objects.dir/__/src/thread.cpp.obj CMakeFiles/cxx_objects.dir/__/src/hash.cpp.obj CMakeFiles/cxx_objects.dir/__/src/stdexcept.cpp.obj CMakeFiles/cxx_objects.dir/__/src/algorithm.cpp.obj CMakeFiles/cxx_objects.dir/__/src/valarray.cpp.obj CMakeFiles/cxx_objects.dir/__/src/condition_variable.cpp.obj CMakeFiles/cxx_objects.dir/__/src/locale.cpp.obj CMakeFiles/cxx_objects.dir/__/src/functional.cpp.obj CMakeFiles/cxx_objects.dir/__/src/bind.cpp.obj CMakeFiles/cxx_objects.dir/__/src/shared_mutex.cpp.obj CMakeFiles/cxx_objects.dir/__/src/any.cpp.obj CMakeFiles/cxx_objects.dir/__/src/exception.cpp.obj CMakeFiles/cxx_objects.dir/__/src/chrono.cpp.obj CMakeFiles/cxx_objects.dir/__/src/system_error.cpp.obj CMakeFiles/cxx_objects.dir/__/src/ios.cpp.obj CMakeFiles/cxx_objects.dir/__/src/string.cpp.obj CMakeFiles/cxx_objects.dir/__/src/optional.cpp.obj CMakeFiles/cxx_objects.dir/__/src/new.cpp.obj CMakeFiles/cxx_objects.dir/__/src/charconv.cpp.obj CMakeFiles/cxx_objects.dir/__/src/vector.cpp.obj CMakeFiles/cxx_objects.dir/__/src/regex.cpp.obj CMakeFiles/cxx_objects.dir/__/src/strstream.cpp.obj CMakeFiles/cxx_objects.dir/__/src/memory.cpp.obj CMakeFiles/cxx_objects.dir/__/src/random.cpp.obj CMakeFiles/cxx_objects.dir/__/src/iostream.cpp.obj CMakeFiles/cxx_objects.dir/__/src/utility.cpp.obj CMakeFiles/cxx_objects.dir/__/src/mutex.cpp.obj CMakeFiles/cxx_objects.dir/__/src/future.cpp.obj CMakeFiles/cxx_objects.dir/__/src/variant.cpp.obj CMakeFiles/cxx_objects.dir/__/src/support/win32/support.cpp.obj CMakeFiles/cxx_objects.dir/__/src/support/win32/locale_win32.cpp.obj CMakeFiles/cxx_objects.dir/__/src/support/win32/thread_win32.cpp.obj

      Responsible for that is CMake which saves the command to use in a link.txt file

      cd /var/tmp/build/libcxx/build/lib && /var/tmp/dist/cmake/bin/cmake -E cmake_link_script CMakeFiles/cxx_static.dir/link.txt --verbose=1

      I am not sure yet where sorting that part differently got solved but updating CMake to the latest stable version, 3.15.3, (we are currently using 3.4.3) seems to solve that part of the puzzle. However, other object files are still included non-deterministically after that part. I am not sure yet where they are coming from...

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to gk:

      However, other object files are still included non-deterministically after that part. I am not sure yet where they are coming from...

      Okay, I think I am at the bottom of that rabbit hole. Thre reason for that issue is that libc++.a is getting merged with the contents of libc++abi.a (which, by the way, itself is reproducible at least after switching to CMake's 3.5.13). Now, that merging gets done in libc++'s merge_archives.py (https://github.com/llvm/llvm-project/blob/release/8.x/libcxx/utils/merge_archives.py). The problem here is:

      files = glob.glob(os.path.join(temp_directory_root, '*.o*'))

      From the doc (https://docs.python.org/2/library/glob.html):

      "The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell, although results are returned in arbitrary order."

      So, doing a simple

      files = sorted(glob.glob(os.path.join(temp_directory_root, '*.o*')))

      gets us finally a reproducible libc++.a and a reproducible 64-bit .exe. I am currently testing the 32-bit version as well and will prepare a proper patch for review tomorrow.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      bug_31538_v2 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_31538_v2) has wo patches for review (on top of #31732 (moved)), which should fix the problem. I rebuilt the macOS bundles as well and they are still matching. Doing currently Linux.

      Trac:
      Status: new to needs_review
      Keywords: TorBrowserTeam201909 deleted, TorBrowserTeam201909R added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Linux 64bit builds are still matching. So, we are good, woo!

    • Tom Ritter
      Tom Ritter @tjr ·

      We seem to use a different technique for merging libraries than Martin: https://searchfox.org/mozilla-central/rev/d1e33e3e11f559952d7d80e722d26a6cf5dd80ac/taskcluster/scripts/misc/build-clang-8-mingw.sh#125

      I don't know if ours is reproducible. But Martin can probably upstream that clang change fairly easily. And I should - if not copy your approach - add a comment about it in our build.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to tom:

      We seem to use a different technique for merging libraries than Martin: https://searchfox.org/mozilla-central/rev/d1e33e3e11f559952d7d80e722d26a6cf5dd80ac/taskcluster/scripts/misc/build-clang-8-mingw.sh#125

      No, that's a different piece. It merges libc++ with libunwind which is actually okay. The problematic one, which you hit as well, is within the libc++ build step where libc++ and libc++abi are getting merged into libc++.

      I don't know if ours is reproducible. But Martin can probably upstream that clang change fairly easily. And I should - if not copy your approach - add a comment about it in our build.

      Upstream changed meanwhile, so the patch in question is only usable for clang 8-based toolchains. However, it seems the new changes still have the same issue potentially. I'll get back to Martin with that once this patch lands and we are good on our side.

    • boklm
      boklm @boklm ·

      Replying to gk:

      bug_31538_v2 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_31538_v2) has wo patches for review (on top of #31732 (moved)), which should fix the problem. I rebuilt the macOS bundles as well and they are still matching. Doing currently Linux.

      The patches look good to me. I am currently doing builds to confirm that it is fixing the issue.

    • boklm
      boklm @boklm ·

      Replying to boklm:

      Replying to gk:

      bug_31538_v2 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_31538_v2) has wo patches for review (on top of #31732 (moved)), which should fix the problem. I rebuilt the macOS bundles as well and they are still matching. Doing currently Linux.

      The patches look good to me. I am currently doing builds to confirm that it is fixing the issue.

      I got matching builds from two machines, so this looks good.

      I merged the commits to master as cd111df46540c3727697bac62898a4c141752080 and a7a90d35e55bb8f9886fd9d827b01eba2060c74b.

      Trac:
      Resolution: N/A to fixed
      Status: needs_review to closed

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      Actualpoints: N/A to 4

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Heh. I just realized that commit fc1ec361d39c403915c3c3c2c74468451fade20f in the libcxx repo wants to address the same problem. (But that did not make it into 8.0) I have not tested it whether it really solves the problem and it is larger then just sorting as we do right now. So, we should be good for now with what we have.

    • Trac closed

      closed

    • Trac changed time estimate to 16h

      changed time estimate to 16h

    • Trac added 32h of time spent

      added 32h of time spent

    • Georg Koppen mentioned in issue #31584 (moved)

      mentioned in issue #31584 (moved)

    • cypherpunks mentioned in issue #31618 (moved)

      mentioned in issue #31618 (moved)

    • Georg Koppen mentioned in issue #31621 (moved)

      mentioned in issue #31621 (moved)

    • Trac mentioned in issue #30322 (moved)

      mentioned in issue #30322 (moved)

    • Trac moved to tpo/applications/tor-browser#31538 (closed)

      moved to tpo/applications/tor-browser#31538 (closed)

    • Trac mentioned in issue tpo/applications/tor-browser#31621 (closed)

      mentioned in issue tpo/applications/tor-browser#31621 (closed)

    Please register or sign in to reply