Skip to content
Snippets Groups Projects
New look: Off

Create a mingw-w64-clang project

    • Closed (moved) Issue created by Georg Koppen @gk

    The mingw-w64 project is currently GCC based. We should start creating a parallel project, mingw-w64-clang, that helps us introducing this new toolchain step by step to all the other components we need for the Windows bundles.

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items 0

    • No child items are currently assigned. Use child items to break down this issue into smaller parts.

    Activity

    • Georg Koppen added GeorgKoppen201908 TorBrowserTeam201908R component::applications/tor browser owner::tbb-team parent::28238 priority::very high resolution::fixed severity::normal status::closed tbb-9.0-must-nightly tbb-rbm type::task labels

      added GeorgKoppen201908 TorBrowserTeam201908R component::applications/tor browser owner::tbb-team parent::28238 priority::very high resolution::fixed severity::normal status::closed tbb-9.0-must-nightly tbb-rbm type::task labels

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      Priority: Medium to High

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Note: taskcluster/scripts/misc/build-clang-trunk-mingw.sh is the script we want to look at.

    • Alexander Hansen Færøy
      Alexander Hansen Færøy @ahf ·
      Developer

      Trac:
      Cc: tom, boklm to tom, boklm, ahf

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving my tickets.

      Trac:
      Keywords: GeorgKoppen201812 deleted, GeorgKoppen201901 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Okay, I have something for review. The unsquashed version is available in bug_28716_v2 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_28716_v2), the squashed one in bug_28716_v3_squashed (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_28716_v3_squashed&id=039f08f5cae5303c5b472a5203096799f0fc6c85). See the commit message of the latter for what I did (mainly following https://hg.mozilla.org/releases/mozilla-esr60/file/a578196b1fe0/taskcluster/scripts/misc/build-clang-trunk-mingw.sh plus a bunch of deviations (I hope I got all of the important ones)).

      I am still testing the toolchain(s), but so far the result looks promising: I was able to compile the 64bit Firefox part with a .mozconfig file that resembles the one we ship (I only had to omit the sandbox compilation, as that one is broken right now, which we need to investigate).

      Trac:
      Status: new to needs_review
      Keywords: TorBrowserTeam201812 deleted, TorBrowserTeam201901R added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      I successfully built both our 32bit and 64bit firefox projects using bug_28328 in my tor-browser-build repo.

      ./rbm/rbm build firefox --target nightly --target torbrowser-windows-x86_64 should be the magical command to invoke (for 64bit; replace "x86_64" with "i686" for 32bit).

      I'll look tomorrow into whether this is actually running (I am not even sure if the vanilla esr60 is supposed to be running right now).

      And, yes, this is with Stylo enabled. \o/ (but sandboxing disabled as the compilation breaks :( )

    • Tom Ritter
      Tom Ritter @tjr ·

      This is missing the patches from this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1506450 Those haven't been backported to esr60 yet; but they will be at some point.

      Another thing that's up in the air is https://bugzilla.mozilla.org/show_bug.cgi?id=1515982 and https://bugzilla.mozilla.org/show_bug.cgi?id=1471698#c9 - the fix in 1515982 doesn't seem to work for me. However I don't think this should afect esr60.

      I sent in a build on TC to use this toolchain and see what comes out: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d4b38148421fa501933a5e09f478bc216bbf6905

      I haven't tested to confirm it runs at this point. I'll check tomorrow.

      I did a file-wise comparison:

      Differing Files:

      Only in tc/clang/include: ansidecl.h
Only in tc/clang/include: bfd.h
Only in tc/clang/include: bfdlink.h
Only in tc/clang/include: c++
Only in tc/clang/include: dis-asm.h
Only in tc/clang/include/llvm/BinaryFormat: WasmRelocs
Only in tc/clang/include/llvm/MC: MCAnalysis
Only in tc/clang/include: plugin-api.h
Only in tc/clang/include: symcat.h
Only in tc/clang/lib/clang/8.0.0/include: sanitizer
Only in tc/clang/lib/clang/8.0.0/include: xray
Only in tc/clang/lib/clang/8.0.0/lib: linux
Only in tc/clang/lib/clang/8.0.0: share
Only in tc/clang/lib: gcc
Only in tc/clang/lib: libasan.a
Only in tc/clang/lib: libasan.la
Only in tc/clang/lib: libasan_preinit.o
Only in tc/clang/lib: libasan.so
Only in tc/clang/lib: libasan.so.1
Only in tc/clang/lib: libasan.so.1.0.0
Only in tc/clang/lib: libatomic.a
Only in tc/clang/lib: libatomic.la
Only in tc/clang/lib: libatomic.so
Only in tc/clang/lib: libatomic.so.1
Only in tc/clang/lib: libatomic.so.1.1.0
Only in tc/clang/lib: libc++.a
Only in tc/clang/lib: libc++abi.a
Only in tc/clang/lib: libc++abi.so
Only in tc/clang/lib: libc++abi.so.1
Only in tc/clang/lib: libc++abi.so.1.0
Only in tc/clang/lib: libc++experimental.a
Only in tc/clang/lib: libc++fs.a
Only in tc/clang/lib: libcilkrts.a
Only in tc/clang/lib: libcilkrts.la
Only in tc/clang/lib: libcilkrts.so
Only in tc/clang/lib: libcilkrts.so.5
Only in tc/clang/lib: libcilkrts.so.5.0.0
Only in tc/clang/lib: libcilkrts.spec
Only in tc/clang/lib: libc++.so
Only in tc/clang/lib: libc++.so.1
Only in tc/clang/lib: libc++.so.1.0
Only in tc/clang/lib: libgcc_s.so
Only in tc/clang/lib: libgcc_s.so.1
Only in tc/clang/lib: libgomp.a
Only in tc/clang/lib: libgomp.la
Only in tc/clang/lib: libgomp.so
Only in tc/clang/lib: libgomp.so.1
Only in tc/clang/lib: libgomp.so.1.0.0
Only in tc/clang/lib: libgomp.spec
Only in tc/clang/lib: libitm.a
Only in tc/clang/lib: libitm.la
Only in tc/clang/lib: libitm.so
Only in tc/clang/lib: libitm.so.1
Only in tc/clang/lib: libitm.so.1.0.0
Only in tc/clang/lib: libitm.spec
Only in geko/mingw-w64-clang/lib: libLLVMAMDGPUAsmParser.a
Only in geko/mingw-w64-clang/lib: libLLVMAMDGPUAsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMAMDGPUCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMAMDGPUDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMAMDGPUDisassembler.a
Only in geko/mingw-w64-clang/lib: libLLVMAMDGPUInfo.a
Only in geko/mingw-w64-clang/lib: libLLVMAMDGPUUtils.a
Only in geko/mingw-w64-clang/lib: libLLVMBPFAsmParser.a
Only in geko/mingw-w64-clang/lib: libLLVMBPFAsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMBPFCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMBPFDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMBPFDisassembler.a
Only in geko/mingw-w64-clang/lib: libLLVMBPFInfo.a
Only in geko/mingw-w64-clang/lib: libLLVMHexagonAsmParser.a
Only in geko/mingw-w64-clang/lib: libLLVMHexagonCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMHexagonDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMHexagonDisassembler.a
Only in geko/mingw-w64-clang/lib: libLLVMHexagonInfo.a
Only in geko/mingw-w64-clang/lib: libLLVMLanaiAsmParser.a
Only in geko/mingw-w64-clang/lib: libLLVMLanaiAsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMLanaiCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMLanaiDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMLanaiDisassembler.a
Only in geko/mingw-w64-clang/lib: libLLVMLanaiInfo.a
Only in geko/mingw-w64-clang/lib: libLLVMMipsAsmParser.a
Only in geko/mingw-w64-clang/lib: libLLVMMipsAsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMMipsCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMMipsDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMMipsDisassembler.a
Only in geko/mingw-w64-clang/lib: libLLVMMipsInfo.a
Only in geko/mingw-w64-clang/lib: libLLVMMSP430AsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMMSP430CodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMMSP430Desc.a
Only in geko/mingw-w64-clang/lib: libLLVMMSP430Info.a
Only in geko/mingw-w64-clang/lib: libLLVMNVPTXAsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMNVPTXCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMNVPTXDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMNVPTXInfo.a
Only in geko/mingw-w64-clang/lib: libLLVMPowerPCAsmParser.a
Only in geko/mingw-w64-clang/lib: libLLVMPowerPCAsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMPowerPCCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMPowerPCDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMPowerPCDisassembler.a
Only in geko/mingw-w64-clang/lib: libLLVMPowerPCInfo.a
Only in geko/mingw-w64-clang/lib: libLLVMSparcAsmParser.a
Only in geko/mingw-w64-clang/lib: libLLVMSparcAsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMSparcCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMSparcDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMSparcDisassembler.a
Only in geko/mingw-w64-clang/lib: libLLVMSparcInfo.a
Only in geko/mingw-w64-clang/lib: libLLVMSystemZAsmParser.a
Only in geko/mingw-w64-clang/lib: libLLVMSystemZAsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMSystemZCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMSystemZDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMSystemZDisassembler.a
Only in geko/mingw-w64-clang/lib: libLLVMSystemZInfo.a
Only in geko/mingw-w64-clang/lib: libLLVMXCoreAsmPrinter.a
Only in geko/mingw-w64-clang/lib: libLLVMXCoreCodeGen.a
Only in geko/mingw-w64-clang/lib: libLLVMXCoreDesc.a
Only in geko/mingw-w64-clang/lib: libLLVMXCoreDisassembler.a
Only in geko/mingw-w64-clang/lib: libLLVMXCoreInfo.a
Only in tc/clang/lib: liblsan.a
Only in tc/clang/lib: liblsan.la
Only in tc/clang/lib: liblsan.so
Only in tc/clang/lib: liblsan.so.0
Only in tc/clang/lib: liblsan.so.0.0.0
Only in tc/clang/lib: libquadmath.a
Only in tc/clang/lib: libquadmath.la
Only in tc/clang/lib: libquadmath.so
Only in tc/clang/lib: libquadmath.so.0
Only in tc/clang/lib: libquadmath.so.0.0.0
Only in tc/clang/lib: libsanitizer.spec
Only in tc/clang/lib: libssp.a
Only in tc/clang/lib: libssp.la
Only in tc/clang/lib: libssp_nonshared.a
Only in tc/clang/lib: libssp_nonshared.la
Only in tc/clang/lib: libssp.so
Only in tc/clang/lib: libssp.so.0
Only in tc/clang/lib: libssp.so.0.0.0
Only in tc/clang/lib: libsupc++.a
Only in tc/clang/lib: libsupc++.la
Only in tc/clang/lib: libtsan.a
Only in tc/clang/lib: libtsan.la
Only in tc/clang/lib: libtsan.so
Only in tc/clang/lib: libtsan.so.0
Only in tc/clang/lib: libtsan.so.0.0.0
Only in tc/clang/lib: libubsan.a
Only in tc/clang/lib: libubsan.la
Only in tc/clang/lib: libubsan.so
Only in tc/clang/lib: libubsan.so.0
Only in tc/clang/lib: libubsan.so.0.0.0
Only in tc/clang/lib: libvtv.a
Only in tc/clang/lib: libvtv.la
Only in tc/clang/lib: libvtv.so
Only in tc/clang/lib: libvtv.so.0
Only in tc/clang/lib: libvtv.so.0.0.0
Only in geko/mingw-w64-clang/x86_64-w64-mingw32/include: expandedresources.h
Only in tc/clang/x86_64-w64-mingw32/include: _mingw_print_pop.h
Only in tc/clang/x86_64-w64-mingw32/include: _mingw_print_push.h
Only in geko/mingw-w64-clang/x86_64-w64-mingw32/lib: libntdllcrt.a
    • Tom Ritter
      Tom Ritter @tjr ·

      Replying to gk:

      I'll look tomorrow into whether this is actually running (I am not even sure if the vanilla esr60 is supposed to be running right now).

      A note; my testing indicates you will need the patch from https://bugzilla.mozilla.org/show_bug.cgi?id=1506450 and https://bugzilla.mozilla.org/show_bug.cgi?id=1481633 to get esr60 to run.

      Testing x64 opt: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bce4d090bb8ed8a92d887cfb0121b07553084508 does not run. This set of patches backports a bunch of build changes that refactor some stuff. 1481633 is included in it; but that's the only change to FF I think.

      https://treeherder.mozilla.org/#/jobs?repo=try&revision=fbb3dab9fe5262f5a9394a6952706894fe15c331 does run. This is the above set of patches plus the single patch listed.

      This is just testing the esr60 branch in TC; not using the toolchain you built...

    • Tom Ritter
      Tom Ritter @tjr ·

      More progress:

      https://treeherder.mozilla.org/#/jobs?repo=try&revision=6f0a54fc3f08a075308a9bf3aa34edca090db206 (which builds on top of fbb3dab9fe5262f5a9394a6952706894fe15c331) enables the sandbox and runs

      https://treeherder.mozilla.org/#/jobs?repo=try&revision=d90df493b2ecb5267009789a47559ebe534d9d87 (which builds on top of 6f0a54fc3f08a075308a9bf3aa34edca090db206) adds pdbs and runs and the pdbs seem to work.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to tom:

      This is missing the patches from this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1506450 Those haven't been backported to esr60 yet; but they will be at some point.

      Yes, that was kind of intentional at this point for making sure we stay a) closer to esr60 as it is right now and, more importantly, b) I had issues to get the wrapper compiled on 32bit due to some weird header inclusion issue. So, I decided to leave that at the moment.

      [snip]

      I did a file-wise comparison:

      Differing Files:

      Yes, I am not surprised given that we build clang differently, and don't copy gcc stuff over. I think at some point we want to match Mozilla closer, given the grown importance of clang for Firefox builds, but not in that bug. I opened #29041 (moved) for that.

      {{{ Only in geko/mingw-w64-clang/x86_64-w64-mingw32/include: expandedresources.h Only in tc/clang/x86_64-w64-mingw32/include: _mingw_print_pop.h Only in tc/clang/x86_64-w64-mingw32/include: _mingw_print_push.h Only in geko/mingw-w64-clang/x86_64-w64-mingw32/lib: libntdllcrt.a }}}

      Those four items look interesing, though...

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to tom:

      Replying to gk:

      I'll look tomorrow into whether this is actually running (I am not even sure if the vanilla esr60 is supposed to be running right now).

      A note; my testing indicates you will need the patch from https://bugzilla.mozilla.org/show_bug.cgi?id=1506450 and https://bugzilla.mozilla.org/show_bug.cgi?id=1481633 to get esr60 to run.

      Why would we need 1481633 fixed on esr60 if the reason it came up (1471132) did not land on esr60? Moreover, from skimming that bug it seems to me that's an issue we should hit during compilation, if at all, which I did not.

      Testing x64 opt: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bce4d090bb8ed8a92d887cfb0121b07553084508 does not run. This set of patches backports a bunch of build changes that refactor some stuff. 1481633 is included in it; but that's the only change to FF I think.

      https://treeherder.mozilla.org/#/jobs?repo=try&revision=fbb3dab9fe5262f5a9394a6952706894fe15c331 does run. This is the above set of patches plus the single patch listed.

      Okay, I guess that's a strong indication of including the fix for bug 1506450 then. Marking this ticket as needs_revision

      Trac:
      Status: needs_review to needs_revision
      Keywords: TorBrowserTeam201901R deleted, TorBrowserTeam201901 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to tom:

      More progress:

      https://treeherder.mozilla.org/#/jobs?repo=try&revision=6f0a54fc3f08a075308a9bf3aa34edca090db206 (which builds on top of fbb3dab9fe5262f5a9394a6952706894fe15c331) enables the sandbox and runs

      https://treeherder.mozilla.org/#/jobs?repo=try&revision=d90df493b2ecb5267009789a47559ebe534d9d87 (which builds on top of 6f0a54fc3f08a075308a9bf3aa34edca090db206) adds pdbs and runs and the pdbs seem to work.

      Yay!

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to gk:

      Replying to tom:

      Replying to gk: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fbb3dab9fe5262f5a9394a6952706894fe15c331 does run. This is the above set of patches plus the single patch listed.

      Okay, I guess that's a strong indication of including the fix for bug 1506450 then. Marking this ticket as needs_revision

      I've updated bug_28716_v2 to add the changes coming with bug 1506450 and rebased bug_28238_v2 on top of it (after adding the mozconfig changes due to the fix for 1506450). 64bit is still compiling fine, I am looking into the header path issue for 32bit.

      FWIW: I have not patched our Firefox related build instructions so that they bundle the urct dlls instead of msvcr100 yet. That will happen in #28238 (moved).

    • Tom Ritter
      Tom Ritter @tjr ·

      Replying to gk:

      I've updated bug_28716_v2 to add the changes coming with bug 1506450 and rebased bug_28238_v2 on top of it (after adding the mozconfig changes due to the fix for 1506450). 64bit is still compiling fine, I am looking into the header path issue for 32bit.

      FWIW: I have not patched our Firefox related build instructions so that they bundle the ucrt dlls instead of msvcr100 yet. That will happen in #28238 (moved).

      I built https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7ca3a0df8ec659bbe75322bea4ab296a91a1c50 with the toolchain from this comment.

      It builds on top of d90df493b2ecb5267009789a47559ebe534d9d87 from above and includes these two patches: https://hg.mozilla.org/try/rev/9857214ebf6b (use your toolchain) https://hg.mozilla.org/try/rev/00a398b1fa4d4c79cf5651635bed03049378b0b2 (disable check_nsmodules because your toolchain doesn't contain a llvm-readelf -> x86_64-blah-readelf symlink which is needed for this)

      I tested the x64 opt build; and it runs and loaded a few websites.

    • Tom Ritter
      Tom Ritter @tjr ·

      Replying to gk:

      Why would we need 1481633 fixed on esr60 if the reason it came up (1471132) did not land on esr60? Moreover, from skimming that bug it seems to me that's an issue we should hit during compilation, if at all, which I did not.

      Hrm. nsModule compilation might have been broken before 1471132 came along; and we just didn't have a mingw-clang build consistently working then.

      A few different things can go wrong with nsModules: at compile time you could have missing symbols. Or at runtime, the resulting symbols in the binary might not be packed or ordered correctly. We do some ugly stuff were we basically make an array in the resulting binary, and then iterate through it like it was an array. If the layout, padding, or order is wrong, the code assumptions break and you get a runtime problem.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to tom:

      Replying to gk:

      I've updated bug_28716_v2 to add the changes coming with bug 1506450 and rebased bug_28238_v2 on top of it (after adding the mozconfig changes due to the fix for 1506450). 64bit is still compiling fine, I am looking into the header path issue for 32bit.

      FWIW: I have not patched our Firefox related build instructions so that they bundle the urct dlls instead of msvcr100 yet. That will happen in #28238 (moved).

      I built https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7ca3a0df8ec659bbe75322bea4ab296a91a1c50 with the toolchain from this comment.

      It builds on top of d90df493b2ecb5267009789a47559ebe534d9d87 from above and includes these two patches: https://hg.mozilla.org/try/rev/9857214ebf6b (use your toolchain) https://hg.mozilla.org/try/rev/00a398b1fa4d4c79cf5651635bed03049378b0b2 (disable check_nsmodules because your toolchain doesn't contain a llvm-readelf -> x86_64-blah-readelf symlink which is needed for this)

      I tested the x64 opt build; and it runs and loaded a few websites.

      That's on Windows 10, right? I tested the artifact (https://queue.taskcluster.net/v1/task/X5K6N8O8SMqb9RarbIrp2w/runs/0/artifacts/public/build/target.zip) on Windows 7 and it seems to crash early on. I have not looked closer why but I noticed that firefox.exe etc. is still linked against msvcrt.dll which is not working for Windows 7. So, we need to teach the toolchain again how to link against a non-default runtime lib (ucrt this time) which has always been a fun exercise...

    • Trac
      Trac @tracbot ·

      Replying to gk:

      So, we need to teach the toolchain again how to link against a non-default runtime lib (ucrt this time) which has always been a fun exercise... It is a default c runtime lib in msvc...

      (tom: dxr seems to go nuts https://dxr.mozilla.org/mozilla-central/source/old-configure#3526)

      Trac:
      Username: watt

    • Tom Ritter
      Tom Ritter @tjr ·

      A few more builds (all successful):

      esr60 rebased to tip with the build refactoring: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0659d6e265f3b624ad6fbac0c9cd7ce246094596

      I don't know what is up with the runtime; I asked Jacek and he said "it's in mingw-w64-crt configure and it seems that the script passes --with-default-msvcrt=ucrt, which should do the right thing". (And yes, I am on Windows 10.) I think I have a Windows 7 VM; I will have to check when I have time...

    • Trac
      Trac @tracbot ·

      Replying to tom: Shouldn't clang be built with mingw-w64 headers (and not before)?

      default_win32_winnt=0x601 - for better threading support.

      (What's up with dxr?)

      Trac:
      Username: watt

    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    Please register or sign in to reply