Opened 9 months ago

Last modified 2 days ago

#28238 new defect

Use mingw-w64/clang toolchain to build Firefox

Reported by: gk Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, GeorgKoppen201907, TorBrowserTeam201907, tbb-9.0-must-nightly
Cc: tom, boklm Actual Points:
Parent ID: #30322 Points:
Reviewer: Sponsor:

Description

We want to use the clang-based mingw-w64 toolchain to build Firefox to be finally able to enabled Stylo on Windows as well. This is the ticket tracking the implementation work

Child Tickets

TicketStatusOwnerSummaryComponent
#28716needs_revisiontbb-teamCreate a mingw-w64-clang projectApplications/Tor Browser
#29013newtbb-teamProvide stack smashing protection for mingw-clang buildsApplications/Tor Browser
#29307closedtbb-teamUse Debian Stretch for cross-compiling our Windows buildsApplications/Tor Browser

Attachments (1)

fxexe_diff.bz2 (1.7 MB) - added by gk 6 months ago.
diff firefox.exe

Download all attachments as: .zip

Change History (38)

comment:1 Changed 9 months ago by gk

We likely start small and just use the toolchain to build the Firefox part which means we'd have two mingw-w64 toolchains for the time being (we had this back then in the old days for macOS, too). I guess other parts of the whole bundle might need quite some amount of work to getting built with that new toolchain and we want to move fast here aiming for Tor Browser 8.5.

However, if it turns out I am wrong and it is indeed easy to use mingw-w64/clang for everything, fine with me.

comment:2 Changed 8 months ago by gk

Keywords: TorBrowserTeam201812 GeorgKoppen201812 added

comment:3 Changed 6 months ago by gk

Keywords: GeorgKoppen201901 added; GeorgKoppen201812 removed

Moving my tickets.

comment:4 Changed 6 months ago by gk

Testing bug_28238 shows that we have some reproducibility issues with mingw-w64/clang. It seems we have to deal at least with timestamp issues in the COFF header. Comparing e.g. firefox.exe from two different runs shows:

--- /dev/fd/63	2019-01-10 09:38:08.159230047 +0100
+++ /dev/fd/62	2019-01-10 09:38:08.159230047 +0100
@@ -6,7 +6,7 @@
 00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f  is program canno
 00000060: 7420 6265 2072 756e 2069 6e20 444f 5320  t be run in DOS 
 00000070: 6d6f 6465 2e24 0000 5045 0000 4c01 0600  mode.$..PE..L...
-00000080: 3b1c 365c 003c 0000 a201 0000 e000 2201  ;.6\.<........".
+00000080: 45ff 365c 003c 0000 a201 0000 e000 2201  E.6\.<........".
 00000090: 0b01 0e00 001e 0000 001a 0000 0000 0000  ................
 000000a0: c013 0000 0010 0000 0000 0000 0000 4000  ..............@.
 000000b0: 0010 0000 0002 0000 0600 0000 0000 0000  ................

comment:5 Changed 6 months ago by gk

Keywords: TorBrowserTeam201901 added; TorBrowserTeam201812 removed

Moving tickets to Jan 2019.

comment:6 in reply to:  4 ; Changed 6 months ago by gk

Replying to gk:

Testing bug_28238 shows that we have some reproducibility issues with mingw-w64/clang. It seems we have to deal at least with timestamp issues in the COFF header. Comparing e.g. firefox.exe from two different runs shows:

--- /dev/fd/63	2019-01-10 09:38:08.159230047 +0100
+++ /dev/fd/62	2019-01-10 09:38:08.159230047 +0100
@@ -6,7 +6,7 @@
 00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f  is program canno
 00000060: 7420 6265 2072 756e 2069 6e20 444f 5320  t be run in DOS 
 00000070: 6d6f 6465 2e24 0000 5045 0000 4c01 0600  mode.$..PE..L...
-00000080: 3b1c 365c 003c 0000 a201 0000 e000 2201  ;.6\.<........".
+00000080: 45ff 365c 003c 0000 a201 0000 e000 2201  E.6\.<........".
 00000090: 0b01 0e00 001e 0000 001a 0000 0000 0000  ................
 000000a0: c013 0000 0010 0000 0000 0000 0000 4000  ..............@.
 000000b0: 0010 0000 0002 0000 0600 0000 0000 0000  ................

FWIW: That's still an unsolved issue after bumping the LLVM revision to r348363.

comment:8 in reply to:  7 Changed 6 months ago by gk

Replying to ld:

https://bugs.chromium.org/p/chromium/issues/detail?id=533657
https://bugs.llvm.org/show_bug.cgi?id=24740#c3

That's about building _clang_ deterministic, yes. Thus, this is more suited for #29041.

comment:10 Changed 6 months ago by gk

https://reviews.llvm.org/rL332613 seems like the relevant revision for the timestamp issue.

comment:11 Changed 6 months ago by ld

  if (Args.hasArg(OPT_no_insert_timestamp))
    Add("-timestamp:0");

comment:12 Changed 6 months ago by gk

So, I have been fighting today with building the whole Firefox related part (including fxc2) without the old mingw-w64/gcc-based toolchain. fxc2 does not want and I don't know why yet. It fails to run with things like

 0:03.01 err:module:import_dll Library api-ms-win-crt-convert-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.01 err:module:import_dll Library api-ms-win-crt-heap-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.01 err:module:import_dll Library api-ms-win-crt-private-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.01 err:module:import_dll Library api-ms-win-crt-runtime-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.01 err:module:import_dll Library api-ms-win-crt-stdio-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.02 err:module:import_dll Library api-ms-win-crt-string-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.02 err:module:import_dll Library api-ms-win-crt-environment-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.02 err:module:import_dll Library api-ms-win-crt-math-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.02 err:module:import_dll Library api-ms-win-crt-time-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.03 err:module:import_dll Library api-ms-win-crt-locale-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.03 err:module:LdrInitializeThunk Main exe initialization for L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe" failed, status c0000135

I need to look closer at the way this is built on Mozilla infra tomorrow where this seems to be working.

comment:13 in reply to:  12 Changed 6 months ago by fxc2

Replying to gk:

So, I have been fighting today with building the whole Firefox related part (including fxc2) without the old mingw-w64/gcc-based toolchain.

No, no, don't fight with it! The new tool chain is much more sane. And it will make you happy.

fxc2 does not want and I don't know why yet. It fails to run with things like

 0:03.01 err:module:import_dll Library api-ms-win-crt-convert-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.01 err:module:import_dll Library api-ms-win-crt-heap-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.01 err:module:import_dll Library api-ms-win-crt-private-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.01 err:module:import_dll Library api-ms-win-crt-runtime-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.01 err:module:import_dll Library api-ms-win-crt-stdio-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.02 err:module:import_dll Library api-ms-win-crt-string-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.02 err:module:import_dll Library api-ms-win-crt-environment-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.02 err:module:import_dll Library api-ms-win-crt-math-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.02 err:module:import_dll Library api-ms-win-crt-time-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.03 err:module:import_dll Library api-ms-win-crt-locale-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found
 0:03.03 err:module:LdrInitializeThunk Main exe initialization for L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe" failed, status c0000135

Do you have wine with ucrt? FWIW, wine 4.0 is out.

I need to look closer at the way this is built on Mozilla infra tomorrow where this seems to be working.

They use https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/scripts/misc/build-wine.sh

comment:14 Changed 6 months ago by gk

Okay, I am close with this ticket. Here is a test bundle which compiles fxc2 and firefox with mingw-w64/clang while mingw-w64/gcc is used for the remaining components. I'd be interested in whether that explodes on anyone's machine:

https://people.torproject.org/~gk/testbuilds/torbrowser-install-win64-tbb-nightly_28238_1_en-US.exe
https://people.torproject.org/~gk/testbuilds/torbrowser-install-win64-tbb-nightly_28238_1_en-US.exe.asc

comment:16 Changed 6 months ago by cypherpunks

Parent ID: #29318

Whoa! Cypherpunks operational!

Of course, that explodes, but whether I'm the only one who is interested in it?

comment:17 Changed 6 months ago by cypherpunks

Parent ID: #29318

oops

comment:18 in reply to:  16 ; Changed 6 months ago by gk

Replying to cypherpunks:

Whoa! Cypherpunks operational!

Of course, that explodes, but whether I'm the only one who is interested in it?

Care to give more context? Which of the bundles did you test? On which Windows version? What happened? What error did you get, if any?

comment:19 Changed 6 months ago by gk

Keywords: GeorgKoppen201902 added; GeorgKoppen201901 removed

Moving my tickets to Feb

comment:20 in reply to:  6 Changed 6 months ago by gk

Replying to gk:

Replying to gk:

Testing bug_28238 shows that we have some reproducibility issues with mingw-w64/clang. It seems we have to deal at least with timestamp issues in the COFF header. Comparing e.g. firefox.exe from two different runs shows:

--- /dev/fd/63	2019-01-10 09:38:08.159230047 +0100
+++ /dev/fd/62	2019-01-10 09:38:08.159230047 +0100
@@ -6,7 +6,7 @@
 00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f  is program canno
 00000060: 7420 6265 2072 756e 2069 6e20 444f 5320  t be run in DOS 
 00000070: 6d6f 6465 2e24 0000 5045 0000 4c01 0600  mode.$..PE..L...
-00000080: 3b1c 365c 003c 0000 a201 0000 e000 2201  ;.6\.<........".
+00000080: 45ff 365c 003c 0000 a201 0000 e000 2201  E.6\.<........".
 00000090: 0b01 0e00 001e 0000 001a 0000 0000 0000  ................
 000000a0: c013 0000 0010 0000 0000 0000 0000 4000  ..............@.
 000000b0: 0010 0000 0002 0000 0600 0000 0000 0000  ................

FWIW: That's still an unsolved issue after bumping the LLVM revision to r348363.

That is solved and in the process of being upstreamed. I now get identical builds on one machine. However, compared to the other one I tested they significantly differ. Not all .dlls/.exe files are affected, though, just firefox.exe, libGLESv2.dll, mozglue.dll, pingsender.exe, plugin-container.exe, plugin-hang-ui.exe, and xul.dll.

Changed 6 months ago by gk

Attachment: fxexe_diff.bz2 added

diff firefox.exe

comment:21 in reply to:  18 Changed 6 months ago by cypherpunks

Replying to gk:

Replying to cypherpunks:

Whoa! Cypherpunks operational!

Of course, that explodes, but whether I'm the only one who is interested in it?

Care to give more context? Which of the bundles did you test? On which Windows version? What happened? What error did you get, if any?

No answers to my questions - no answers to your questions. But don't worry, it's a blocker.
There are over 1M daily users on Windows and no one here. How so? Are they fakes? Or don't care? And you, do you care about security on Windows as a primary target, or Windows is incompatible with security for you?
(5 vs 5 questions)

comment:22 Changed 6 months ago by tom

x64 build: On first run; tor failed; noting:

2/5/19, 16:46:34.346 [NOTICE] Bootstrapped 40% (loading_keys): Loading authority key certs
2/5/19, 16:46:34.952 [WARN] ISO time "2019-02-05 16:00:00\r" was unparseable
2/5/19, 16:46:34.952 [WARN] Unable to parse networkstatus consensus

On second run (I ran firefox.exe directly but I don't think that mattered) it worked and I was able to load a website, youtube, play a video, and hit an onion site.

x86 build: same problem with the authoritity certificates. On second run (this time I used the Start Tor Browser shortcut) it worked; again with a website, onion, and youtube. (I actually exited using IPv6 too!) I confirmed ASLR on the x86 build.

All on Windows 10.

comment:23 in reply to:  22 Changed 6 months ago by boklm

Replying to tom:

x64 build: On first run; tor failed; noting:

2/5/19, 16:46:34.346 [NOTICE] Bootstrapped 40% (loading_keys): Loading authority key certs
2/5/19, 16:46:34.952 [WARN] ISO time "2019-02-05 16:00:00\r" was unparseable
2/5/19, 16:46:34.952 [WARN] Unable to parse networkstatus consensus

On second run (I ran firefox.exe directly but I don't think that mattered) it worked and I was able to load a website, youtube, play a video, and hit an onion site.

x86 build: same problem with the authoritity certificates. On second run (this time I used the Start Tor Browser shortcut) it worked; again with a website, onion, and youtube. (I actually exited using IPv6 too!) I confirmed ASLR on the x86 build.

All on Windows 10.

This looks like #28614.

comment:24 Changed 5 months ago by gk

Keywords: TorBrowserTeam201902 added; TorBrowserTeam201901 removed

Moving tickets to February.

comment:25 Changed 5 months ago by gk

Keywords: TorBrowserTeam201903 added; TorBrowserTeam201902 removed

Moving my tickets to March.

comment:26 Changed 5 months ago by gk

Keywords: GeorgKoppen201903 added; GeorgKoppen201902 removed

Now for my keyword.

comment:27 Changed 4 months ago by gk

Keywords: TorBrowserTeam201904 added; TorBrowserTeam201903 removed

Moving tickets to April.

comment:28 Changed 3 months ago by gk

Keywords: GeorgKoppen201904 added; GeorgKoppen201903 removed

Moving my tickets for April

comment:29 Changed 3 months ago by gk

Parent ID: #29318#30321

comment:30 Changed 3 months ago by gk

Parent ID: #30321#30322

comment:31 Changed 2 months ago by gk

Keywords: TorBrowserTeam201905 added; TorBrowserTeam201904 removed

Moving tickets to May

comment:32 Changed 2 months ago by gk

Keywords: GeorgKoppen201905 added; GeorgKoppen201904 removed

Move my tickets.

comment:33 Changed 6 weeks ago by gk

Keywords: TorBrowserTeam201906 added; TorBrowserTeam201905 removed

Moving tickets to June

comment:34 Changed 6 weeks ago by gk

Keywords: GeorgKoppen201906 added; GeorgKoppen201905 removed

Moving my tickets to June

comment:35 Changed 3 weeks ago by gk

Keywords: GeorgKoppen201907 added; GeorgKoppen201906 removed

Moving my tickets to July.

comment:36 Changed 12 days ago by gk

Keywords: TorBrowserTeam201907 added; TorBrowserTeam201906 removed

Moving tickets to July

comment:37 Changed 2 days ago by gk

Keywords: tbb-9.0-must-nightly added

Starting with 9.0 keywords

Note: See TracTickets for help on using tickets.