Build a Windows 64 toolchain based on mingw-w64

In tor-browser-build.git, we need to add support for building a mingw-w64 toolchain for creating Windows 64bit binaries.

added TorBrowserTeam201711R component::applications/tor browser owner::tbb-team parent::20636 priority::medium resolution::fixed severity::normal status::closed tbb-rbm type::task labels

This is done with commit 80026a6348856f161ef2ea70d06b0c93939e68b5 in branch bug_23228_v1: https://oniongit.eu/boklm/tor-browser-build/commit/80026a6348856f161ef2ea70d06b0c93939e68b5

Trac:
Status: new to needs_review
Keywords: N/A deleted, TorBrowserTeam201708R added

A build of mingw-w64 for Windows 64 can be done with:

$ ./rbm/rbm build mingw-w64 --target torbrowser-windows-x86_64 --target alpha

Moving reviews to September.

Trac:
Keywords: TorBrowserTeam201708R deleted, TorBrowserTeam201709R added

An updated version of the patch is in commit b9904226f49a7c5a15c31e8466d7af57f8239538 from branch bug_20636_v5: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_20636_v5&id=b9904226f49a7c5a15c31e8466d7af57f8239538

In this new version of the patch we remove the projects/mingw-w64/i686-w64-mingw32-* files (which are now created in projects/mingw-w64/build).

Replying to boklm:

An updated version of the patch is in commit b9904226f49a7c5a15c31e8466d7af57f8239538 from branch bug_20636_v5: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_20636_v5&id=b9904226f49a7c5a15c31e8466d7af57f8239538

In this new version of the patch we remove the projects/mingw-w64/i686-w64-mingw32-* files (which are now created in projects/mingw-w64/build).

I think this needs revision as we need the relocation patch for 64-bit builds as well. Having the DLL characteristics showing proper output as the script we use does is necessary but not sufficient. We need an relocation table as well. Otherwise we land in the same situation as the bitcoin people (https://github.com/bitcoin/bitcoin/issues/8248) (I guess we can point them to our stuff to fix their problem?)). (comment:description:ticket:10505 summarizes the requirements, too)

There is an old message on the mingw-w64 mailing list that summarizes all the pitfalls and requirements good: https://sourceforge.net/p/mingw-w64/mailman/message/31034877/

skruffy wrote the patch for Tor Browser on Windows 32 platforms (PE) but we need to fix it for 64-bit (PE+ or probably PEP in binutils lingo). I think the way to go is to look at the pep-dll.c to understand what it does and add the missing things (I guess we could think about patching pep.em as well to rename pe_dll_enable_reloc_section to pep_dll_enable_reloc_section to make things more straightforward, but maybe not. I have not looked that close).

Something for a different ticket I guess: Given that the script that checks the headers for ASLR is not enough to be sure we have ASLR enabled we should think about a better test.

Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201709R deleted, TorBrowserTeam201709 added

Oh, I forgot to mention: https://sourceware.org/bugzilla/show_bug.cgi?id=19011 is a good read as well. The ffmpeg people seem to use a different workaround to avoid the crashing (mentioned in the mingw-w64 mailing list message above) when specifying -pie (see: https://github.com/TheRyuu/FFmpeg/commit/91b668acd6decec0a6f8d20bf56e2644f96adcb9).

I opened #23458 (moved) to check if we could use the FFmpeg workaround.

Just looking at the patch skruffy wrote: what about the attached patch trying to take the 64bit case into account? The compile issue is gone. Not sure if ASLR is working, though. If there are some binaries with and without the patch I could test I guess. Maybe tor binaries would already be enough.

Trac:
0001-64bit-fixups.patch

This fixup patch is adding your binutils patch to the build: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_20636_v6&id=8eba1f29a8cd4f48278b1ad84052a5da4a9225f7

I did a build of tor expert-bundle with and without this patch.

Without the patch: https://people.torproject.org/~boklm/tmp/bug_23228/no-patch/tor-0.3.1.5-alpha-windows-x86_64-b5dec4/tor-win32-0.3.1.5-alpha.zip.asc https://people.torproject.org/~boklm/tmp/bug_23228/no-patch/tor-0.3.1.5-alpha-windows-x86_64-b5dec4/tor-win32-0.3.1.5-alpha.zip

With the patch: https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/tor-0.3.1.5-alpha-windows-x86_64-4974f5/tor-win32-0.3.1.5-alpha.zip.asc https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/tor-0.3.1.5-alpha-windows-x86_64-4974f5/tor-win32-0.3.1.5-alpha.zip

I will do the same with a firefox build.

Ah, I just noticed that I forgot re-adding the --enable-reloc-section flag to the x86_64 builds, so the builds with the patch are probably wrong. I will redo them.

Replying to boklm:

Ah, I just noticed that I forgot re-adding the --enable-reloc-section flag to the x86_64 builds, so the builds with the patch are probably wrong. I will redo them.

I can confirm that they match the no-patch-behavior. :)

The patch adding the flag back: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_20636_v6&id=90c730d131d3e003ff626ed613f37c43ada0cb7e

And the build with the patch and the flag: https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/tor-0.3.1.5-alpha-windows-x86_64-3eb3d4/tor-win32-0.3.1.5-alpha.zip.asc https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/tor-0.3.1.5-alpha-windows-x86_64-3eb3d4/tor-win32-0.3.1.5-alpha.zip

Firefox build without the patch: https://people.torproject.org/~boklm/tmp/bug_23228/no-patch/firefox-716975763878-windows-x86_64-394d3c/tor-browser.tar.gz.asc https://people.torproject.org/~boklm/tmp/bug_23228/no-patch/firefox-716975763878-windows-x86_64-394d3c/tor-browser.tar.gz https://people.torproject.org/~boklm/tmp/bug_23228/no-patch/firefox-716975763878-windows-x86_64-394d3c/mar-tools-win64.zip.asc https://people.torproject.org/~boklm/tmp/bug_23228/no-patch/firefox-716975763878-windows-x86_64-394d3c/mar-tools-win64.zip

Firefox build with the patch (and flag): https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/firefox-716975763878-windows-x86_64-8bd72e/tor-browser.tar.gz.asc https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/firefox-716975763878-windows-x86_64-8bd72e/tor-browser.tar.gz https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/firefox-716975763878-windows-x86_64-8bd72e/mar-tools-win64.zip.asc https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/firefox-716975763878-windows-x86_64-8bd72e/mar-tools-win64.zip

You are building Firefox without adapted config https://dxr.mozilla.org/mozilla-esr52/source/old-configure.in#1216, so there could be surprises.

Replying to boklm:

The patch adding the flag back: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_20636_v6&id=90c730d131d3e003ff626ed613f37c43ada0cb7e

And the build with the patch and the flag: https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/tor-0.3.1.5-alpha-windows-x86_64-3eb3d4/tor-win32-0.3.1.5-alpha.zip.asc https://people.torproject.org/~boklm/tmp/bug_23228/with-patch/tor-0.3.1.5-alpha-windows-x86_64-3eb3d4/tor-win32-0.3.1.5-alpha.zip

Looks better. The ImageBase address changes for me on different restarts, so, yay! You can test that yourself with e.g. ProcessExplorer (opening the lower pane and there should be a column for the addresses). The .exe without the patch has 0x400000 as ImageBase address.

This might need more testing, I guess, and someone familiar with binutils to look over the changes. Not finding anyone to do so should at least not block nightly builds with the new toolchain.

One dllexport function, and you don't need all that patches (like Firefox without --disable-sandbox).

I can confirm that both tor.exe and firefox.exe have 0x400000 as ImageBase in the version without patch, and something different in the version with the patch.

Replying to boklm:

I can confirm that both tor.exe and firefox.exe have 0x400000 as ImageBase in the version without patch, and something different in the version with the patch.

You mean something different with every restart, right? If so, I guess we can start using that idea and close #23458 (moved)? If you agree could you provide an updated patch for this ticket for review?

We could fix ld, if you wish.

Replying to cypherpunks:

We could fix ld, if you wish.

Yes, I wish but aren't we doing that? If not, how would a fix look like?

Replying to gk:

Replying to cypherpunks:

We could fix ld, if you wish.

Yes, I wish but aren't we doing that? If not, how would a fix look like? That patches are hacks and are not upstreamable. The fix would change the behavior of ld back to normal and would fulfill the requirements of https://sourceware.org/bugzilla/show_bug.cgi?id=19011.

So, if you're ready to get it going till merged upstream, then I can provide the required info.

Replying to cypherpunks:

So, if you're ready to get it going till merged upstream, then I can provide the required info.

Not sure what exactly you want from us. Of course, we would like to upstream things. What special info is it that you have and we may lack?

I can write what to fix and how, if you agree to do all other things of this process.

Items for October 2017

Trac:
Keywords: TorBrowserTeam201709 deleted, TorBrowserTeam201710 added

Replying to gk:

Replying to boklm:

I can confirm that both tor.exe and firefox.exe have 0x400000 as ImageBase in the version without patch, and something different in the version with the patch.

You mean something different with every restart, right? If so, I guess we can start using that idea and close #23458 (moved)? If you agree could you provide an updated patch for this ticket for review?

Yes. I checked again with a new build that ImageBase is changing on each restart.

My branch bug_23228_v2 has a patch for review: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_23228_v2&id=1a3203baca12e9dcd8b5aa22ee1cdeb8b80fb727

Trac:
Keywords: TorBrowserTeam201710 deleted, TorBrowserTeam201710R added
Status: needs_revision to needs_review

Moving review to November

Trac:
Keywords: TorBrowserTeam201710R deleted, TorBrowserTeam201711R added

Looks good to me. This is commit b82d82d9cf9cb751e088085fb1093026b20a0e4d on master, thanks. boklm: Could you update the other patches for the win64 series accordingly so that we get nightlies built asap?

Trac:
Resolution: N/A to fixed
Status: needs_review to closed

closed

mentioned in issue #23456 (moved)

mentioned in issue #23458 (moved)

moved to tpo/applications/tor-browser#23228 (closed)

mentioned in issue tpo/applications/tor-browser#23458 (closed)