Opened 7 months ago

Closed 6 months ago

#26204 closed defect (fixed)

Bundle d3dcompiler_47.dll for Tor Browser 8

Reported by: gk Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: ff60-esr, TorBrowserTeam201806
Cc: sukhbir, tom Actual Points:
Parent ID: #26203 Points:
Reviewer: Sponsor:

Description

See: https://bugzilla.mozilla.org/show_bug.cgi?id=1460620 and the related fxc/fxc2-sage (https://bugzilla.mozilla.org/show_bug.cgi?id=1370007) for context. But regardless we need to bundle d3dcompiler_47.dll as we are otherwise risking breaking Tor Browser for a bunch of users.

Child Tickets

Change History (21)

comment:1 Changed 7 months ago by gk

Cc: sukhbir added; sukhe removed

CCing the real sukhe

comment:2 Changed 7 months ago by sukhbir

Status: newneeds_review

For review:

https://github.com/azadi/tor-browser-build-1/tree/bug-26204

This commit adds support for building and bundling fxc2.

Since the Makefile upstream is pretty simple and has no install targets for example, I manually do that in the fxc2/build file. Ideally, there are a few changes we could make upstream:

  • add a proper version
  • improve the Makefile
  • create and sign a release

But I am not sure if that is required and I am assuming that the current changes on our end are enough. If you feel that we should submit a patch upstream instead, let me know.

comment:3 Changed 7 months ago by boklm

Status: needs_reviewneeds_revision

The patch looks good to me, although I did not try to build the firefox part.

However I think we are missing the part where we copy the d3dcompiler_47.dll file to the bundle (I think in $distdir/Browser).

And a minor thing: you can indent the make lines in the IF to make that easier to read. Or maybe it can be simplified like this:

diff --git a/projects/fxc2/build b/projects/fxc2/build
index 6ff3808..1b48326 100644
--- a/projects/fxc2/build
+++ b/projects/fxc2/build
@@ -8,11 +8,7 @@ mkdir -p /var/tmp/build
 tar -C /var/tmp/build -xf [% project %]-[% c("version") %].tar.gz
 cd /var/tmp/build/[% project %]-[% c("version") %]
 
-[% IF c("arch") == "i686" -%]
-make x86
-[% ELSE -%]
-make x64
-[% END -%]
+make [% IF c("var/windows-i686") %]x86[% ELSE %]x64[% END %]
 
 mkdir -p /var/tmp/dist/fxc2/bin/
 cp fxc2.exe /var/tmp/dist/fxc2/bin/

comment:4 in reply to:  3 ; Changed 7 months ago by sukhbir

Replying to boklm:

Thanks for the review!

The patch looks good to me, although I did not try to build the firefox part.

However I think we are missing the part where we copy the d3dcompiler_47.dll file to the bundle (I think in $distdir/Browser).

I added it by setting fxcdir and copying the dll. Is that fine or can we do better?

And a minor thing: you can indent the make lines in the IF to make that easier to read. Or maybe it can be simplified like this:

diff --git a/projects/fxc2/build b/projects/fxc2/build
index 6ff3808..1b48326 100644
--- a/projects/fxc2/build
+++ b/projects/fxc2/build
@@ -8,11 +8,7 @@ mkdir -p /var/tmp/build
 tar -C /var/tmp/build -xf [% project %]-[% c("version") %].tar.gz
 cd /var/tmp/build/[% project %]-[% c("version") %]
 
-[% IF c("arch") == "i686" -%]
-make x86
-[% ELSE -%]
-make x64
-[% END -%]
+make [% IF c("var/windows-i686") %]x86[% ELSE %]x64[% END %]
 
 mkdir -p /var/tmp/dist/fxc2/bin/
 cp fxc2.exe /var/tmp/dist/fxc2/bin/

Updated; to maintain consistency, I went with adding the indent instead of doing it one line.

(Same branch.)

comment:5 Changed 7 months ago by sukhbir

The Firefox build detects fxc2 but fails due to the other toolchain errors:

 0:03.71 checking for fxc... /var/tmp/dist/fxc2/bin/fxc2.exe
 0:03.71 checking for wine... /usr/bin/wine
 0:03.94 checking for cpuid.h... yes
 [snip]
 0:04.92 checking for rustc... not found
 0:04.92 checking for cargo... not found
 0:04.92 ERROR: Rust compiler not found.
 0:04.92 To compile rust language sources, you must have 'rustc' in your path.
 0:04.92 See https://www.rust-lang.org/ for more information.
 0:04.92 
 0:04.92 You can install rust by running './mach bootstrap'
 0:04.92 or by directly running the installer from https://rustup.rs/
 0:04.92 
 0:04.95 *** Fix above errors and then restart with\
 0:04.95                "/usr/bin/make -f client.mk build"
 0:04.95 client.mk:145: recipe for target 'configure' failed
 0:04.95 make: *** [configure] Error 1

comment:6 in reply to:  4 ; Changed 7 months ago by boklm

Replying to sukhbir:

Replying to boklm:

Thanks for the review!

The patch looks good to me, although I did not try to build the firefox part.

However I think we are missing the part where we copy the d3dcompiler_47.dll file to the bundle (I think in $distdir/Browser).

I added it by setting fxcdir and copying the dll. Is that fine or can we do better?

This looks good. If you set fxcdir before setting PATH, you can use it while seting PATH.

An other thing I noticed while looking at the patch again: in projects/fxc2/build you can replace /var/tmp/dist/fxc2/bin/ in the 3 lines it is used by "$distdir/bin/".

comment:7 in reply to:  4 ; Changed 7 months ago by tom

Replying to sukhbir:

I added it by setting fxcdir and copying the dll. Is that fine or can we do better?

The patch here: https://bugzilla.mozilla.org/show_bug.cgi?id=1460620 will get it into dist/bin in a portable way. I haven't landed it because I haven't been able to investigate the test failures indicating that sometimes we don't find the dll and sometimes we do.

comment:8 in reply to:  6 Changed 7 months ago by sukhbir

Replying to boklm:

Replying to sukhbir:

Replying to boklm:

Thanks for the review!

The patch looks good to me, although I did not try to build the firefox part.

However I think we are missing the part where we copy the d3dcompiler_47.dll file to the bundle (I think in $distdir/Browser).

I added it by setting fxcdir and copying the dll. Is that fine or can we do better?

This looks good. If you set fxcdir before setting PATH, you can use it while seting PATH.

An other thing I noticed while looking at the patch again: in projects/fxc2/build you can replace /var/tmp/dist/fxc2/bin/ in the 3 lines it is used by "$distdir/bin/".

Done, thanks.

comment:9 in reply to:  7 Changed 7 months ago by sukhbir

Replying to tom:

Replying to sukhbir:

I added it by setting fxcdir and copying the dll. Is that fine or can we do better?

The patch here: https://bugzilla.mozilla.org/show_bug.cgi?id=1460620 will get it into dist/bin in a portable way. I haven't landed it because I haven't been able to investigate the test failures indicating that sometimes we don't find the dll and sometimes we do.

Thanks; CCed.

comment:10 Changed 7 months ago by sukhbir

Status: needs_revisionneeds_review

comment:11 Changed 7 months ago by boklm

Keywords: TorBrowserTeam201805R added; TorBrowserTeam201805 removed

comment:12 Changed 7 months ago by gk

Cc: tom added
Keywords: TorBrowserTeam201805 added; TorBrowserTeam201805R removed
Status: needs_reviewneeds_revision

So, we want to use a 32bit .dll for our 64bit Windows builds, too? It seems to me Mozilla is using a 64-bit one for 64-bit Windows and a 32-bit one for 32-bit Windows:

# The DirectX SDK libraries are split into x86 and x64 sub-directories
case "${target_cpu}" in
i*86)
  MOZ_D3D_CPU_SUFFIX=x86
  ;;
x86_64)
  MOZ_D3D_CPU_SUFFIX=x64
  ;;
esac

and

MOZ_D3DCOMPILER_VISTA_DLL_PATH="$WINDOWSSDKDIR/Redist/D3D/$MOZ_D3D_CPU_SUFFIX/$MOZ_D3DCOMPILER_VISTA_DLL"

I think we should do the same. Tom, could you push a trustworthy .dll into your repo? And could you even give us a signed git tag (that would rock)?

Last edited 7 months ago by gk (previous) (diff)

comment:13 Changed 7 months ago by tom

Alright I decided to try to be thorough about this. Here are _all_ the d3d compilers on my system. (Lots of apps ship their own copy...)

 Get-ChildItem -Path C:\ -Filter d3dcompiler_47.dll -Recurse -ErrorAction SilentlyContinue -Force | foreach-object { "{3}`t{2}`t{1}`t{0}" -f $_.FullName, [
System.Diagnostics.FileVersionInfo]::GetVersionInfo($_.FullName).FileVersion, [System.BitConverter]::ToString($md5.ComputeHash([System.IO.File]::ReadAllBytes($_.FullName))), (Get-ExeTargetM
achine $_.FullName).TargetMachine } | sort
x64     22-2D-02-0B-D3-3C-90-17-0A-82-96-AD-C1-B7-03-6A 10.0.17134.12 (WinBuild.160101.0800)            C:\Users\Tom\AppData\Local\Google\Chrome SxS\Application\69.0.3447.2\d3dcompiler_47.dll
x64     30-A7-C8-AB-59-39-65-0E-A9-B3-40-3B-BD-B6-0C-CA 10.0.14393.33 (rs1_release_sec.160727-1952)     C:\Program Files (x86)\Windows Kits\10\bin\x64\d3dcompiler_47.dll
x64     30-A7-C8-AB-59-39-65-0E-A9-B3-40-3B-BD-B6-0C-CA 10.0.14393.33 (rs1_release_sec.160727-1952)     C:\Users\Tom\AppData\Local\Temp\moz_mapi\d3dcompiler_47.dll
x64     38-75-75-BA-5B-0A-E8-1A-4E-A8-22-9C-09-3B-80-E1 10.0.15063.468 (WinBuild.160101.0800)           C:\Program Files (x86)\Google\Chrome\Application\66.0.3359.170\d3dcompiler_47.dll
x64     38-75-75-BA-5B-0A-E8-1A-4E-A8-22-9C-09-3B-80-E1 10.0.15063.468 (WinBuild.160101.0800)           C:\Program Files (x86)\Google\Chrome\Application\67.0.3396.62\d3dcompiler_47.dll
x64     38-75-75-BA-5B-0A-E8-1A-4E-A8-22-9C-09-3B-80-E1 10.0.15063.468 (WinBuild.160101.0800)           C:\Program Files (x86)\Mozilla Firefox\d3dcompiler_47.dll
x64     4D-3F-13-42-B7-DD-FB-46-99-0A-EC-98-30-03-15-9C 6.3.9600.16384 (winblue_rtm.130821-1623)        <private - game>
x64     69-9E-CC-49-CA-12-08-2D-DB-DA-30-66-96-90-EE-35 6.3.9600.16384 (winblue_rtm.130821-1623)        C:\Program Files (x86)\Windows Kits\8.1\bin\x64\d3dcompiler_47.dll
x64     8C-18-D8-E8-7E-33-F9-B2-F9-E5-7C-25-C0-EC-1B-46 10.0.16299.15 (WinBuild.160101.0800)            C:\mozilla-unified\obj-x86_64-pc-mingw32\dist\bin\d3dcompiler_47.dll
x64     8C-18-D8-E8-7E-33-F9-B2-F9-E5-7C-25-C0-EC-1B-46 10.0.16299.15 (WinBuild.160101.0800)            C:\Program Files (x86)\Windows Kits\10\Redist\D3D\x64\d3dcompiler_47.dll
x64     AC-18-E3-19-07-F2-97-1F-90-A8-A4-C2-64-0E-A7-76 10.0.16299.15 (WinBuild.160101.0800)            C:\Program Files (x86)\Windows Kits\10\bin\10.0.16299.0\x64\d3dcompiler_47.dll
x64     B0-AE-3A-A9-DD-1E-BD-60-BD-F5-1C-B9-48-34-CD-04 6.3.9600.16384 (winblue_rtm.130821-1623)        <private - game>
x64     B0-AE-3A-A9-DD-1E-BD-60-BD-F5-1C-B9-48-34-CD-04 6.3.9600.16384 (winblue_rtm.130821-1623)        C:\Program Files (x86)\Windows Kits\8.1\Redist\D3D\x64\d3dcompiler_47.dll
x64     B0-AE-3A-A9-DD-1E-BD-60-BD-F5-1C-B9-48-34-CD-04 6.3.9600.16384 (winblue_rtm.130821-1623)        C:\Program Files\Docker Toolbox\kitematic\d3dcompiler_47.dll
x64     B0-AE-3A-A9-DD-1E-BD-60-BD-F5-1C-B9-48-34-CD-04 6.3.9600.16384 (winblue_rtm.130821-1623)        C:\Program Files\Docker\Kitematic\d3dcompiler_47.dll
x64     B0-AE-3A-A9-DD-1E-BD-60-BD-F5-1C-B9-48-34-CD-04 6.3.9600.16384 (winblue_rtm.130821-1623)        <private - app>
x64     B0-AE-3A-A9-DD-1E-BD-60-BD-F5-1C-B9-48-34-CD-04 6.3.9600.16384 (winblue_rtm.130821-1623)        <private - app>
x64     BC-46-46-DB-3E-EB-D0-33-15-CC-8E-22-EC-1A-B9-0B 10.0.15063.674 (WinBuild.160101.0800)           C:\Program Files\Nightly\d3dcompiler_47.dll
x64     BD-5C-C2-5A-E4-9A-B3-3A-6A-BB-5D-7A-55-BF-D3-C1 10.0.10572.1000 (th2_release_sa.151014-2155)    C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Remote Debugger\x64\D3DCompiler_47.dll
x64     D5-09-B2-31-24-B7-CB-A5-7E-80-F5-9C-E1-32-04-A6 6.3.9600.16384 (winblue_rtm.130821-1623)        <private - app>
x64     DA-75-4D-87-F7-69-CB-21-D9-D2-84-7C-A8-75-41-52 10.0.10150.0 (th1.150616-1659)                  C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64\d3dcompiler_47.dll
x64     DA-75-4D-87-F7-69-CB-21-D9-D2-84-7C-A8-75-41-52 10.0.10150.0 (th1.150616-1659)                  C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\bin\HostX64\x64\d3dcompiler_47.dll
x64     DA-75-4D-87-F7-69-CB-21-D9-D2-84-7C-A8-75-41-52 10.0.10150.0 (th1.150616-1659)                  C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.13.26128\bin\Hostx64\x64\d3dcompiler_47.dll
x64     DA-C3-6E-09-CA-47-F4-97-88-3B-45-70-61-19-9F-9F 10.0.16299.15 (WinBuild.160101.0800)            C:\Windows\System32\D3DCompiler_47.dll
x64     DA-C3-6E-09-CA-47-F4-97-88-3B-45-70-61-19-9F-9F 10.0.16299.15 (WinBuild.160101.0800)            C:\Windows\WinSxS\amd64_microsoft-windows-directx-d3dcompiler_31bf3856ad364e35_10.0.16299.15_none_0c3e46468d34dbee\D3DCompiler_47.dll
x86     09-64-CB-D3-AB-63-C4-98-93-82-C2-68-86-43-C2-3B 10.0.14393.795 (rs1_release_sec.170105-1850)    <private - game>
x86     1D-9E-B7-DF-27-A4-8F-48-FE-3F-0D-18-31-BA-F3-66 10.0.10572.1000 (th2_release_sa.151014-2155)    C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Remote Debugger\x86\D3DCompiler_47.dll
x86     23-68-13-63-EA-C3-B6-84-C7-1E-8B-DE-82-D5-60-6F 6.3.9600.16384 (winblue_rtm.130821-1623)        <private - game>
x86     6A-7D-AA-DA-02-40-49-FE-13-7B-48-81-E9-20-85-82 10.0.16299.15 (WinBuild.160101.0800)            C:\Program Files (x86)\Windows Kits\10\bin\10.0.16299.0\x86\d3dcompiler_47.dll
x86     70-6F-89-4C-8E-31-07-F5-EA-C8-B7-A5-35-76-A8-33 6.3.9600.16384 (winblue_rtm.130821-1623)        C:\Program Files (x86)\Windows Kits\8.1\bin\x86\d3dcompiler_47.dll
x86     70-6F-89-4C-8E-31-07-F5-EA-C8-B7-A5-35-76-A8-33 6.3.9600.16384 (winblue_rtm.130821-1623)        <private - game>
x86     70-6F-89-4C-8E-31-07-F5-EA-C8-B7-A5-35-76-A8-33 6.3.9600.16384 (winblue_rtm.130821-1623)        <private - game>
x86     73-AC-F6-BC-7F-4A-BB-2A-6F-3C-65-AD-F1-D2-F3-A8 10.0.10586.212 (th2_release_sec.160328-1908)    <private - app>
x86     8D-56-95-F0-B0-A0-33-0F-E0-78-02-E4-F3-57-6B-15 10.0.10586.212 (th2_release_sec.160328-1908)    <private - game>
x86     A8-30-1C-BD-C3-18-1B-3E-F8-9F-C5-1F-B2-ED-DD-30 10.0.10150.0 (th1.150616-1659)                  C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\d3dcompiler_47.dll
x86     A8-30-1C-BD-C3-18-1B-3E-F8-9F-C5-1F-B2-ED-DD-30 10.0.10150.0 (th1.150616-1659)                  C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\bin\HostX86\x86\d3dcompiler_47.dll
x86     A8-30-1C-BD-C3-18-1B-3E-F8-9F-C5-1F-B2-ED-DD-30 10.0.10150.0 (th1.150616-1659)                  C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.13.26128\bin\Hostx86\x86\d3dcompiler_47.dll
x86     C5-B3-62-BC-E8-6B-B0-AD-31-49-C4-54-02-01-33-1D 6.3.9600.16384 (winblue_rtm.130821-1623)        C:\Program Files (x86)\Microsoft Visual Studio\Installer\d3dcompiler_47.dll
x86     C5-B3-62-BC-E8-6B-B0-AD-31-49-C4-54-02-01-33-1D 6.3.9600.16384 (winblue_rtm.130821-1623)        C:\Program Files (x86)\Windows Kits\8.1\Redist\D3D\x86\d3dcompiler_47.dll
x86     C5-B3-62-BC-E8-6B-B0-AD-31-49-C4-54-02-01-33-1D 6.3.9600.16384 (winblue_rtm.130821-1623)        <private - app>
x86     C9-4A-85-19-52-CC-F8-94-B6-67-D9-65-B3-99-1C-45 10.0.14393.33 (rs1_release_sec.160727-1952)     C:\Program Files (x86)\Windows Kits\10\bin\x86\d3dcompiler_47.dll
x86     CB-3E-07-DF-9B-B0-31-A1-22-E1-86-54-E2-A8-92-2B 10.0.15063.400 (WinBuild.160101.0800)           <private - app>
x86     CB-3E-07-DF-9B-B0-31-A1-22-E1-86-54-E2-A8-92-2B 10.0.15063.400 (WinBuild.160101.0800)           <private - app>
x86     CF-C3-9F-97-FF-3B-32-D4-E9-DA-84-5F-D4-60-35-EC 10.0.14393.33 (rs1_release_sec.160727-1952)     C:\Program Files (x86)\Mozilla Thunderbird\d3dcompiler_47.dll
x86     CF-C3-9F-97-FF-3B-32-D4-E9-DA-84-5F-D4-60-35-EC 10.0.14393.33 (rs1_release_sec.160727-1952)     C:\Program Files (x86)\Mozilla Thunderbird\updated\d3dcompiler_47.dll
x86     D8-08-AC-B5-34-36-CA-8B-F0-4F-88-D3-B4-02-00-CB 10.0.16299.15 (WinBuild.160101.0800)            C:\Program Files (x86)\Windows Kits\10\Redist\D3D\x86\d3dcompiler_47.dll
x86     E1-6C-A7-6A-14-1D-E6-3A-7C-A6-61-E4-89-F9-1E-7E 10.0.15063.468 (WinBuild.160101.0800)           <private - game>
x86     E1-6C-A7-6A-14-1D-E6-3A-7C-A6-61-E4-89-F9-1E-7E 10.0.15063.468 (WinBuild.160101.0800)           <private - game>
x86     E1-6C-A7-6A-14-1D-E6-3A-7C-A6-61-E4-89-F9-1E-7E 10.0.15063.468 (WinBuild.160101.0800)           <private - game>
x86     E3-CC-43-AA-6F-5D-F7-94-94-EB-A9-75-CA-8E-5A-DB 10.0.16299.15 (WinBuild.160101.0800)            C:\Windows\SysWOW64\D3DCompiler_47.dll
x86     E3-CC-43-AA-6F-5D-F7-94-94-EB-A9-75-CA-8E-5A-DB 10.0.16299.15 (WinBuild.160101.0800)            C:\Windows\WinSxS\wow64_microsoft-windows-directx-d3dcompiler_31bf3856ad364e35_10.0.16299.15_none_1692f098c1959de9\D3DCompiler_47.dll

Which would you like? =)

comment:14 in reply to:  13 ; Changed 7 months ago by gk

Replying to tom:

Alright I decided to try to be thorough about this. Here are _all_ the d3d compilers on my system. (Lots of apps ship their own copy...)

[snip]

Which would you like? =)

That's easy: we want the same libraries Mozilla bundles in ESR 60. :)

comment:15 in reply to:  14 Changed 7 months ago by cypherpunks

Replying to gk:

Replying to tom:

Alright I decided to try to be thorough about this. Here are _all_ the d3d compilers on my system. (Lots of apps ship their own copy...)

[snip]

Which would you like? =)

That's easy: we want the same libraries Mozilla bundles in ESR 60. :)

And why don't you want the same libraries of M$ C Runtime for, e.g., #12968?

comment:16 Changed 6 months ago by gk

Keywords: TorBrowserTeam201806 added; TorBrowserTeam201805 removed

Moving our tickets to June 2018

comment:19 Changed 6 months ago by sukhbir

For now, we are doing this:

https://github.com/mozilla/fxc2/pull/1

(Copying the DLLs from the Firefox build.)

comment:20 in reply to:  19 Changed 6 months ago by cypherpunks

Replying to sukhbir:

For now, we are doing this:

https://github.com/mozilla/fxc2/pull/1

(Copying the DLLs from the Firefox build.)

If you'd like to update them with every Windows update, then sure.

comment:21 Changed 6 months ago by gk

Resolution: fixed
Status: needs_revisionclosed

Fixed with commit 57693e01837f0fc7c60bb056e5b752f5f67a987b on tor-browser-build's master branch.

Note: See TracTickets for help on using tickets.