Opened 6 months ago

Closed 5 months ago

Last modified 5 months ago

#28874 closed defect (fixed)

https://browserleaks.com/webgl crashes tab on 64bit Tor Browser for Windows

Reported by: gk Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-crash, tbb-rbm, TorBrowserTeam201901R
Cc: tom Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by gk)

On our blog one user noted (https://blog.torproject.org/comment/278911#comment-278911) that https://browserleaks.com/webgl is crashing their tab reliably using 8.5a6 in the 64bit flavor for Windows. This does not happen with the 32bit variant. And neither does this happen with vanilla Firfox 60.4.0esr.

It seems that a bare Firefox compiled our way without any Tor Browser patches is crashing as well. Thus, I wonder whether we hit a GCC bug or something that is only affecting some 64bit system. Mysterious.

Child Tickets

Change History (21)

comment:1 Changed 6 months ago by gk

Description: modified (diff)

comment:2 Changed 6 months ago by gk

I tried to create debug builds to help narrowing this down but that only resulted in sadness and #28875. :(

comment:3 Changed 6 months ago by gk

Keywords: TorBrowserTeam201812 added
Status: newneeds_information

Does the vanilla 64bit build work for you, cypherpunk? (see: https://queue.taskcluster.net/v1/task/crV_nUFPRjarCLDD-mmZRQ/runs/0/artifacts/public/build/target.zip for an artifact. Unzipping the target.zip should give you a firefox folder to test).

Last edited 6 months ago by gk (previous) (diff)

comment:4 in reply to:  3 Changed 6 months ago by gk

Replying to gk:

Does the vanilla 64bit build work for you, cypherpunk? (see: https://queue.taskcluster.net/v1/task/crV_nUFPRjarCLDD-mmZRQ/runs/0/artifacts/public/build/target.zip for an artifact. Unzipping the target.zip should give you a firefox folder to test).

That one is crashing as well. It seems the user hits

        if (!mResolvedDefaultFB) {
            gfxCriticalNote << funcName << ": Failed to create mResolvedDefaultFB.";
            return false;
        }

in dom/canvas/WebGLContext.cpp immediately before the crash.

comment:5 Changed 6 months ago by gk

Hey, could you test a debug build to gather more information about what is going on?

https://people.torproject.org/~gk/testbuilds/torbrowser-install-win64-tbb-nightly_debug_en-US.exe
https://people.torproject.org/~gk/testbuilds/torbrowser-install-win64-tbb-nightly_debug_en-US.exe.asc

It will crash during first start (which is #28875) but should work afterwards. It would be interesting to see if we hit any assertions before crashing.

comment:6 Changed 6 months ago by gl

not sure how to test it, but here is what I've got:
it crashed during first start as expected;
it got stuck at connect to tor phase during second and third start due to #28405 and #28614;
it connected and "crashed" from webgl as expected during fourth start; nothing in debug logs;
then I set e10s off and got

WARN: initializeD3DDevice(799): Failed creating Debug D3D11 device - falling back to release runtime.
Initializing context 000002b11d306db0 surface 000002b11d364500 on display 000002b11fa9e5b0
Initializing context 000002b11530efc0 surface 000002b11d362300 on display 000002b11fa9e5b0
Initializing context 000002b11c731c20 surface 000002b11d358c00 on display 000002b11fa9e5b0
JavaScript error: , line 0: uncaught exception: TagError: adsbygoogle.push() error: All ins elements in the DOM with class=adsbygoogle already have ads in them.
JavaScript error: , line 0: uncaught exception: TagError: adsbygoogle.push() error: All ins elements in the DOM with class=adsbygoogle already have ads in them.
Initializing context 000002b1140d85f0 surface 000002b115358920 on display 000002b11fa9e5b0
ERR: ensureInitialized(141): D3D compiler module not found.
[gl:000002b127ca1ee0] void mozilla::gl::GLContext::fLinkProgram(GLuint): Generated unexpected GL_OUT_OF_MEMORY error. (0x0505)
Hit MOZ_CRASH(Unexpected error with MOZ_GL_DEBUG_ABORT_ON_ERROR. (Run with MOZ_GL_DEBUG_ABORT_ON_ERROR=0 to disable)) at /var/tmp/build/firefox-8cac0295141b/gfx/gl/GLContext.cpp:3030

with usual "EXCEPTION_ACCESS_VIOLATION_READ" in xul.dll

comment:7 in reply to:  6 ; Changed 6 months ago by gk

Thanks!

Replying to gl:

ERR: ensureInitialized(141): D3D compiler module not found.

I guess that's the reason for the trouble you are seeing. The relevant code around it is:

#if !defined(ANGLE_ENABLE_WINDOWS_STORE)
#if defined(ANGLE_PRELOADED_D3DCOMPILER_MODULE_NAMES)
    // Find a D3DCompiler module that had already been loaded based on a predefined list of versions.
    static const char *d3dCompilerNames[] = ANGLE_PRELOADED_D3DCOMPILER_MODULE_NAMES;

    for (size_t i = 0; i < ArraySize(d3dCompilerNames); ++i)
    {
        if (GetModuleHandleExA(0, d3dCompilerNames[i], &mD3DCompilerModule))
        {
            break;
        }
    }
#endif  // ANGLE_PRELOADED_D3DCOMPILER_MODULE_NAMES

    if (!mD3DCompilerModule)
    {
        // Load the version of the D3DCompiler DLL associated with the Direct3D version ANGLE was built with.
        mD3DCompilerModule = LoadLibrary(D3DCOMPILER_DLL);
    }

    if (!mD3DCompilerModule)
    {
        ERR() << "D3D compiler module not found.";
        return gl::OutOfMemory() << "D3D compiler module not found.";
    }

    mD3DCompileFunc = reinterpret_cast<pD3DCompile>(GetProcAddress(mD3DCompilerModule, "D3DCompile"));
    ASSERT(mD3DCompileFunc);

    mD3DDisassembleFunc = reinterpret_cast<pD3DDisassemble>(GetProcAddress(mD3DCompilerModule, "D3DDisassemble"));
    ASSERT(mD3DDisassembleFunc);

#else
    // D3D Shader compiler is linked already into this module, so the export
    // can be directly assigned.
    mD3DCompilerModule = nullptr;
    mD3DCompileFunc = reinterpret_cast<pD3DCompile>(D3DCompile);
    mD3DDisassembleFunc = reinterpret_cast<pD3DDisassemble>(D3DDisassemble);
#endif

in HLSLCompiler.cpp

We are not defining ANGLE_PRELOADED_D3DCOMPILER_MODULE_NAMES in https://dxr.mozilla.org/mozilla-esr60/source/gfx/angle/update-angle.py#257. But I wonder where D3DCOMPILER_DLL is supposed to come from in our case. From d3dcompiler.rs??

comment:9 Changed 6 months ago by gl

what should I check? and why is that 64-bit-specific?

comment:10 in reply to:  7 Changed 6 months ago by tom

Replying to gk:

We are not defining ANGLE_PRELOADED_D3DCOMPILER_MODULE_NAMES in https://dxr.mozilla.org/mozilla-esr60/source/gfx/angle/update-angle.py#257. But I wonder where D3DCOMPILER_DLL is supposed to come from in our case. From d3dcompiler.rs??

Nope: https://github.com/apitrace/dxsdk/blob/master/Include/d3dcompiler.h#L18

Since all this seems to be related to DirectX; I suppose getting some info about that might be helpful.

Start -> Run -> dxdiag will get diagnostic information. At a minimum I'd say go to the Display page and copy that info (either with a screenshot, or save the info to a text file and copy it out.)

comment:11 Changed 6 months ago by gk

gl: any update on this issue?

comment:13 Changed 6 months ago by tom

Thanks cypherpunk; that's probably it. I'm sending in some trials on TaskCluster to see if that solves my 8-month old error. I pinged Jacek about whether MinGW should update or we should override.

comment:14 Changed 6 months ago by gk

gl or anyone else who hits this problem: does it get solved by

https://people.torproject.org/~gk/testbuilds/torbrowser-install-win64-tbb-nightly_28874_en-US.exe
https://people.torproject.org/~gk/testbuilds/torbrowser-install-win64-tbb-nightly_28874_en-US.exe.asc

? (You might need to restart Tor Browser due to things like #28405)

The bundle got built with tom's mingw-w64 fix, fwiw.

Last edited 6 months ago by gk (previous) (diff)

comment:15 Changed 6 months ago by jacek

D3DCOMPILER_DLL define is fixed in upstream mingw-w64 now. I recall that some declarations also vary between d3dcompiler versions, so if similar problems persist, it might be good place to look.

comment:16 Changed 6 months ago by gk

Keywords: TorBrowserTeam201901 added; TorBrowserTeam201812 removed

Moving tickets to Jan 2019.

comment:17 in reply to:  15 ; Changed 5 months ago by gk

Keywords: tbb-rbm TorBrowserTeam201901R added; TorBrowserTeam201901 removed
Status: needs_informationneeds_review

Replying to jacek:

D3DCOMPILER_DLL define is fixed in upstream mingw-w64 now. I recall that some declarations also vary between d3dcompiler versions, so if similar problems persist, it might be good place to look.

Thanks! It seems we are fine for now by just bumping the mingw-w64 commit we use (see the test results in comment:32:ticket:28823 compared to comment:28:ticket:28823). bug_28874 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_28874&id=5e875c30e825bd726558a3b9fea63b59d2a80d15) has the respective patch for review.

comment:19 in reply to:  17 Changed 5 months ago by boklm

Resolution: fixed
Status: needs_reviewclosed

Replying to gk:

Thanks! It seems we are fine for now by just bumping the mingw-w64 commit we use (see the test results in comment:32:ticket:28823 compared to comment:28:ticket:28823). bug_28874 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_28874&id=5e875c30e825bd726558a3b9fea63b59d2a80d15) has the respective patch for review.

This looks good to me. I cherry-picked this patch as commit 87f892495a13e878922147429e4c2b7a423f0d8e on master.

comment:21 Changed 5 months ago by tom

I passed that to Jacek; and he thinks that should be fixed as well; so we may need another bump - although I'm not sure if/how thee values are actually used.

Thank you!

Note: See TracTickets for help on using tickets.