TBB 3.0a1 crashes on startup on WinXP (missing entry points for _vsnprintf_s in msvcrt.dll)
firefox.exe - Entry Point Not Found The procedure entry point snprintf could not be located in the dynamic link library msvcrt.dll
Trac:
Username: dope457
Activity
changed milestone to %Tor: unspecified
Replying to dope457:
firefox.exe - Entry Point Not Found The procedure entry point snprintf could not be located in the dynamic link library msvcrt.dll
Same error for me. Windows XP with MSVCRT.DLL version 7.0.2600.5512 (xpsp.080413-2111)
Trac:
Username: Logforme
In the V8 code, they apparently undef MINGW_HAS_SECURE_API for this reason. Not sure if that is a valid fix for us or not: http://www.mail-archive.com/v8-dev@googlegroups.com/msg58589.html https://qt.gitorious.org/qt/qtjsbackend/commit/7370d84356ed4551ec1d72800666992817901b11/diffs
Trac:
Keywords: bundle 3.0, firefox deleted, tbb-3.0, tbb-crash added
I don't think MINGW_HAS_SECURE_API is involved here. sprintf() is not available at all in the runtime lib (only _sprintf()) see e.g.: http://mingw.5.n7.nabble.com/snprintf-sscanf-mismatch-for-p-td20596.html
Trac:
Cc: N/A to g.koppen@jondos.de-
Okay, this move breaks the build. It turns out, there is a bunch of files where _vsnprintf_s is used/needed and there are other secure functions deployed as well (like _vsnwprintf_s). That probably amounts to a lot of patching. :-/ And disabling the secure API for all Windows users is not funny either. Are there any other things we could do here? Asking the affected users to get a newer msvcrt? Or bundling a newer crt with the Tor Browser? Something along the lines: "If you intend to distribute your build to others, you will need to set WIN32_REDIST_DIR=$VCINSTALLDIR\redist\x86\Microsoft.VC80.CRT in your mozconfig to get the Microsoft CRT DLLs packaged along with the application." as mentioned on https://developer.mozilla.org/en-US/docs/Developer_Guide/Build_Instructions/Windows_Prerequisites maybe?
Trac:
Status: new to needs_information
Summary: TBB 3.0a1 crashes on startup on WinXP (missing entry points for snprintf in msvcrt.dll) to TBB 3.0a1 crashes on startup on WinXP (missing entry points for _vsnprintf_s in msvcrt.dll) -
Is this what we need: https://www.microsoft.com/en-us/download/confirmation.aspx?id=3387
Notice that while that page is https, the final exe download is performed over http (as is the case for all of the Microsoft development tool binaries). Go go microsoft security!
Unless we can find a canonical version of that file, it seems we might be at an impasse?
-
Replying to mikeperry:
Is this what we need: https://www.microsoft.com/en-us/download/confirmation.aspx?id=3387
Well, sort of. I'd take https://www.microsoft.com/en-us/download/details.aspx?id=5555 as VC 2010 (and thus the respective runtime libs) is the minimum to compile Fx nowadays and get it going. But I don't know yet whether this idea works at all as I've read things like "MinGW does not use Visual C++ Redistributable" and "MinGW uses the Microsoft runtime libraries, distributed with the Windows operating system.".
Unless we can find a canonical version of that file, it seems we might be at an impasse?
Hmm... I gonna ask the Microsoft support if there is something we could do here (maybe we get some hashes of the canonical version, dunno). Sounds like fun. :)
Trac:
Status: needs_information to new -
Excuse me coming in here, but it seems to me that this bug is stalled due to a misunderstanding.
On Windows XP Firefox fails to start, because the function _vsnprintf_s() is not present in msvcrt.dll. _vsnprintf_s() and all the *_s() functions are present in the Visual Studio runtimes, because they are a Microsoft addition as "security enhanced" versions of the functions without the _s suffix. As far as I can tell they have also been backported to msvcrt.dll on Vista/7 only. This is a Windows system file and you may not take the file from Vista/7 and copy it to XP (it won't work even if you did).
Accordingly, the way forward looks like one of the following: (1) Compile with Visual Studio 2010 or VS 2012 (2) Target only Vista/7 upwards (effectively a WONTFIX for XP) (3) Write your own *_s() functions. Required are 5 variations of *printf_s(), _wcslwr_s(), _wfopen_s(), fopen_s(), rand_s() and _controlfp_s() (10 in all). The first 9 are called from xul.dll and the 10th from gkmedias.dll. I have walked all the 3.0a1 package (.exe and .dll files) through Dependency Walker to find what is missing. This option looks like a major effort, except that others have already done it e.g. the WINE project, for which the source code is available under GNU Lesser GPL 2.1. (4) Reconfigure MinGW to link to the equivalent of the Visual Studio runtimes e.g. msvcr90.dll (which is the runtime for VS 2008). I don't know if this works, not having tried it myself, but others have suggested that the options: gcc foo.c -nostdlib -lmsvcr90 -lgcc ... might work.
Incidentally, undef-ing MINGW_HAS_SECURE_API (as done in Chromium V8) will not work unless you follow option (3) because V8 provides its own *_s() functions in the file platform-win32.cc.
Trac:
Username: James_A
Cc: g.koppen@jondos.de to g.koppen@jondos.de, jmaxhaven@yahoo.co.uk -
Ok, the exe from https://www.microsoft.com/en-us/download/details.aspx?id=5555 has sha256sum 8162b2d665ca52884507ede19549e99939ce4ea4a638c537fa653539819138c8 from a few Tor exits.
I ran it in wine, and it installed this: 111ba3befa7b5fb1e736e34aa7130ab148122c16177bcbc686c7f87b6b4ea694 .wine/drive_c/windows/winsxs/x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_deadbeef/msvcr90.dll
Is that what we want? Does anyone have a different hash for msvcr90.dll?
-
Replying to mikeperry:
Ok, the exe from https://www.microsoft.com/en-us/download/details.aspx?id=5555 has sha256sum 8162b2d665ca52884507ede19549e99939ce4ea4a638c537fa653539819138c8 from a few Tor exits.
I can confirm this with Tor-independent means. And the .exe is signed by Microsoft which is much better than nothing.
I ran it in wine, and it installed this: 111ba3befa7b5fb1e736e34aa7130ab148122c16177bcbc686c7f87b6b4ea694 .wine/drive_c/windows/winsxs/x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_deadbeef/msvcr90.dll
Is that what we want?
No, actually not. That one is a runtime lib for VS 2008. Weird, that it is included into the 2010 package at all. I'd like to have the *100.dlls. That said, we could grab them from Mozilla as they are shipped with Firefox. Maybe Mozilla has found a good way to verify that the dlls are untampered. Anyway, I failed badly trying to get into contact with the responsible people at Microsoft (surprise!).
-
Replying to mikeperry:
Ah, ok. Here's my hash for msvcr100.dll. MSVCR100_HASH=1221a09484964a6f38af5e34ee292b9afefccb3dc6e55435fd3aaf7c235d9067
With the help of #build on irc.mozilla.org, I think I've got it linking in. I will commit those changes shortly.
I confrm that a copy of msvcr100.dll (32-bit) RTM with a verified correct digital signature (signed by Microsoft Code Signing PCA) has the SHA256 hash of 1221a09484964a6f38af5e34ee292b9afefccb3dc6e55435fd3aaf7c235d9067
If you post a test build somewhere I can verify it.
Trac:
Username: James_A -
Replying to mikeperry:
Ok, builders/tor-browser-builder commit 3f83d43a323ba18479894f5c12e4c99ff80882f7 should build a Windows Bundle that links against and includes that DLL.
If anyone has a WinXP vm that would like me to put a test build somewhere to verify that it no longer crashes, I can do that later this week.
Yeah, I can also verify it.
Trac:
Username: dope457 -
Here's a test build courtesy of gk: https://people.torproject.org/~mikeperry/3.0a3pre-test/torbrowser-install-3.0-alpha-2_en-US.exe
cca9211633d3e59111e1c411e98e1ea3e1c94111867fb9de62a78dd89f5e16e4 torbrowser-install-3.0-alpha-2_en-US.exe
gk also noticed that Firefox also ships with msvcp100.dll, so this build may still crash, depending on if we actually need that DLL for mingw-w64 builds too, or not. Feedback on that would be very helpful.
-
Replying to mikeperry:
Here's a test build courtesy of gk: https://people.torproject.org/~mikeperry/3.0a3pre-test/torbrowser-install-3.0-alpha-2_en-US.exe
cca9211633d3e59111e1c411e98e1ea3e1c94111867fb9de62a78dd89f5e16e4 torbrowser-install-3.0-alpha-2_en-US.exe
Sorry, but that build fails to start on Windows XP, with exactly the same error message popping-up as before: "The procedure entry point _vsnprintf_s could not be located in the dynamic link library msvcrt.dll".
Inspection of the binary (firefox.exe) using both Microsoft's Dependency Walker and a binary file editor, shows that it has not linked to msvcr100.dll at all, but still links to msvcrt.dll.
Trac:
Username: James_A -
Ok, commit 8cc3dbc4db0307699d292f6fac7615dae2849779 should try harder not to link against the default msvcrt.dll: https://people.torproject.org/~mikeperry/3.0a3pre-test2/torbrowser-install-3.0-alpha-2_en-US.exe
f94362246a49f6f645b321df254a47242e0f702bfbb177c08578017987b9ffe5 torbrowser-install-3.0-alpha-2_en-US.exe
-
Replying to mikeperry:
Ok, commit 8cc3dbc4db0307699d292f6fac7615dae2849779 should try harder not to link against the default msvcrt.dll: https://people.torproject.org/~mikeperry/3.0a3pre-test2/torbrowser-install-3.0-alpha-2_en-US.exe
No change, same error as before. I cannot find any file that links to msvcr100.dll. They all still link to msvcrt.dll.
The following reference suggests that just adding -lmsvcr100 won't work, a new .spec file needs to be created to stop msvcrt.dll being linked as well (it also suggests that lib*.dll will need to be recompiled as well, to avoid cross-CRT heap and stack clashes). See http://sourceforge.net/mailarchive/message.php?msg_id=31072902
Trac:
Username: James_A -
Okay. Some status report here. The mscvr100 is getting linked (according to the dependency walker). Just put the LDFLAGS before configure is run, put in the proper path to the libmsvcr100 and get rid of -nostdlib (and probably -lgcc; it does no harm to not have it currently). Now, that's the good news. The bad news is that the Tor Browser is now even failing to start on Win7 showing the error message: "missing entry points for _strcmpi in msvcr100.dll". I tried several things to get that issue solved, so far without any luck. The weird thing is that Tor Browser itself does not use this function at all. So, not sure who is actually needing it (my attempts to trace the function calls failed badly). My best bet is a possible interference with the still linked msvcrt (where _strcmpi is indeed still available)). That brings us probably closer to the need of recompiling the lib*.dlls we ship (as well), which is already looming on the horizon. I still hope we can avoid that, though...
-
Ok! GK emailed me a TBB built with a patched mingw that might actually finally solve this issue. I uploaded it here: https://people.torproject.org/~mikeperry/3.0a3-winxp/torbrowser-install-3.0-alpha-3_en-US.exe
sha256sum: 0b5b6b25cadc30a4cdadaf88c79a7c8dde8f6a781bd9cc1f453577053fcadec1
Can any WinXP users try that out and confirm/deny?
-
Ok. The XPCOM error was due to the libxul and other support libraries still being linked against a libgcc and libstdc++ that still used the msvcrt. GK rebuilt those libs against msvcr100.dll and provided me with this binary:
https://people.torproject.org/~mikeperry/3.0a3-winxp2/torbrowser-install-3.0-alpha-3_en-US.exe
6651ad131583fcdbf5276d9f7c3ecc3c4fab94f6416a4548b712669c0bfa28e5 torbrowser-install-3.0-alpha-3_en-US.exe
-
Replying to mikeperry:
Ugh. sqrt2 just reported Win8 works, but WinXP still has the "Couldn't load XPCOM" error.
Apparently msvcrt.dll is still getting pulled in and _controlfp_s is still missing there.
Just to be sure, can somebody confirm that the actual error messages are still the same as in comment 28? "Couldn't load XPCOM" could be caused by a lot of things...
-
Replying to mikeperry:
sqrt2 reports that according to cygwin's 'strings', gkmedias.dll and msvcr100.dll itself try to import that _controlfp_s function.
Well, msvcr100 provides this function (http://msdn.microsoft.com/en-us/library/c9676k6h%28v=vs.100%29.aspx) and gkmedias does indeed need it (https://mxr.mozilla.org/mozilla-esr17/search?string=_controlfp_s). But that should be fine if we link against msvcr100.dll...
-
Replying to gk:
Replying to mikeperry:
Ugh. sqrt2 just reported Win8 works, but WinXP still has the "Couldn't load XPCOM" error.
Apparently msvcrt.dll is still getting pulled in and _controlfp_s is still missing there.
Just to be sure, can somebody confirm that the actual error messages are still the same as in comment 28? "Couldn't load XPCOM" could be caused by a lot of things...
Yep, it is still the same.
Trac:
Username: dope457 -
Okay, I have two ways to proceed further. It turns out, somewhat surprisingly, that gkmedias.dll is indeed still linked against msvcrt (which probably explains the recurring error message). Getting rid of that one too might be an idea... The other way is to write patches for mingw itself. I talked with ktietz from the mingw people about that and this has been done before already (see: mingw-w64-code/trunk/mingw-w64-crt/secapi/). Not sure how much work that is in our case but we could avoid all this building with mscvr100 stuff including the binary blob we ship. Might be worth thinking about.
-
Ok, gk emailed me another build w/ gkmedias linked against only msvcr100.dll: https://people.torproject.org/~mikeperry/3.0a3-winxp3/torbrowser-install-3.0-alpha-3_en-US.exe
24b645fdf30b35d11008b779d04d3cf19dd2fc9d9f7a450281bb757bff00bd68 torbrowser-install-3.0-alpha-3_en-US.exe
dope457, others: Can you try this one out?
-
Replying to mikeperry:
Ok, gk emailed me another build w/ gkmedias linked against only msvcr100.dll: https://people.torproject.org/~mikeperry/3.0a3-winxp3/torbrowser-install-3.0-alpha-3_en-US.exe
24b645fdf30b35d11008b779d04d3cf19dd2fc9d9f7a450281bb757bff00bd68 torbrowser-install-3.0-alpha-3_en-US.exe
dope457, others: Can you try this one out?
I am happy to say...it works! and seems stable.
Trac:
Username: dope457 -
That's good news! Now, how should the patch look like, Mike? We have, roughly speaking, three options:
-
Just take my precompiled mingw runtime libs and adjust the Windows gitian-firefox.yml accordingly.
-
Include the modification and compilation of the mingw runtime libs into the Gitian system and modify gitian-firefox.yml accordingly (we could use then our own compiled mingw to compile everything although this is orthogonal to this bug).
-
Go for the second way outlined in comment 35: write the proper mingw patches and avoid the mess with linking against msvcr100 and above all the additional binary blob we deliver (msvcr100.dll).
(Note: Apart from that there are still some rough edges to fix in my current setup: the nss libs are not linked against msvcr100 yet (for some unknown reason); I need to apply a small patch deep down in Mozilla code (nspr) to avoid the _strcmpi issue in comment 25 which is in fact a mingw issue I am still investigating)
Trac:
Status: new to needs_information -
-
I see no issues with shipping the msvcr100.dll from Microsoft. We can mirror it on people.tp.o and authenticate it by hash. I suspect this will be more expedient than Option 3.
I think we should compile your mingw runtime modifications from source though (option 2). If that means we need to keep a mingw runtime patch around for a little while, I guess we can do that. Did you talk to them about merging your changes?
Wrt the _strcmpi issue, is that specific to Win7 and/or https navigation? People with Win8 reported success with your previous two builds. Not sure if that's just because they didn't try an https site or not, though...
-
Replying to mikeperry:
I see no issues with shipping the msvcr100.dll from Microsoft. We can mirror it on people.tp.o and authenticate it by hash. I suspect this will be more expedient than Option 3.
Okay, sounds good to me.
I think we should compile your mingw runtime modifications from source though (option 2). If that means we need to keep a mingw runtime patch around for a little while, I guess we can do that. Did you talk to them about merging your changes?
Well, I need to patch gcc code to get libgcc linked against mscvr100 and not msvcrt. Thus, I probably need to get a gcc patch merged. I've not looked into that process yet. But that should be doable with the help of mingw folks. The NSPR patch is working around a mingw issue which should be fixed in rev6179. But that is not the case for some reason. I am currently looking into it.
Wrt the _strcmpi issue, is that specific to Win7 and/or https navigation? People with Win8 reported success with your previous two builds. Not sure if that's just because they didn't try an https site or not, though...
No, no. I don't send builds to you that already break on my testing machines. Thus, there should be no _strcmpi issues for downloadable test builds.
I'll prepare a patch along the lines of option 2) then.
Trac:
Status: needs_information to new 
Trac:
Cc: g.koppen@jondos.de, jmaxhaven@yahoo.co.uk to g.koppen@jondos.de, jmaxhaven@yahoo.co.uk, mcs, brade-
Okay, here comes the patch. I wrote the changes for fetch-inputs.sh in a way that it fits to my local build procedure (I had no access to some tpo mirros). Not sure, if that is a thing you'd like to change. Apart from that I am almost happy with it. I plan to work on resolving the _strcmpi issue once and for all but am definitely not interested in developing an upstream patch for the hard-coded msvcrt in the gcc source. I tested the patch locally but could not find any determinism issue. We might want to test with a build done on a different machine, though.
