Opened 6 years ago

Closed 6 years ago

Last modified 5 years ago

#9084 closed defect (fixed)

TBB 3.0a1 crashes on startup on WinXP (missing entry points for _vsnprintf_s in msvcrt.dll)

Reported by: dope457 Owned by: erinn
Priority: High Milestone: Tor: unspecified
Component: Applications/Tor bundles/installation Version: Tor: unspecified
Severity: Keywords: tbb-3.0-stable-blocker, tbb-crash
Cc: g.koppen@…, jmaxhaven@…, mcs, brade Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

firefox.exe - Entry Point Not Found
The procedure entry point snprintf could not be located in the dynamic link library msvcrt.dll

Child Tickets

Attachments (3)

bug.PNG (402.5 KB) - added by dope457 6 years ago.
firefox-crash.JPG (232.5 KB) - added by Logforme 6 years ago.
0001-fix-for-bug-9084-which-allows-running-TBB3.0-on-Wind.patch (43.5 KB) - added by gk 6 years ago.

Download all attachments as: .zip

Change History (56)

Changed 6 years ago by dope457

Attachment: bug.PNG added

Changed 6 years ago by Logforme

Attachment: firefox-crash.JPG added

comment:1 in reply to:  description Changed 6 years ago by Logforme

Replying to dope457:

firefox.exe - Entry Point Not Found
The procedure entry point snprintf could not be located in the dynamic link library msvcrt.dll

Same error for me. Windows XP with MSVCRT.DLL version 7.0.2600.5512 (xpsp.080413-2111)

comment:2 Changed 6 years ago by mikeperry

Keywords: tbb-3.0 tbb-crash added; firefox bundle 3.0 removed

In the V8 code, they apparently undef MINGW_HAS_SECURE_API for this reason. Not sure if that is a valid fix for us or not:
http://www.mail-archive.com/v8-dev@googlegroups.com/msg58589.html
https://qt.gitorious.org/qt/qtjsbackend/commit/7370d84356ed4551ec1d72800666992817901b11/diffs

comment:3 Changed 6 years ago by gk

Cc: g.koppen@… added

I don't think MINGW_HAS_SECURE_API is involved here. sprintf() is not available at all in the runtime lib (only _sprintf()) see e.g.:
http://mingw.5.n7.nabble.com/snprintf-sscanf-mismatch-for-p-td20596.html

comment:4 Changed 6 years ago by gk

mikeperry: Argh... you are probably right. I did not look at the screenshots but only at the description :( (and i meant snprintf() and _snprintf() in my former comment).

comment:5 Changed 6 years ago by mikeperry

Keywords: MikePerry201306 added

gk noted that we may be able to remove the --enable-secure-api from the firefox descriptors and this should fix the issue. I believe he said he would test that (which would be helpful, as I don't have WinXP).

comment:6 Changed 6 years ago by mikeperry

Summary: Crash on startup (torbrowser-install-3.0-alpha-1_en-US.exe)TBB 3.0a1 crashes on startup on WinXP (missing entry points for snprintf in msvcrt.dll)

comment:7 Changed 6 years ago by gk

Status: newneeds_information
Summary: TBB 3.0a1 crashes on startup on WinXP (missing entry points for snprintf in msvcrt.dll)TBB 3.0a1 crashes on startup on WinXP (missing entry points for _vsnprintf_s in msvcrt.dll)

Okay, this move breaks the build. It turns out, there is a bunch of files where _vsnprintf_s is used/needed and there are other secure functions deployed as well (like _vsnwprintf_s). That probably amounts to a lot of patching. :-/ And disabling the secure API for all Windows users is not funny either. Are there any other things we could do here? Asking the affected users to get a newer msvcrt? Or bundling a newer crt with the Tor Browser? Something along the lines: "If you intend to distribute your build to others, you will need to set WIN32_REDIST_DIR=$VCINSTALLDIR\redist\x86\Microsoft.VC80.CRT in your mozconfig to get the Microsoft CRT DLLs packaged along with the application." as mentioned on https://developer.mozilla.org/en-US/docs/Developer_Guide/Build_Instructions/Windows_Prerequisites maybe?

comment:8 Changed 6 years ago by mikeperry

Is this what we need: https://www.microsoft.com/en-us/download/confirmation.aspx?id=3387

Notice that while that page is https, the final exe download is performed over http (as is the case for all of the Microsoft development tool binaries). Go go microsoft security!

Unless we can find a canonical version of that file, it seems we might be at an impasse?

comment:9 in reply to:  8 Changed 6 years ago by gk

Status: needs_informationnew

Replying to mikeperry:

Is this what we need: https://www.microsoft.com/en-us/download/confirmation.aspx?id=3387

Well, sort of. I'd take https://www.microsoft.com/en-us/download/details.aspx?id=5555 as VC 2010 (and thus the respective runtime libs) is the minimum to compile Fx nowadays and get it going. But I don't know yet whether this idea works at all as I've read things like "MinGW does not use Visual C++ Redistributable" and "MinGW uses the Microsoft runtime libraries, distributed with the Windows operating system.".

Unless we can find a canonical version of that file, it seems we might be at an impasse?

Hmm... I gonna ask the Microsoft support if there is something we could do here (maybe we get some hashes of the canonical version, dunno). Sounds like fun. :)

comment:10 Changed 6 years ago by mikeperry

Keywords: MikePerry201307 added; MikePerry201306 removed

comment:11 Changed 6 years ago by James_A

Cc: jmaxhaven@… added

Excuse me coming in here, but it seems to me that this bug is stalled due to a misunderstanding.

On Windows XP Firefox fails to start, because the function _vsnprintf_s() is not present in msvcrt.dll. _vsnprintf_s() and all the *_s() functions are present in the Visual Studio runtimes, because they are a Microsoft addition as "security enhanced" versions of the functions without the _s suffix. As far as I can tell they have also been backported to msvcrt.dll on Vista/7 only. This is a Windows system file and you may not take the file from Vista/7 and copy it to XP (it won't work even if you did).

Accordingly, the way forward looks like one of the following:
(1) Compile with Visual Studio 2010 or VS 2012
(2) Target only Vista/7 upwards (effectively a WONTFIX for XP)
(3) Write your own *_s() functions. Required are 5 variations of *printf_s(), _wcslwr_s(), _wfopen_s(), fopen_s(), rand_s() and _controlfp_s() (10 in all). The first 9 are called from xul.dll and the 10th from gkmedias.dll. I have walked all the 3.0a1 package (.exe and .dll files) through Dependency Walker to find what is missing. This option looks like a major effort, except that others have already done it e.g. the WINE project, for which the source code is available under GNU Lesser GPL 2.1.
(4) Reconfigure MinGW to link to the equivalent of the Visual Studio runtimes e.g. msvcr90.dll (which is the runtime for VS 2008). I don't know if this works, not having tried it myself, but others have suggested that the options:

gcc foo.c -nostdlib -lmsvcr90 -lgcc ...

might work.

Incidentally, undef-ing MINGW_HAS_SECURE_API (as done in Chromium V8) will not work unless you follow option (3) because V8 provides its own *_s() functions in the file platform-win32.cc.

comment:12 Changed 6 years ago by mikeperry

Ok, the exe from https://www.microsoft.com/en-us/download/details.aspx?id=5555 has sha256sum 8162b2d665ca52884507ede19549e99939ce4ea4a638c537fa653539819138c8 from a few Tor exits.

I ran it in wine, and it installed this:
111ba3befa7b5fb1e736e34aa7130ab148122c16177bcbc686c7f87b6b4ea694 .wine/drive_c/windows/winsxs/x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_deadbeef/msvcr90.dll

Is that what we want? Does anyone have a different hash for msvcr90.dll?

comment:13 in reply to:  12 Changed 6 years ago by gk

Replying to mikeperry:

Ok, the exe from https://www.microsoft.com/en-us/download/details.aspx?id=5555 has sha256sum 8162b2d665ca52884507ede19549e99939ce4ea4a638c537fa653539819138c8 from a few Tor exits.

I can confirm this with Tor-independent means. And the .exe is signed by Microsoft which is much better than nothing.

I ran it in wine, and it installed this:
111ba3befa7b5fb1e736e34aa7130ab148122c16177bcbc686c7f87b6b4ea694 .wine/drive_c/windows/winsxs/x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_deadbeef/msvcr90.dll

Is that what we want?

No, actually not. That one is a runtime lib for VS 2008. Weird, that it is included into the 2010 package at all. I'd like to have the *100.dlls. That said, we could grab them from Mozilla as they are shipped with Firefox. Maybe Mozilla has found a good way to verify that the dlls are untampered. Anyway, I failed badly trying to get into contact with the responsible people at Microsoft (surprise!).

comment:14 Changed 6 years ago by mikeperry

Ah, ok. Here's my hash for msvcr100.dll.
MSVCR100_HASH=1221a09484964a6f38af5e34ee292b9afefccb3dc6e55435fd3aaf7c235d9067

With the help of #build on irc.mozilla.org, I think I've got it linking in. I will commit those changes shortly.

comment:15 Changed 6 years ago by mikeperry

Ok, builders/tor-browser-builder commit 3f83d43a323ba18479894f5c12e4c99ff80882f7 should build a Windows Bundle that links against and includes that DLL.

If anyone has a WinXP vm that would like me to put a test build somewhere to verify that it no longer crashes, I can do that later this week.

comment:16 in reply to:  14 Changed 6 years ago by James_A

Replying to mikeperry:

Ah, ok. Here's my hash for msvcr100.dll.
MSVCR100_HASH=1221a09484964a6f38af5e34ee292b9afefccb3dc6e55435fd3aaf7c235d9067

With the help of #build on irc.mozilla.org, I think I've got it linking in. I will commit those changes shortly.

I confrm that a copy of msvcr100.dll (32-bit) RTM with a verified correct digital signature (signed by Microsoft Code Signing PCA) has the SHA256 hash of 1221a09484964a6f38af5e34ee292b9afefccb3dc6e55435fd3aaf7c235d9067

If you post a test build somewhere I can verify it.

comment:17 in reply to:  15 Changed 6 years ago by dope457

Replying to mikeperry:

Ok, builders/tor-browser-builder commit 3f83d43a323ba18479894f5c12e4c99ff80882f7 should build a Windows Bundle that links against and includes that DLL.

If anyone has a WinXP vm that would like me to put a test build somewhere to verify that it no longer crashes, I can do that later this week.

Yeah, I can also verify it.

comment:18 Changed 6 years ago by mikeperry

Here's a test build courtesy of gk: https://people.torproject.org/~mikeperry/3.0a3pre-test/torbrowser-install-3.0-alpha-2_en-US.exe

cca9211633d3e59111e1c411e98e1ea3e1c94111867fb9de62a78dd89f5e16e4 torbrowser-install-3.0-alpha-2_en-US.exe

gk also noticed that Firefox also ships with msvcp100.dll, so this build may still crash, depending on if we actually need that DLL for mingw-w64 builds too, or not. Feedback on that would be very helpful.

comment:19 in reply to:  18 Changed 6 years ago by James_A

Replying to mikeperry:

Here's a test build courtesy of gk: https://people.torproject.org/~mikeperry/3.0a3pre-test/torbrowser-install-3.0-alpha-2_en-US.exe

cca9211633d3e59111e1c411e98e1ea3e1c94111867fb9de62a78dd89f5e16e4 torbrowser-install-3.0-alpha-2_en-US.exe

Sorry, but that build fails to start on Windows XP, with exactly the same error message popping-up as before: "The procedure entry point _vsnprintf_s could not be located in the dynamic link library msvcrt.dll".

Inspection of the binary (firefox.exe) using both Microsoft's Dependency Walker and a binary file editor, shows that it has not linked to msvcr100.dll at all, but still links to msvcrt.dll.

comment:20 Changed 6 years ago by dope457

Can confirm same result as James_A.

comment:21 Changed 6 years ago by mikeperry

Ok, commit 8cc3dbc4db0307699d292f6fac7615dae2849779 should try harder not to link against the default msvcrt.dll:
https://people.torproject.org/~mikeperry/3.0a3pre-test2/torbrowser-install-3.0-alpha-2_en-US.exe

f94362246a49f6f645b321df254a47242e0f702bfbb177c08578017987b9ffe5 torbrowser-install-3.0-alpha-2_en-US.exe

comment:22 in reply to:  21 Changed 6 years ago by James_A

Replying to mikeperry:

Ok, commit 8cc3dbc4db0307699d292f6fac7615dae2849779 should try harder not to link against the default msvcrt.dll:
https://people.torproject.org/~mikeperry/3.0a3pre-test2/torbrowser-install-3.0-alpha-2_en-US.exe

No change, same error as before. I cannot find any file that links to msvcr100.dll. They all still link to msvcrt.dll.

The following reference suggests that just adding -lmsvcr100 won't work, a new .spec file needs to be created to stop msvcrt.dll being linked as well (it also suggests that lib*.dll will need to be recompiled as well, to avoid cross-CRT heap and stack clashes). See http://sourceforge.net/mailarchive/message.php?msg_id=31072902

comment:23 Changed 6 years ago by mikeperry

Keywords: MikePerry201308 added; MikePerry201307 removed

comment:24 Changed 6 years ago by mikeperry

Keywords: tbb-3.0-stable-blocker added; tbb-3.0 removed

comment:25 Changed 6 years ago by gk

Okay. Some status report here. The mscvr100 is getting linked (according to the dependency walker). Just put the LDFLAGS _before_ configure is run, put in the proper path to the libmsvcr100 and get rid of -nostdlib (and probably -lgcc; it does no harm to not have it currently). Now, that's the good news. The bad news is that the Tor Browser is now even failing to start on Win7 showing the error message: "missing entry points for _strcmpi in msvcr100.dll". I tried several things to get that issue solved, so far without any luck. The weird thing is that Tor Browser itself does not use this function at all. So, not sure who is actually needing it (my attempts to trace the function calls failed badly). My best bet is a possible interference with the still linked msvcrt (where _strcmpi is indeed still available)). That brings us probably closer to the need of recompiling the lib*.dlls we ship (as well), which is already looming on the horizon. I still hope we can avoid that, though...

comment:26 Changed 6 years ago by mikeperry

Keywords: MikePerry201308 removed

comment:27 Changed 6 years ago by mikeperry

Ok! GK emailed me a TBB built with a patched mingw that *might* actually finally solve this issue. I uploaded it here: https://people.torproject.org/~mikeperry/3.0a3-winxp/torbrowser-install-3.0-alpha-3_en-US.exe

sha256sum: 0b5b6b25cadc30a4cdadaf88c79a7c8dde8f6a781bd9cc1f453577053fcadec1

Can any WinXP users try that out and confirm/deny?

comment:28 Changed 6 years ago by dope457

It gives me this error 3 times:

"Missing entry points for _controlfp_s in msvcrt.dll"

and then:

"Couldn't load XPCOM."

comment:29 Changed 6 years ago by mikeperry

Ok. The XPCOM error was due to the libxul and other support libraries still being linked against a libgcc and libstdc++ that still used the msvcrt. GK rebuilt those libs against msvcr100.dll and provided me with this binary:

https://people.torproject.org/~mikeperry/3.0a3-winxp2/torbrowser-install-3.0-alpha-3_en-US.exe

6651ad131583fcdbf5276d9f7c3ecc3c4fab94f6416a4548b712669c0bfa28e5 torbrowser-install-3.0-alpha-3_en-US.exe

comment:30 Changed 6 years ago by mikeperry

Ugh. sqrt2 just reported Win8 works, but WinXP still has the "Couldn't load XPCOM" error.

Apparently msvcrt.dll is still getting pulled in and _controlfp_s is still missing there.

comment:31 Changed 6 years ago by mikeperry

sqrt2 reports that according to cygwin's 'strings', gkmedias.dll and msvcr100.dll itself try to import that _controlfp_s function.

comment:32 in reply to:  30 ; Changed 6 years ago by gk

Replying to mikeperry:

Ugh. sqrt2 just reported Win8 works, but WinXP still has the "Couldn't load XPCOM" error.

Apparently msvcrt.dll is still getting pulled in and _controlfp_s is still missing there.

Just to be sure, can somebody confirm that the actual error messages are still the same as in comment 28? "Couldn't load XPCOM" could be caused by a lot of things...

comment:33 in reply to:  31 Changed 6 years ago by gk

Replying to mikeperry:

sqrt2 reports that according to cygwin's 'strings', gkmedias.dll and msvcr100.dll itself try to import that _controlfp_s function.

Well, msvcr100 provides this function (http://msdn.microsoft.com/en-us/library/c9676k6h%28v=vs.100%29.aspx) and gkmedias does indeed need it (https://mxr.mozilla.org/mozilla-esr17/search?string=_controlfp_s). But that should be fine if we link against msvcr100.dll...

comment:34 in reply to:  32 Changed 6 years ago by dope457

Replying to gk:

Replying to mikeperry:

Ugh. sqrt2 just reported Win8 works, but WinXP still has the "Couldn't load XPCOM" error.

Apparently msvcrt.dll is still getting pulled in and _controlfp_s is still missing there.

Just to be sure, can somebody confirm that the actual error messages are still the same as in comment 28? "Couldn't load XPCOM" could be caused by a lot of things...

Yep, it is still the same.

comment:35 Changed 6 years ago by gk

Okay, I have two ways to proceed further. It turns out, somewhat surprisingly, that gkmedias.dll is indeed still linked against msvcrt (which probably explains the recurring error message). Getting rid of that one too might be an idea... The other way is to write patches for mingw itself. I talked with ktietz from the mingw people about that and this has been done before already (see: mingw-w64-code/trunk/mingw-w64-crt/secapi/). Not sure how much work that is in our case but we could avoid all this building with mscvr100 stuff including the binary blob we ship. Might be worth thinking about.

comment:36 Changed 6 years ago by mikeperry

Ok, gk emailed me another build w/ gkmedias linked against only msvcr100.dll:
https://people.torproject.org/~mikeperry/3.0a3-winxp3/torbrowser-install-3.0-alpha-3_en-US.exe

24b645fdf30b35d11008b779d04d3cf19dd2fc9d9f7a450281bb757bff00bd68 torbrowser-install-3.0-alpha-3_en-US.exe

dope457, others: Can you try this one out?

comment:37 in reply to:  36 Changed 6 years ago by dope457

Replying to mikeperry:

Ok, gk emailed me another build w/ gkmedias linked against only msvcr100.dll:
https://people.torproject.org/~mikeperry/3.0a3-winxp3/torbrowser-install-3.0-alpha-3_en-US.exe

24b645fdf30b35d11008b779d04d3cf19dd2fc9d9f7a450281bb757bff00bd68 torbrowser-install-3.0-alpha-3_en-US.exe

dope457, others: Can you try this one out?

I am happy to say...it works! and seems stable.

comment:38 Changed 6 years ago by gk

Status: newneeds_information

That's good news! Now, how should the patch look like, Mike? We have, roughly speaking, three options:

1) Just take my precompiled mingw runtime libs and adjust the Windows gitian-firefox.yml accordingly.

2) Include the modification and compilation of the mingw runtime libs into the Gitian system and modify gitian-firefox.yml accordingly (we could use then our own compiled mingw to compile everything although this is orthogonal to this bug).

3) Go for the second way outlined in comment 35: write the proper mingw patches and avoid the mess with linking against msvcr100 and above all the additional binary blob we deliver (msvcr100.dll).

(Note: Apart from that there are still some rough edges to fix in my current setup: the nss libs are not linked against msvcr100 yet (for some unknown reason); I need to apply a small patch deep down in Mozilla code (nspr) to avoid the _strcmpi issue in comment 25 which is in fact a mingw issue I am still investigating)

Last edited 6 years ago by gk (previous) (diff)

comment:39 Changed 6 years ago by mikeperry

I see no issues with shipping the msvcr100.dll from Microsoft. We can mirror it on people.tp.o and authenticate it by hash. I suspect this will be more expedient than Option 3.

I think we should compile your mingw runtime modifications from source though (option 2). If that means we need to keep a mingw runtime patch around for a little while, I guess we can do that. Did you talk to them about merging your changes?

Wrt the _strcmpi issue, is that specific to Win7 and/or https navigation? People with Win8 reported success with your previous two builds. Not sure if that's just because they didn't try an https site or not, though...

comment:40 in reply to:  39 Changed 6 years ago by gk

Status: needs_informationnew

Replying to mikeperry:

I see no issues with shipping the msvcr100.dll from Microsoft. We can mirror it on people.tp.o and authenticate it by hash. I suspect this will be more expedient than Option 3.

Okay, sounds good to me.

I think we should compile your mingw runtime modifications from source though (option 2). If that means we need to keep a mingw runtime patch around for a little while, I guess we can do that. Did you talk to them about merging your changes?

Well, I need to patch gcc code to get libgcc linked against mscvr100 and not msvcrt. Thus, I probably need to get a gcc patch merged. I've not looked into that process yet. But that should be doable with the help of mingw folks. The NSPR patch is working around a mingw issue which should be fixed in rev6179. But that is not the case for some reason. I am currently looking into it.

Wrt the _strcmpi issue, is that specific to Win7 and/or https navigation? People with Win8 reported success with your previous two builds. Not sure if that's just because they didn't try an https site or not, though...

No, no. I don't send builds to you that already break on my testing machines. Thus, there should be no _strcmpi issues for downloadable test builds.

I'll prepare a patch along the lines of option 2) then.

comment:41 Changed 6 years ago by mcs

Cc: mcs brade added

comment:42 Changed 6 years ago by gk

Okay, here comes the patch. I wrote the changes for fetch-inputs.sh in a way that it fits to my local build procedure (I had no access to some tpo mirros). Not sure, if that is a thing you'd like to change. Apart from that I am almost happy with it. I plan to work on resolving the _strcmpi issue once and for all but am definitely not interested in developing an upstream patch for the hard-coded msvcrt in the gcc source. I tested the patch locally but could not find any determinism issue. We might want to test with a build done on a different machine, though.

comment:43 Changed 6 years ago by gk

Status: newneeds_review

comment:44 Changed 6 years ago by mikeperry

Hrmm. Should we maybe patch the nspr use of strcmpi directly in git rather than from the descriptor? Or do we expect that we'll be able to drop that sed line very soon with later mingw releases?

Also, why did you choose gcc-4.6.3 and binutils 2.22? Is that just because that's what Ubuntu uses, or was there another recommendation from the mingw people?

comment:45 in reply to:  44 Changed 6 years ago by gk

Replying to mikeperry:

Hrmm. Should we maybe patch the nspr use of strcmpi directly in git rather than from the descriptor? Or do we expect that we'll be able to drop that sed line very soon with later mingw releases?

The latter, although it is not a mingw issue according to one of the devs but a problem on our side which I think we should figure out (I plan to do so at least).

Also, why did you choose gcc-4.6.3 and binutils 2.22? Is that just because that's what Ubuntu uses, or was there another recommendation from the mingw people?

The former. I thought it would be a good conservative approach to not introduce more issues than necessary although in theory the latest versions of both should do it as well.

comment:46 Changed 6 years ago by mikeperry

Resolution: fixed
Status: needs_reviewclosed

Ok, I merged this! I'm going to merge a few other things and then try a Linux build out for a while, and then we can hopefully tag 3.0beta1!

Thanks a lot for this gk!

comment:47 Changed 6 years ago by mikeperry

Ok, here's an official candidate build of 3.0beta1: https://people.torproject.org/~mikeperry/3.0b1-winxp/torbrowser-install-3.0-beta-1_en-US.exe

92a8b1dcca06314ea876a684cb2b4833d98588327c03c761803280afecb11c34 torbrowser-install-3.0-beta-1_en-US.exe

Can some WinXP people verify the patches were correctly merged and that everything still works?

comment:48 Changed 6 years ago by mikeperry

Alright, apparently there is an issue from #9114 that is present in that build ("Your Firefox profile cannot be loaded. It may be missing or inaccessible").. Stay tuned for another one later.

comment:49 Changed 6 years ago by mikeperry

Ok, this one should now work:
https://people.torproject.org/~mikeperry/3.0b1-winxp2/torbrowser-install-3.0-beta-1_en-US.exe
b2e8cf8f42890eee521550e9a0c829ca467deaaf9e026117abf5f84810532a3e torbrowser-install-3.0-beta-1_en-US.exe

Can anyone on WinXP confirm/deny?

Last edited 6 years ago by mikeperry (previous) (diff)

comment:50 Changed 6 years ago by dope457

Works great.
Though browser says it is out of date.

comment:51 Changed 6 years ago by mcs

Works for me on an old WinXP Dell laptop.

comment:52 Changed 6 years ago by cypherpunks

Works on an old XP desktop

comment:53 Changed 5 years ago by BugMagnet

works for me tyvm

v3a4 failed, b1 runs (with non-exit relay in background and regular ff open) - nice

Note: See TracTickets for help on using tickets.