Opened 3 years ago

Closed 5 weeks ago

Last modified 5 weeks ago

#18935 closed defect (fixed)

MS bugs - Danger! Windows updates break everything on their way!

Reported by: bugzilla Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Critical Keywords: tbb-crash
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

READ THIS BEFORE USING TOR BROWSER ON WINDOWS!

With the April's 2016 patch Tuesday Microsoft has shown to everybody what grade of software it's going to supply from now on:

KB3147071 - Welcome new "fast-ring"/"pre-beta" quality (see #18171), now for production servers ;) Discussion: https://www.dslreports.com/forum/r30702359-MS-patch-installing-old-vulnerable-version-of-ntdll-dll Proof from MS filelist:

x64 Windows 8.1 and Windows Server 2012 R2
File name,File version,File size,Date,Time,Platform,Service branch,
Ntdll.dll,"6.3.9600.18194","1,737,080","13-Jan-2016","21:26","x64","Not applicable",

And together with the

KB3146706 - Welcome BSOD 0x0000006B (http://www.windowsnewscenter.com/2016/04/26/is-microsoft-using-security-patch-kb-3146706-to-break-pirate-copies-of/)
they present you:

Problem with application hangs due to EMET 5.5 EAF mitigation after Windows 7 April 2016 updates
https://social.technet.microsoft.com/Forums/en-US/7a681da9-d1d3-4566-a13d-55af0de0f2a5/problem-with-application-hangs-due-to-emet-55-eaf-mitigation-after-windows-7-april-2016-updates?forum=emet

KB3149090 - can be PITA too (https://www.reddit.com/r/sysadmin/comments/4ewule/windows_32bit_emet_kb3146706_issue/)

KB3145739 - can prevent you from installing all the April's crap until you install it manually.

And even Windows Update can daunt you to stop installing April's crap by going to very long several days checking for it with full CPU load... (can be fixed as the previous item)

Hangs and crashes of Tor Browser that are probably related to this crap are listed in comments:

Child Tickets

Change History (32)

comment:1 Changed 3 years ago by bugzilla

Faulting application name: firefox.exe, version: 38.7.1.0
Faulting module name: ntdll.dll, version: 6.1.7601.19160
Exception code: 0x80000004
Fault offset: 0x0005aec5

Also with IE:

Faulting application name: iexplore.exe, version: 11.0.9600.18231, time stamp: 0x56b8edd6
Faulting module name: ntdll.dll, version: 6.1.7601.19160, time stamp: 0x56bcd4d2
Exception code: 0x80000004
Fault offset: 0x0005aec5

and

Faulting application name: iexplore.exe, version: 11.0.9600.18231, time stamp: 0x56b8edd6
Faulting module name: KERNELBASE.dll, version: 6.1.7601.19135, time stamp: 0x56a1c680
Exception code: 0x80000004
Fault offset: 0x000075fd

comment:2 Changed 3 years ago by bugzilla

New versions show new crashes:

Faulting application name: firefox.exe, version: 38.8.0.0, time stamp: 0x00000000
Faulting module name: ntdll.dll, version: 6.1.7601.19160, time stamp: 0x56bcd4d2
Exception code: 0xc00000fd
Fault offset: 0x0005d2fd

and

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: kernel32.dll, version: 6.1.7601.19135, time stamp: 0x56a1c67f
Exception code: 0xc00000fd
Fault offset: 0x0004cf52

comment:3 Changed 3 years ago by bugzilla

Tor Browser hangs even when EMET service is disabled and in "Audit only" mode (log only), despite that M$ claims that this mode doesn't interfere with apps!

comment:4 Changed 3 years ago by bugzilla

There are so many crashes in 6.0a5 and all this MS crap that this ticket becomes recycle bin for crash reports - pick any to a separate ticket if you like :)

comment:5 Changed 3 years ago by bugzilla

Check Server/Errors in Browser Console and

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: WINMM.DLL, version: 6.1.7601.17514, time stamp: 0x4ce7ba42
Exception code: 0xc00000fd
Fault offset: 0x00003276

comment:6 Changed 3 years ago by bugzilla

Security suite intercepts at "random" situations xul.dll calls to physical memory (WHY IS IT DOING IT?!!) and if denied:

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: xul.dll, version: 45.1.0.0, time stamp: 0x00000000
Exception code: 0xc00000fd
Fault offset: 0x0258b0b2

or

Faulting application name: firefox.exe, version: 38.7.1.0, time stamp: 0x00000000
Faulting module name: xul.dll, version: 38.7.1.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00041416

comment:7 Changed 3 years ago by bugzilla

On Win 7 with all updates, no EMET:

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: nss3.dll, version: 3.21.1.0, time stamp: 0x00000000
Exception code: 0xc00000fd
Fault offset: 0x000b1c83

STR: https://trac.torproject.org/projects/tor/ticket/18937#comment:11

comment:8 Changed 3 years ago by bugzilla

On Win 7 with all updates, no EMET:

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: kernel32.dll, version: 6.1.7601.23392, time stamp: 0x56eb2fb8
Exception code: 0xc00000fd
Fault offset: 0x0004cf52

STR: https://trac.torproject.org/projects/tor/ticket/18937#comment:14
Differs from comment 2 by kernel32's version.

Last edited 3 years ago by bugzilla (previous) (diff)

comment:9 Changed 3 years ago by bugzilla

Well, fully updated system with EAF disabled brought us to:

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: xul.dll, version: 45.1.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0003fb34

Seems opening a lot of Trac tickets in the background is a good stress-test.

UPD:

Faulting application name: firefox.exe, version: 38.8.0.0, time stamp: 0x00000000
Faulting module name: xul.dll, version: 38.8.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00041432
Last edited 3 years ago by bugzilla (previous) (diff)

comment:10 Changed 3 years ago by bugzilla

And

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: kernel32.dll, version: 6.1.7601.23392, time stamp: 0x56eb2fb8
Exception code: 0xc00000fd
Fault offset: 0x0004ce8d

comment:11 in reply to:  9 ; Changed 3 years ago by gk

Status: newneeds_information

Replying to bugzilla:

Well, fully updated system with EAF disabled brought us to:

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: xul.dll, version: 45.1.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0003fb34

Seems opening a lot of Trac tickets in the background is a good stress-test.

Seems to work for me on an up-to-date Win 7 system. I bookmarked 20 trac tickets and opened them at the same time. Everything still worked. Do you have more detailed steps to reproduce the crashes?

At a more generic level: What are we supposed to do with this ticket and the numerous issues you mentioned? How is this still actionable?

comment:12 in reply to:  11 ; Changed 3 years ago by bugzilla

Replying to gk:

Replying to bugzilla:

Well, fully updated system with EAF disabled brought us to:

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: xul.dll, version: 45.1.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0003fb34

Seems opening a lot of Trac tickets in the background is a good stress-test.

Seems to work for me on an up-to-date Win 7 system. I bookmarked 20 trac tickets and opened them at the same time. Everything still worked. Do you have more detailed steps to reproduce the crashes?

At a more generic level: What are we supposed to do with this ticket and the numerous issues you mentioned? How is this still actionable?

You installed the latest EMET on an up-to-date Win 7, selected Max Sec, copied the ruleset for FF to TB's firefox.exe, disabled EAF in it, selected Audit Mode (or welcome SimExecFlow https://trac.torproject.org/projects/tor/ticket/13893#comment:56), and restarted the system. Then you did STR, right?

At generic: this ticket is a general warning for users of TBB on Windows that shows what is happening in various situations if they do or don't install or install partially the latest updates from MS, install EMET, etc. Also it is a recycle bin for all TBB crashes on Windows that users can report into. And if somebody has a free time and willing to investigate some he is welcome.

TBB Team: it's not worth effort to dig the MS crap.
0xc0000005 = ACCESS_VIOLATION or SIGSEGV or GPF - it could be anything...
0xc00000fd = STACK_OVERFLOW - it seems to be the thing you've discovered on hardened...
And your CYGWIN WIN 6.3 x64 test machine is affected with vulnerable ntdll.dll
That's all.

comment:13 in reply to:  12 Changed 3 years ago by gk

Resolution: duplicate
Status: needs_informationclosed

Replying to bugzilla:

Replying to gk:

Replying to bugzilla:

Well, fully updated system with EAF disabled brought us to:

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: xul.dll, version: 45.1.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0003fb34

Seems opening a lot of Trac tickets in the background is a good stress-test.

Seems to work for me on an up-to-date Win 7 system. I bookmarked 20 trac tickets and opened them at the same time. Everything still worked. Do you have more detailed steps to reproduce the crashes?

At a more generic level: What are we supposed to do with this ticket and the numerous issues you mentioned? How is this still actionable?

You installed the latest EMET on an up-to-date Win 7, selected Max Sec, copied the ruleset for FF to TB's firefox.exe, disabled EAF in it, selected Audit Mode (or welcome SimExecFlow https://trac.torproject.org/projects/tor/ticket/13893#comment:56), and restarted the system. Then you did STR, right?

Actually, no. I assumed as done in other comments in this ticket this was without EMET. So, I am marking this a duplicate of the EMET ticket (#13893). Please, file separate bugs for the other crashes making sure they are no duplicate. Having steps to reproduce them would rock.

comment:14 Changed 3 years ago by bugzilla

This ticket was filed as a tracking bug for all issues on Windows, because of its crappy new updates. You need both 32 and 64 bit Windows test machines to discover them. They appear so often that it's boring to re-login and write comments... And it's not proved that it's because of EMET. So, your decision looks strangely...
Do you prefer a new ticket for each crash? And how can it be, in general, that stack overflows have STR?

comment:15 Changed 3 years ago by gk

I am not sure I understand the latter question. How can you be sure you fixed a bug without having steps to reproduce it? I prefer a new ticket for each crash which is not a duplicate and not caused by the same underlying problem, yes. Otherwise I can't work on it properly, like investigating and closing the case.

comment:16 Changed 3 years ago by bugzilla

Huh, of course, you can be sure you've fixed a bug without STR if you know what you're doing. Just reading the code and understanding the processes are enough to search, localize and fix the bugs. E.g. see https://mxr.mozilla.org/mozilla-esr45/source/configure.in#5448. Do you see that -DPIC is missing? So much time passed, so much testing passed... Where are the tickets? And what kind should they be? Just reports of random crashes in non-related tickets, and you've written that it's your preferred option... OK, if so, please, post here how you can do things like that, based on this example: how you do search in Bugzilla (criteria), how you detect that found tickets are caused by -DPIC underlying problem, and, of course, your STR for this issue! (And who told you that every bug had STR...)

comment:17 Changed 3 years ago by bugzilla

Oh, and you haven't answered the main question: why did you close this ticket about MS & related bugs as a dupe of TBB's vulnerability one?
Bug in comment:8 is still actual (and, yes, without EMET), but what about STR? It happens sometimes with STR in comment:7, sometimes even when JS off, and even when Trac is updating the writing of this comment! What to do with it?

comment:18 in reply to:  16 Changed 3 years ago by gk

Replying to bugzilla:

Huh, of course, you can be sure you've fixed a bug without STR if you know what you're doing. Just reading the code and understanding the processes are enough to search, localize and fix the bugs. E.g. see https://mxr.mozilla.org/mozilla-esr45/source/configure.in#5448. Do you see that -DPIC is missing? So much time passed, so much testing passed... Where are the tickets? And what kind should they be? Just reports of random crashes in non-related tickets, and you've written that it's your preferred option... OK, if so, please, post here how you can do things like that, based on this example: how you do search in Bugzilla (criteria), how you detect that found tickets are caused by -DPIC underlying problem, and, of course, your STR for this issue! (And who told you that every bug had STR...)

I think you misunderstood me. I was talking about STR in this context: crashes on closed source software which oneself cannot reproduce. Even if I think I have a fix and give it to a user to test she/he needs to have some steps to say, "Yes, it works" or "No, it does not work".

I closed this ticket as duplicate because this seemed to be mainly about EMET.

comment:19 in reply to:  11 Changed 3 years ago by bugzilla

Replying to gk:

Replying to bugzilla:

Well, fully updated system with EAF disabled brought us to:

Faulting application name: firefox.exe, version: 45.1.0.0, time stamp: 0x00000000
Faulting module name: xul.dll, version: 45.1.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0003fb34

Seems opening a lot of Trac tickets in the background is a good stress-test.

Seems to work for me on an up-to-date Win 7 system. I bookmarked 20 trac tickets and opened them at the same time. Everything still worked. Do you have more detailed steps to reproduce the crashes?

UPD: If you're concerned about this particular bug, the news is that it still occurs with May updates (Win7 32-bit) and without EMET. (esr38.8-based TBB is affected too)
STR: nothing definite. Bug was invoked by opening the timeline on Trac, enabling Temporarily allow all this page on High and opening a lot of tickets in the background. (From some moment it becomes visible that GUI corruption begins - this is the mark that opening a few new webpages will lead to the bug or STACK_OVERFLOW.)

Last edited 3 years ago by bugzilla (previous) (diff)

comment:20 Changed 3 years ago by bugzilla

Well, MS started to fix its bugs in a specific way. It's called Monthly update rollup.

This update includes quality improvements. No new operating system features are being introduced and no new security updates are included.

It is optional! But

We recommend that you apply this update rollup as part of your regular maintenance routines.

It was treated as a joke at first, but it still continues...

So, the needed updates are:
KB3161647 Windows Update Client for Windows 7 and Windows Server 2008 R2: June 2016
KB3163644 Microsoft Office 2010 doesn't start when EMET is enabled in Windows 7 or Windows Server 2008 R2

This issue occurs when the Enhanced Mitigation Experience Toolkit (EMET) is enabled and security update 3146706 or convenience rollup update 3125574 is installed.

Have you guessed that it's about Windoze (not EMET) bug with EAF? :)
Don't try to download them because, yes, they are in the first Monthly update rollup:
June 2016 update rollup for Windows 7 SP1 and Windows Server 2008 R2 SP1
But don't try to find it in your Optional updates because update rollups are cumulative, and this one was superseded by:
July 2016 update rollup for Windows 7 SP1 and Windows Server 2008 R2 SP1
But to find out info about it you need https://support.microsoft.com/en-us/help/22801/windows-7-and-windows-server-2008-r2-update-history
And, surprise, update rollups are no longer cumulative, so don't use
August 2016 update rollup for Windows 7 SP1 and Windows Server 2008 R2 SP1

This update doesn't replace any previously released update. 

Funny? No? Don't worry, welcome Monthly Rollups!
https://arstechnica.com/business/2016/08/windows-7-8-1-moving-to-windows-10s-cumulative-update-model/?comments=1&start=200

October 2016's Patch Tuesday will see the release of the first Monthly Rollup for Windows 7 and 8.1.

and, yes,

Subsequent months will have new Monthly Rollups, and these will be cumulative, incorporating the content of all previous Monthly Rollups.

but

Initially, these Monthly Rollups will only contain new patches released from October 2016 onward.

but

Over the next year, Microsoft says that it will extend them to go back in time, slowly integrated all the patches released since the last "baseline."

So MS will get you sooner or later ;)

comment:21 in reply to:  20 Changed 3 years ago by bugzilla

So, as MS stated, July 2016 update rollup for Windows 7 SP1 and Windows Server 2008 R2 SP1 is enough to fix issues with EAF, but

Faulting application name: firefox.exe, version: 45.3.0.0, time stamp: 0x00000000
Faulting module name: KERNELBASE.dll, version: 6.1.7601.23392, time stamp: 0x56eb2fb9
Exception code: 0x80000004
Fault offset: 0x00006840

TBB 6.5a2 rare startup crash (EMET 5.5 max).

comment:22 Changed 3 years ago by bugzilla

Mitigation caveat:

EAF mitigation should not be applied to: programs and libraries protected that use packers or compressors, DRM or software with anti-debugging code, debuggers, and security software such as antivirus, sandbox, firewalls, etc.

EMET.dll (EMET SHIM) is added to every process even when EMET service is not started.
And similar startup crash:

Faulting application name: firefox.exe, version: 45.3.0.0, time stamp: 0x00000000
Faulting module name: kernel32.dll, version: 6.1.7601.23392, time stamp: 0x56eb2fb8
Exception code: 0x80000004
Fault offset: 0x0004c4d3

comment:23 in reply to:  20 Changed 3 years ago by bugzilla

Replying to bugzilla:

So MS will get you sooner or later ;)

Sooner! MS moved their patches to Recommended and broke everything as it had been before!!!
​July 2016 update rollup installed as Optional was replaced by Recommended too...

comment:24 Changed 3 years ago by bugzilla

It is called by M$:

Notice
The July 2016 update rollup for Windows 7 Service Pack 1 (SP1) and Windows Server 2008 R2 SP1 was re-released on September 13, 2016 to address an issue in one of the included updates and to improve the overall reliability of the update rollup.

BUT WELCOME KB3175024!
https://support.microsoft.com/en-us/kb/3175024

Known issue 2
When you try to start an application, the application freezes very early in the process and does not completely start.

Cause
This issue occurs because the Export Address table Filtering (EAF) mitigation is active on the application.

comment:25 Changed 20 months ago by cypherpunks

Tor Browser crashed on Windows 7 with 0xc00000fd = STACK_OVERFLOW, mentioned in this ticket only:

<EventData>

<Data>firefox.exe</Data>
<Data>52.6.0.6607</Data>
<Data>00000000</Data>
<Data>kernel32.dll</Data>
<Data>6.1.7601.24000</Data>
<Data>5a4996cc</Data>
<Data>c00000fd</Data>
<Data>0004719b</Data>

It happened when I had just switched a tab.

comment:26 Changed 18 months ago by cypherpunks

Fucking Microsoft! It made Firefox more than 10x times slower with its recent messy patching efforts!
Starting this year, you need 3rd-party support to figure out what the hell happens with Windows Updates, e.g. https://support.symantec.com/en_US/article.INFO4782.html
Because doing nothing, you can end up with no updates at all! One of the official statements says: "In cases where customers can't install or run antivirus software, Microsoft recommends manually setting the registry key as described below in order to receive the latest Windows security updates."
Every month brings new crappy updates to broader categories of systems https://www.bleepingcomputer.com/news/security/microsoft-removes-antivirus-registry-key-check-for-windows-10-users/
This makes determining when it will affect you almost impossible. But some reports appeared even before that hell, see https://www.dedoimedo.com/computers/firefox-emet-eaf-slowness-fix.html
And this situation tends to be worse with no response from Microsoft!
https://social.technet.microsoft.com/Forums/security/en-US/ef778e3e-89f0-4127-aea8-cef59036c480/emet-552-windows-7-sp1-eaf-and-performance-degradation?forum=emet
But the real hell will start sooner than later!
https://www.heise.de/ct/artikel/Exclusive-Spectre-NG-Multiple-new-Intel-CPU-flaws-revealed-several-serious-4040648.html

comment:27 in reply to:  25 Changed 16 months ago by cypherpunks

Replying to cypherpunks:

Tor Browser crashed on Windows 7 with 0xc00000fd = STACK_OVERFLOW, mentioned in this ticket only:

<EventData>

<Data>firefox.exe</Data>
<Data>52.6.0.6607</Data>
<Data>00000000</Data>
<Data>kernel32.dll</Data>
<Data>6.1.7601.24000</Data>
<Data>5a4996cc</Data>
<Data>c00000fd</Data>
<Data>0004719b</Data>

It happened when I had just switched a tab.

Hey, I have a similar crash of a chrome process (clean TBB 7.5.5) while just browsing:

Faulting application name: firefox.exe, version: 52.8.1.6607, time stamp: 0x00000000
Faulting module name: ntdll.dll, version: 6.1.7601.24150, time stamp: 0x5b0cb9eb
Exception code: 0xc00000fd
Fault offset: 0x0005a733

comment:28 Changed 5 weeks ago by cypherpunks

Resolution: duplicate
Status: closedreopened

Seems to be a ticket for MS EAF crap...
If "Validate access for modules that are commonly abused by exploits" is checked in 10.0.18362.356:

Problem signature
Problem Event Name:	BEX64
Application Name:	firefox.exe
Application Version:	68.1.0.7030
Application Timestamp:	00000000
Fault Module Name:	PayloadRestrictions.dll
Fault Module Version:	10.0.18362.1
Fault Module Timestamp:	c290dd32
Exception Offset:	000000000003b5d8
Exception Code:	c0000409
Exception Data:	0000000000000033
OS Version:	10.0.18362.2.0.0.768.101
Locale ID:	1033

comment:29 Changed 5 weeks ago by gk

Resolution: fixed
Status: reopenedclosed

Nope. That's only a ticket that nobody is looking at anymore. Please open a new one, thanks!

comment:30 Changed 5 weeks ago by cypherpunks

Well, that seems to happen only if WOW64 Tor Browser was started previously. MS doesn't deserve a new ticket, so let them read this one :P

comment:32 Changed 5 weeks ago by cypherpunks

Hah, works flawlessly for years. It's not a bug in Firefox. Read this ticket carefully.

Note: See TracTickets for help on using tickets.