Opened 8 years ago

Closed 7 years ago

Last modified 7 years ago

#4956 closed defect (fixed)

TBB for Windows plus Kaspersky 2012 equals BSOD

Reported by: runa Owned by: sebastian
Priority: High Milestone: Tor: 0.2.2.x-final
Component: Core Tor/Tor Version:
Severity: Keywords: tor-client
Cc: g.koppen@…, nextgens, Shondoit Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I've heard about this issue from three different users; running TBB for Windows (version 2.2.35-4) on a machine with Kaspersky 2012 installed will result in a bluescreen. Will see if I can debug more.

Child Tickets

Change History (58)

comment:1 Changed 8 years ago by gk

Cc: g.koppen@… added

comment:2 Changed 8 years ago by runa

One user is running the "Internet Security 2012" version.

comment:3 Changed 8 years ago by runa

Windows 7, 64-bit, TBB version 2.2.35-4, Kaspersky Internet Security v.12.0.0.374; I can run TBB successfully the first time, the second time the system freezes up when Vidalia is starting the Tor software.

comment:4 Changed 8 years ago by Sebastian

Have you tried contacting Kaspersky?

comment:5 Changed 8 years ago by Sebastian

Also, does it happen if you just launch tor, or does it have to be Vidalia that's launching Tor?

comment:6 in reply to:  4 Changed 8 years ago by runa

Replying to Sebastian:

Have you tried contacting Kaspersky?

Only via Twitter, so I guess that doesn't count. It's on my list of things to do.

comment:7 Changed 8 years ago by runa

Owner: changed from erinn to runa
Status: newassigned

comment:8 Changed 8 years ago by runa

Thank you for contacting Kaspersky Lab Technical Support''. I'll update this ticket when I hear back.

comment:9 in reply to:  8 Changed 8 years ago by elian

Do you have an update?

Replying to runa:

Thank you for contacting Kaspersky Lab Technical Support''. I'll update this ticket when I hear back.

comment:10 Changed 8 years ago by runa

Owner: changed from runa to sebastian

Kaspersky had a look and came back with the following:

Dear customer,

Dear Runa, I have just heard back from our escalation team.
This is what they have found.

I was able to reproduce bsod and the newly created dump was analyzed by our developers.

There is a bsod while system makes call from tor driver:
Call Site
nt!MiResolvePageFileFault
nt! ?? ::FNODOBFM::`string'
nt!MiDispatchFault
nt!MmAccessFault
nt!KiPageFault
tor

This info should be analyzed by Tor developers for further investigation.

They also uploaded dump to mentioned above ftp folder (Lisa\myMEMORY3.zip)

So it would seem the issue needs to be passed back to Tor for further testing

The myMEMORY3.zip dump file can be found on http://bayfiles.com/file/a3cf/07Lr8P/myMEMORY3.zip

comment:11 Changed 8 years ago by Sebastian

Maybe Nick knows what to do with that, I certainly don't. I wonder if someone can reproduce the bsod with just tor and kaspersky?

comment:12 Changed 8 years ago by arma

There is no Tor driver -- it seems that whoever wrote that thinks Tor runs some drivers, rather than just a program called Tor.

rransom suggests that it could be the fact that Tor uses mmap to interact with its cached descriptor file. That's a plausible guess. Runa, can you ask your contact if an ordinary program using mmap might produce the behavior he saw? And if so, if he still thinks it's a bug in the ordinary program?

comment:13 in reply to:  11 Changed 8 years ago by runa

Replying to Sebastian:

Maybe Nick knows what to do with that, I certainly don't. I wonder if someone can reproduce the bsod with just tor and kaspersky?

I can give it a try tomorrow.

comment:14 in reply to:  12 Changed 8 years ago by runa

Replying to arma:

There is no Tor driver -- it seems that whoever wrote that thinks Tor runs some drivers, rather than just a program called Tor.

rransom suggests that it could be the fact that Tor uses mmap to interact with its cached descriptor file. That's a plausible guess. Runa, can you ask your contact if an ordinary program using mmap might produce the behavior he saw? And if so, if he still thinks it's a bug in the ordinary program?

I can test it tomorrow if you can name "an ordinary program using mmap"?

comment:15 Changed 8 years ago by arma

The "ordinary program using mmap" is called Tor.

comment:16 Changed 8 years ago by arma

If a) Kaspersky can't solve it (e.g. it's a Windows bug), and b) it is in fact mmap-related, then a next step is to investigate a torrc option to opt not to use mmap.

comment:17 Changed 8 years ago by arma

Cc: nextgens added

comment:18 Changed 8 years ago by arma

Our unnamed friend on irc points out http://www.auslogics.com/en/software/disk-defrag/history/ which has a changelog entry of "fixed BSOD for computers that have Kaspersky Internet Security installed".

Seems very likely that our mmap, or a race with disk access, is the issue here.

comment:19 Changed 8 years ago by nickm

Component: Tor bundles/installationTor Client
Milestone: Tor: 0.2.2.x-final

I have no idea what to do with that file; yeesh. Also, I have no idea what the "tor driver" is. Does that computer have some weird driver claiming to be tor?

To experiment with a windows tor with no mmap, just change the "#elif defined(_WIN32)" on compat.c line 216 to something like #elif 0, so that it uses the fallback implementation. If that makes the problem go away, it looks like it's time to implement "kaspersky mode".

comment:20 Changed 8 years ago by nickm

(If you're tweaking the Tor 0.2.2.x code, the case to disable is the "#elif defined(MS_WINDOWS)" around line 171 of compat.c)

comment:21 Changed 8 years ago by nickm

In case the mmap theory turns out to be right, let's get a head start on that with a torrc option; the branch enable_mmap_option in my public repository would be the seed of a fix there. It's a branch on 0.2.2. It needs documentation; it might be b0rken.

comment:22 Changed 8 years ago by VenZe

Hello!
While i tested this case from our side (tried to find problem component in KIS), i got several dumps, one of that is linked above by runa. Another dump leads to problem with disk access as arma sad:

ntfs!NtfsPagingFileIo_04BAD9A0

nt!KeBugCheckEx+0x0
ntfs!NtfsPagingFileIo+0x155
ntfs! ?? ::FNODOBFM::`string'+0x8a29
fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x24f
fltmgr!FltpDispatch+0xcf
nt!IoPageRead+0x252
nt!MiIssueHardFault+0x255
nt!MmAccessFault+0x14bb
nt!KiPageFault+0x16e
0x75733b05

As additional, sometimes system was OK, but i get next error (leaded to Vidalia):
http://images.netbynet.ru/imgs/ff585f98f027e030f5a9fb963507bcc1.png

Hope it will help you.
Regards.

comment:23 Changed 8 years ago by runa

I downloaded TBB 2.2.35-12 and got a BSOD on the third run.

I downloaded and installed the Expert Bundle, but was not been able to trigger a BSOD in the 30-or-so times I used it.

I installed the Vidalia Bundle with just Tor and Vidalia; I got a BSOD the second time I allowed Tor to start via the Vidalia interface.

comment:24 Changed 8 years ago by cypherpunks

Hey BSODers!
Report your processor and graphics controller.

comment:25 Changed 8 years ago by cypherpunks

#4179 related.

comment:26 in reply to:  24 ; Changed 8 years ago by cypherpunks

Replying to cypherpunks:

Hey BSODers!
Report your processor and graphics controller.

I'm running Kaspersky 2012 and did have blue screen two times after installation. However, since two weeks ago I haven't had any issues. I cannot say for sure my BSOD was from Kasperskay and TBB issue. This is the first I have read about this issue with Tor.

That said, when the BSODs happened, I had this on-board graphics card and processor:

Intel i5-2300 2.3 GHz
Intel HD graphics (is that what you requested?)

After I upgraded my graphics card (to an EVGA GeForce GTX 560Ti) I haven't seen any BSODs, however: Post hoc ergo propter hoc ;-)

(Windows 7 Home Premium 64bit SP1)

comment:27 in reply to:  26 Changed 8 years ago by cypherpunks

Replying to cypherpunks:

I'm running Kaspersky 2012 and did have blue screen two times after installation. However, since two weeks ago I haven't had any issues. I cannot say for sure my BSOD was from Kasperskay and TBB issue. This is the first I have read about this issue with Tor.

That said, when the BSODs happened

When your BSODs happened? Are you writing about instant BSOD during TBB start or random crashes during surf?

Folks, why you need this task if you can't report what processor type used?

comment:28 Changed 8 years ago by cypherpunks

Resolution: wontfix
Status: assignedclosed

NOBODY CARE

comment:29 Changed 8 years ago by nickm

Resolution: wontfix
Status: closedreopened

Don't do that.

comment:30 Changed 8 years ago by nickm

Runa, when you managed to reproduce the bsod, what processor(s) did you have? How long did it take, what version of windows, what version of KIS, etc?

comment:31 Changed 8 years ago by elian

For me, the BSOD occurs during TBB start, I never get to see the browser.

Some system specs:
Intel Core 2 Duo
Radeon 5770
Windows 7 64 bit
KIS 2012

comment:32 in reply to:  31 Changed 8 years ago by elian

Clarification: I never get to see the browser when a BSOD occurs. It's not that I always get a BSOD.

Replying to elian:

For me, the BSOD occurs during TBB start, I never get to see the browser.

Some system specs:
Intel Core 2 Duo
Radeon 5770
Windows 7 64 bit
KIS 2012

comment:33 Changed 8 years ago by nextgens

I can reproduce the bug too (win7 x64 with KIS2012)

  • disabling mmap fixes it
  • 'pausing' KIS fixes it

processor/graphic controller is irrelevant, I've reproduced it with a minimal installation in a VM.

I didn't manage to get a mem-dump where KIS appears on the stack-trace, but then again, it doesn't prove anything...

10:01 < armadev> next step i guess is we need to decide if we can just disable

mmap preemptively for windows users, simply because a few of
them want to run a buggy program

comment:34 in reply to:  33 Changed 8 years ago by cypherpunks

Replying to nextgens:

processor/graphic controller is irrelevant, I've reproduced it with a minimal installation in a VM.

Double BullShit. Processor is relevant. VM is not emulator.

comment:35 in reply to:  33 Changed 8 years ago by cypherpunks

Replying to nextgens:

  • disabling mmap fixes it

10:01 < armadev> next step i guess is we need to decide if we can just disable

mmap preemptively for windows users, simply because a few of
them want to run a buggy program

Or better to fix way that tor mmaping files?

comment:36 Changed 8 years ago by cypherpunks

How can mmap trigger bsod only for case of vidalia running tor? It's problem with vidalia that runs tor so it never can use mmap safely.

comment:37 in reply to:  30 Changed 8 years ago by runa

Replying to nickm:

Runa, when you managed to reproduce the bsod, what processor(s) did you have? How long did it take, what version of windows, what version of KIS, etc?

I can't check processor info at the moment, but I can say that I have managed to reproduce the bsod on two different laptops and one virtual machine. All have Windows 7 64-bit and Kaspersky Internet Security v.12.0.0.374.

With TBB version 2.2.35-4; I can run TBB successfully the first time, the second time the system freezes up when Vidalia is starting the Tor software.

With TBB version 2.2.35-12; I don't always get a bsod the second time I run TBB, but it usually happens three or four times when you run TBB ten times in total.

comment:38 Changed 8 years ago by VenZe

My system is Core i5 750 + Radeon HD4500 series. Test environment was VM-Win7x64 + TBB 2.2.35-12+KIS2012.
2Runa: Could you try to reproduce problem with disabled Web-AV in KIS settings?

comment:39 Changed 8 years ago by dumbnewbie

I've been using the latest TBB (2.2.35-12) since it was released but I've only suffered this issue for past two days. There has only been one instance when I've started TBB in the past two days that the computer didn't freeze. The only change to the computer in the past two days has been from updating KAV database.


Specs


CPU: Intel Core 2 Q6600
GFX: EVGA GTX 460

OS : Windows 7 Pro x64
KAS: KAV 2012
TOR: TBB 2.2.35-12


I don't get a dump because the computer only freezes and requires a hard reboot, but the Windows Application Event Log has an entry for the issue from TBB.

Faulting application name: tor.exe, version: 0.0.0.0, time stamp: 0x4fad69a1
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x77b89c12
Faulting process id: 0x12e4
Faulting application start time: 0x01cd38a3371570d2
Faulting application path: C:\Program Files (x86)\TBB\App\tor.exe
Faulting module path: unknown
Report Id: 76b59e20-a496-11e1-8f3f-0018f336c734

comment:40 Changed 8 years ago by Shondoit

Cc: Shondoit added

Intel Core i7 740
AMD Radeon HD6500M
VM with:
Windows 7 Ultimate 64-bit
TBB 2.2.35-12
KIS 2012

I've noticed different forms of crashing.
The most common one is a freeze as mentioned in comment 3.
Sometimes it's a full-blown blue screen.

comment:41 Changed 8 years ago by arma

Priority: normalmajor

Sounds like we should patch Tor to not use mmap on Windows, and then we will have some breathing room to solve the problem in peace (or not bother to) without making our Windows users think we suck.

comment:42 in reply to:  38 Changed 8 years ago by runa

Replying to VenZe:

2Runa: Could you try to reproduce problem with disabled Web-AV in KIS settings?

Yes, I am able to reproduce with Web-AV disabled.

comment:43 Changed 8 years ago by runa

Tor runs fine after I disable the Kaspersky network filter (obviously not a good idea to recommend people do this).

comment:44 in reply to:  41 Changed 8 years ago by runa

Replying to arma:

Sounds like we should patch Tor to not use mmap on Windows, and then we will have some breathing room to solve the problem in peace (or not bother to) without making our Windows users think we suck.

Will this actually help? Tor itself is not the problem, Vidalia is. Unless there is a difference in the way Tor starts up when it's standalone-Tor vs TBB-Tor.

comment:45 Changed 8 years ago by marshray

I got someone who knows a lot more about kernel debugging than I do to help me look at the crash dump from http://bayfiles.com/file/a3cf/07Lr8P/myMEMORY3.zip . Here are the findings, they may be useful to Kaspersky:

  • The address 4c0748 is in the tor.exe process and is the address of the next instruction for Tor.exe to run. This code has been paged out. When the OS goes to execute the instruction it causes a page fault, which is normal. But when the OS tries to load it from the pagefile the kernel encounters a corrupted PTE (page table entry). This creates a double-fault situation which results in a bluescreen.
  • Tor has no drivers or any other code in the kernel.
  • There is nothing Tor.exe is doing wrong with mapped files that could this. The 'mapping' that triggers the crash is the tor.exe image itself. The PTEs were corrupted at some point before that.
  • There is no reason to think that changing Tor to not use a mapped file would be a real fix for the problem, although it may mask it for a while.
  • The problem is most likely Kaspersky's kernel code.

comment:46 in reply to:  36 Changed 8 years ago by Shondoit

Replying to nextgens:

  • disabling mmap fixes it

It doesn't for me. 3 runs, 2 bsods.

Replying to cypherpunks:

How can mmap trigger bsod only for case of vidalia running tor?

Apparently it doesn't. It's one big red herring.

comment:47 Changed 7 years ago by VenZe

I'll send marshray's message to our developers. Thx

comment:48 Changed 7 years ago by VenZe

Please, try to update KIS from ftp://dnl-test.kaspersky-labs.com/test/ids (uncheck Kaspersky Lab update servers) and tell me if it helps or not? I am not able to reproduce this issue on my VM machine anymore.

comment:49 in reply to:  45 ; Changed 7 years ago by cypherpunks

Replying to marshray:

  • The address 4c0748 is in the tor.exe process and is the address of the next instruction for Tor.exe to run. This code has been paged out. When the OS goes to execute the instruction it causes a page fault, which is normal. But when the OS tries to load it from the pagefile the kernel encounters a corrupted PTE (page table entry). This creates a double-fault situation which results in a bluescreen.

The address 4c0748 belongs to first instruction of nt_service_set_state(). It called (no mater if tor ran as service or not) only once during initial of start-up from tor_main(). How page with it has been paged out? Is it about OS caching to page file?

comment:50 in reply to:  48 Changed 7 years ago by runa

Replying to VenZe:

Please, try to update KIS from ftp://dnl-test.kaspersky-labs.com/test/ids (uncheck Kaspersky Lab update servers) and tell me if it helps or not? I am not able to reproduce this issue on my VM machine anymore.

After updating KIS, I successfully ran the latest Tor Browser Bundle (version 2.2.35-13) 30 times without a single problem. Looks like this problem has been fixed.

comment:51 Changed 7 years ago by Shondoit

I'd like to confirm this but can't since the update process gives a time-out and fails.
Updated with the default Kaspersky Lab Update servers to the very latest available.
I then added the ftp link and removed the Kaspersky Lab update servers and reran the update, which in turn failed.

comment:52 Changed 7 years ago by VenZe

This link contains only latest network drivers and used for testing purposes only.
2Shondoit: may be some internet connection problems?
2Runa and others who tested this link: you may probably need to reinstall product, since reverting product updates to default servers may cause KIS to fail update process in future.
Summary: this drivers will be included in new version of KIS, that will be available in nearest future.
For current version workaround is to delete NDIS filter, but it is not recommended.

comment:53 in reply to:  49 Changed 7 years ago by marshray

Replying to cypherpunks:

The address 4c0748 belongs to first instruction of nt_service_set_state(). It called (no mater if tor ran as service or not) only once during initial of start-up from tor_main(). How page with it has been paged out? Is it about OS caching to page file?

Strange. It was definitely inpaging tor.exe. I probably just typed the wrong address.

But it sounds like maybe the issue has been fixed? I'll keep an eye on the bug and if it's not clearly fixed I'll peek in that crashdump again.

comment:54 Changed 7 years ago by skeith

this is also happened to me, using Vidalia + KIS 2012. I usually disable KIS for a moment till Vidalia get the first handshake

comment:55 Changed 7 years ago by Sebastian

We got this resolved with Kasperksy, right?

comment:56 Changed 7 years ago by VenZe

Resolution: fixed
Status: reopenedclosed

Update KIS2012 ~> KIS2013 and it should be enough.

comment:57 Changed 7 years ago by nickm

Keywords: tor-client added

comment:58 Changed 7 years ago by nickm

Component: Tor ClientTor
Note: See TracTickets for help on using tickets.