Opened 5 years ago

Closed 20 months ago

#10804 closed defect (worksforme)

Tor Launcher extension hangs periodically when Tor Browser is started

Reported by: trlaunch01 Owned by: brade
Priority: High Milestone:
Component: Applications/Tor Launcher Version:
Severity: Normal Keywords: tbb-usability-stoppoint-app
Cc: gk, mcs, Emory+Hubbard, Manstein, mjdbb1, Gravitas, mttp Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Sometimes i observe that Tor Browser firefox browser window is not coming up after progress dialog completed. The process "firefox.exe" can be found within process list, but it never become visible or exit.
Each further launch of Tor Browser creates another firefox.exe, which will never open it's window. If i terminate the hanged firefox processes manually, then next launch of Tor Browser completes successfully.
I have did a research and found that the hang happen in Tor Launcher extension in file tl-protocol.js in the call of "readBytes" method.
There is also a related comment in source code:

TODO: readBytes() will sometimes hang if the control connection is opened
immediately after tor opens its listener socket. Why?

So, i assume, the problem is known to you, but is still not fixed.

Tested OS: WinXP
Tor Browser Bundle version: 3.5.1

Child Tickets

Attachments (4)

tor_log.txt (3.3 KB) - added by trlaunch01 5 years ago.
0001-Fix.patch (2.4 KB) - added by asmith 5 years ago.
Fix
nonblocking.patch (8 bytes) - added by cypherpunks 5 years ago.
Nothing
0001-testhang.patch (162 bytes) - added by testhang 5 years ago.
testhang

Download all attachments as: .zip

Change History (53)

comment:1 Changed 5 years ago by gk

Cc: gk added
Version: Tor: unspecified

comment:2 Changed 5 years ago by mcs

Cc: mcs added

comment:3 Changed 5 years ago by mcs

We placed the TODO comment in the code because we encountered some hangs during development of Tor Launcher (if I remember correctly, the hang occurred the first time Tor Launcher issued a control port command which is long before the progress dialog finishes). We do not know the root cause and we do not know how to reproduce the problem.

If you make sure that no tor.exe of firefox.exe processes are running before you open "Start Tor Browser.exe", how often does this hang occur?

To help us debug the problem, please do the following:
1) Start TBB (you'll have to get past the hang)
2) Open about:config
3) Change extensions.torlauncher.loglevel to 0
4) Change extensions.torlauncher.logmethod to 0
5) Exit the Tor Browser
6) From a cmd prompt, cd to the Browser subdirectory within TBB and start the browser by using the command: firefox -console

If you can reproduce the hang with the above debugging enabled, please send us the log output from the Console window.

comment:4 Changed 5 years ago by cypherpunks

Changed 5 years ago by trlaunch01

Attachment: tor_log.txt added

comment:5 Changed 5 years ago by trlaunch01

I was able to reproduce hang when launched firefox -console with tuned log level.
Please see all console output in attached file tor_log.txt

General remark: bug reproduces oftenly(each 3d time) when system is under load(for example intensive disk writes), or rarely otherwise

comment:6 Changed 5 years ago by mcs

Keywords: TorBrowserTeam201407 added

comment:7 Changed 5 years ago by mcs

Keywords: TorBrowserTeam201408 added; TorBrowserTeam201407 removed

comment:8 Changed 5 years ago by gk

Cc: Emory Hubbard added

Closed #12983 as duplicate of this.

comment:9 Changed 5 years ago by gk

Cc: Emory+Hubbard added; Emory Hubbard removed

comment:10 Changed 5 years ago by mikeperry

Keywords: TorBrowserTeam201410 added; TorBrowserTeam201408 removed

We probably should wait until after we get some working 31ESR builds before digging into this. Setting it to October to keep it out of the way until then. If we get 31ESR ready earlier, we can move this back.

comment:11 Changed 5 years ago by gk

Cc: Manstein added

Closed #13058 as duplicate.

comment:12 Changed 5 years ago by gk

Cc: mjdbb1 added

Closed #12887 as duplicate.

comment:13 Changed 5 years ago by mjdbb1

It seems that if I disable the addon "HTTPS-Everywhere 4.0.0" I can no longer replicate this issue on any machine I've tried.

Edit: As soon as I wrote that I discovered #13072:
https://trac.torproject.org/projects/tor/ticket/13072

Same issue?

Last edited 5 years ago by mjdbb1 (previous) (diff)

comment:14 Changed 5 years ago by gk

Cc: Gravitas added

Marked #13072 as duplicate of this.

comment:15 Changed 5 years ago by cypherpunks

Alas timeout is not about keep-alive.

Last edited 5 years ago by cypherpunks (previous) (diff)

comment:16 Changed 5 years ago by cypherpunks

I have did a research and found that the hang happen in Tor Launcher extension in file tl-protocol.js in the call of "readBytes" method.

How did you managed to track down it to Tor Launcher?

comment:17 Changed 5 years ago by cypherpunks

Hang that coming up after progress dialog completed is not about Tor Launcher.

Hang happens only with blocking socket, so it could to happen on TorSendCommand or TorRetrieveBootstrapStatus and friends, which unlikely could to bypass hang on writeBytes and to hang on readBytes() latter.
And if thread hangs then why progress window vanishes?

More likely two different bugs in here, hang in readBytes() of Tor Launcher long before progress completes, and hang triggered by torbutton_local_tor_check of Torbutton just after progress completes and before browser window should to appear. You can to reproduce it, place while(true){} loop to begin of torbutton_send_ctrl_cmd and observe the same as described behavior.

comment:18 in reply to:  17 Changed 5 years ago by mcs

Replying to cypherpunks:

More likely two different bugs in here, hang in readBytes() of Tor Launcher long before progress completes, and hang triggered by torbutton_local_tor_check of Torbutton just after progress completes and before browser window should to appear. You can to reproduce it, place while(true){} loop to begin of torbutton_send_ctrl_cmd and observe the same as described behavior.

Thanks cypherpunks. You may be correct about there being two different hangs, but I am most suspicious of the interaction between HTTPS Everywhere's check.t.o connection (see ticket:9693#comment:5) and the torbutton_local_tor_check().

Unfortunately, I have only been able to very very very rarely reproduce a hang (and only when I have been unprepared to debug it). Adding a while(true) loop inside torbutton_send_ctrl_cmd() causes what appears to be a hang, but if I change it to a "for (var i = 0; i < 1000; ++i)" loop, then normal startup finishes after the for loop is done. So that is not a true deadlock situation.

Gravitas -- thank you for all of your debugging in #13072. Can you try an experiment? Use about:config to toggle extensions.torbutton.local_tor_check to false and also set extensions.torbutton.test_url to an empty string. Modifying those two prefs will effectively disable all checks done by Torbutton. I'd like to know if doing this eliminates the hang.

Separately, you could also experiment by commenting out this line in HTTPS Everywhere:

this.testProxySettings();

(inside ssl-observatory.js).

comment:19 Changed 5 years ago by Gravitas

Sure, will try the experiments you recommended and get back to you in a day, max two. It hangs about 1 in 4 times on startup on my machine, so reproducing it is no problem. As I mentioned in the previous post, its probably because my machine is quite speedy (its a 12 core Xeon).

comment:20 Changed 5 years ago by Gravitas

Your intuition seems to be correct. I used about:config to toggle extensions.torbutton.local_tor_check to false, and also set extensions.torbutton.test_url to an empty string. I restarted 25 times, and it worked flawlessly. I then reset said settings back their defaults, and got a hang on try number 2.

I then commented out "this.testProxySettings();" inside ssl-observatory.js. I restarted 25 times, and it worked flawlessly. I then commented the line in again, and got a hang on tries number 6, 7 and 8.

I am happy to run any more tests that you would like.

Version: I was using HTTPS-Everywhere v4.0.1 (which may be slightly different to the default one supplied with Tor Browser). In any case, this particular bug exhibits itself with all versions of HTTPS-Everywhere.

Last edited 5 years ago by Gravitas (previous) (diff)

comment:21 Changed 5 years ago by cypherpunks

but I am most suspicious of the interaction between HTTPS Everywhere's check.t.o connection (see ticket:9693#comment:5) and the torbutton_local_tor_check().

It's the same problem as diagnosed and fixed for hang on New Identity control port access by #9531, torbutton_local_tor_check() using control port on browser start-up.

So that is not a true deadlock situation.

Yes, it was imitation of deadlock which triggered on start-up while accessing control port soon after TLS connection somewhere in browser launched.

Changed 5 years ago by asmith

Attachment: 0001-Fix.patch added

Fix

comment:22 Changed 5 years ago by asmith

Gravitas, can you try to test with attached patch?

comment:23 Changed 5 years ago by sedidevnull

For me, who had this issue, the patch works in all of the > 25 start attempts I've done.

comment:24 Changed 5 years ago by mttp

Cc: mttp added

comment:25 Changed 5 years ago by cypherpunks

Hang inside of Torbutton should be fixed for 1.6.12.3 version. This bug is about Tor Launcher hang now. Proper way to fix this bug is to use non-blocking sockets, which will require massive code refactoring.

comment:26 Changed 5 years ago by cypherpunks

Nothing

Last edited 5 years ago by cypherpunks (previous) (diff)

comment:27 Changed 5 years ago by cypherpunks

Nothing

Last edited 5 years ago by cypherpunks (previous) (diff)

Changed 5 years ago by cypherpunks

Attachment: nonblocking.patch added

Nothing

comment:28 Changed 5 years ago by Gravitas

Gravitas, can you try to test with attached patch?

Although I'm familiar with C/C++/C#, I've never used JavaScript.

I tried downloading both versions of TorBrowser for Windows (3.6.6 and 4.0-alpha-3) and searching all *.js files for the text inside the patch, with no luck.

Could you point me in the right direction for some instructions on how to test the patch "0001-Fix.patch"? The easiest way would be if you attached a .zip file that contained a modified version of Tor Browser, so I could download and test it out. I can also provide links to an anonymous FTP server if this would be easier.

comment:29 in reply to:  28 Changed 5 years ago by cypherpunks

Replying to Gravitas:

Could you point me in the right direction for some instructions on how to test the patch "0001-Fix.patch"?

Patch was shipped with new Torbutton version for Tor Browser 3.6.6. It should to fix hang. Can you confirm?

comment:30 Changed 5 years ago by muckybucket

I am still having this issue with Tor Browser 3.6.6

comment:31 Changed 5 years ago by mjdbb1

I too am still experiencing the issue on 3.6.6 (although I will say it doesn't seem to be as frequent). Again, if I disable HTTPS-Anywhere, it seems to be completely gone.

comment:32 Changed 5 years ago by trlaunch01

On my side the issue also became a bit more rare with 3.6.6. But it is still in. I would be glad to check if the problem reproduces without HTTPS-Anywhere, but cannot do this because HTTPS-Anywhere is a required option for me

comment:33 Changed 5 years ago by Gravitas

The issue has completely disappeared for me in Tor Browser v3.6.6.

Before this release, I could reliably reproduce it on my machine.

I just restarted it 25 times, and it worked flawlessly every time. Whatever change was made, its made it much better.

comment:34 in reply to:  32 Changed 5 years ago by cypherpunks

Replying to muckybucket:

I am still having this issue with Tor Browser 3.6.6

Replying to mjdbb1:

I too am still experiencing the issue on 3.6.6 (although I will say it doesn't seem to be as frequent). Again, if I disable HTTPS-Anywhere, it seems to be completely gone.

Replying to trlaunch01:

On my side the issue also became a bit more rare with 3.6.6. But it is still in. I would be glad to check if the problem reproduces without HTTPS-Anywhere, but cannot do this because HTTPS-Anywhere is a required option for me

What number of cores reported for your hosts, folks? (what CPU installed?)

comment:35 Changed 5 years ago by Gravitas

Dell T7500, dual Xeon CPU, each CPU has 6 cores running @ 2.66Ghz, Hyperthreading turned on, for a total of 24 hyperthreaded cores. SSD has read/write speeds of 200MByte/sec.

On my machine, it used to hang consistently around 25% of the time on startup, with the latest version 3.66, it is rock solid (see my post above).

comment:36 Changed 5 years ago by mjdbb1

Intel Core2 Duo E7500 2.93GHz.

comment:37 Changed 5 years ago by trlaunch01

What number of cores reported for your hosts, folks? (what CPU installed?)

The issue does not depend directly on number of cores. It produces on processors with single, dual and quad cores. Approximately 1 of 30 launches of Tor Browser leads to hung. The problem can be reproduced very oftenly if hard disk is under load by some other app.
When i modified the code of Tor Launcher extension and inserted "Sleep(5seconds)" right after creation of connection to tor process and before sending any commands, the problem stopped reproducing on my side - just for you information

comment:38 Changed 5 years ago by cypherpunks

When i modified the code of Tor Launcher extension and inserted "Sleep(5seconds)" right after creation of connection to tor process and before sending any commands, the problem stopped reproducing on my side - just for you information

Could you to post all code you inserted? (As I know, JS have no support for Sleep() command).

comment:39 Changed 5 years ago by mikeperry

Keywords: TorBrowserTeam201411 added; TorBrowserTeam201410 removed

comment:40 Changed 5 years ago by testhang

Anybody with:

  1. some free time
  2. bug reproducibility (easy to do if 0001-Fix.patch reverted for torbutton)
  3. resources to build browser (64bit platform and 4G+ virt mem minimum)

Could you to contact me (email: testhang (at) sigaint.org) to test c++ (two-line) patch?

Last edited 5 years ago by testhang (previous) (diff)

comment:41 in reply to:  40 ; Changed 5 years ago by brade

Replying to testhang:

Anybody with:

  1. some free time
  2. bug reproducibility (easy to do if 0001-Fix.patch reverted for torbutton)
  3. resources to build browser (64bit platform and 4G+ virt mem minimum)

Could you to contact me (email: testhang (at) sigaint.org) to test c++ (two-line) patch?

Can you post the patch here?

Possibly someone could create a build with your patch and someone else could test.

I haven't been able to reproduce the bug within Tor Launcher (even before Torbutton was patched) so I am not a candidate to confirm that the bug is fixed.

comment:42 in reply to:  41 ; Changed 5 years ago by testhang

Replying to brade:

Can you post the patch here?

Attached.

Changed 5 years ago by testhang

Attachment: 0001-testhang.patch added

testhang

comment:43 in reply to:  42 Changed 5 years ago by testhang

Replying to testhang:

Attached.

Detached.

comment:44 Changed 4 years ago by mikeperry

Keywords: TorBrowserTeam201412 added; TorBrowserTeam201411 removed

comment:45 Changed 4 years ago by mikeperry

Keywords: TorBrowserTeam201412 removed

comment:46 Changed 4 years ago by mcs

Possible duplicates: #15182 and #15669.

comment:47 Changed 4 years ago by mcs

Keywords: tbb-usability-stoppoint-app added; tor launcher removed

comment:48 in reply to:  4 Changed 3 years ago by bugzilla

Severity: Normal

Replying to cypherpunks:

it's related to #9531
Reason is ​https://bugzilla.mozilla.org/show_bug.cgi?id=698882

Mozilla's adventures continue in https://bugzilla.mozilla.org/show_bug.cgi?id=1279163, and the bug is marked fixed for FF48, but it was backed out, because it worked only with the new shiny MustDie compiler! (so Mozilla promises it for FF49.) TBB doesn't have such awful dependencies, so does anybody want to backport the fix?

comment:49 Changed 20 months ago by gk

Resolution: worksforme
Status: newclosed

I think the hangs are gone. If not we should reopen this tickets with new steps to reproduce.

Note: See TracTickets for help on using tickets.