Opened 22 months ago

Closed 21 months ago

Last modified 21 months ago

#23258 closed defect (fixed)

HTTPS Everywhere is not working when noscript is enabled globally

Reported by: Dbryrtfbcbhgf Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Major Keywords: TorBrowserTeam201708R
Cc: legind Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

HTTPS Everywhere auto updated and now there is a new icon in the toolbar, clicking on the new icon shows a blank popup with 2 checkboxes.
TorBrowser 7.0.4
Https everywhere 2017.8.15

Child Tickets

Attachments (3)

bug.png (198.1 KB) - added by Dbryrtfbcbhgf 22 months ago.
Image showing bug
23258-7.5a4.png (16.6 KB) - added by dcf 22 months ago.
bug.2.png (51.8 KB) - added by Dbryrtfbcbhgf 21 months ago.

Download all attachments as: .zip

Change History (45)

comment:1 Changed 22 months ago by Dbryrtfbcbhgf

HTTPS everywhere changelog 2017.8.15

  • Incorporating numerous fixes for WebExtensions to work within Firefox
  • Firefox: Creating Embedded WebExtension wrapper to migrate legacy settings
  • Removing legacy XPCOM codebase, unifying Firefox and Chrome codebase
  • Ruleset updates

https://www.eff.org/files/Changelog.txt

Changed 22 months ago by Dbryrtfbcbhgf

Attachment: bug.png added

Image showing bug

Changed 22 months ago by dcf

Attachment: 23258-7.5a4.png added

comment:2 Changed 22 months ago by dcf

I guess I have the same on GNU/Linux with 7.5a4. I don't remember if the icon was on the toolbar before or not. In my case, the panel isn't blank, unlike Dbryrtfbcbhgf's bug.png.


comment:3 Changed 22 months ago by gk

Component: Applications/Tor BrowserHTTPS Everywhere/EFF-HTTPS Everywhere
Owner: changed from tbb-team to jsha
Status: newneeds_information

The new icon should be as expected but the blank one not. legind: Do you track this bug somewhere?

comment:4 Changed 22 months ago by gk

Owner: changed from jsha to legind
Status: needs_informationassigned

comment:5 Changed 22 months ago by gk

Status: assignedneeds_information

comment:6 in reply to:  2 Changed 22 months ago by cypherpunks

Replying to dcf:

I guess I have the same on GNU/Linux with 7.5a4. I don't remember if the icon was on the toolbar before or not. In my case, the panel isn't blank, unlike Dbryrtfbcbhgf's bug.png.

This only happens when the security level is set to high or medium, only the low level shows the panel correctly.

Last edited 22 months ago by cypherpunks (previous) (diff)

comment:7 Changed 22 months ago by cypherpunks

Additionally HTTPS Everywhere has stopped redirecting pages to HTTPS according to the rules. For example, http://imgur.com/ used to redirect to HTTPS, now it does not.

comment:8 in reply to:  7 Changed 22 months ago by Lolint

Replying to cypherpunks:

Additionally HTTPS Everywhere has stopped redirecting pages to HTTPS according to the rules. For example, http://imgur.com/ used to redirect to HTTPS, now it does not.

I confirm, even at Low security setting redirects don't happen.

Also rulesets aren't shown when clicking on the HTTPS Everywhere icon, so we can't disable some, unlike with other versions.

Last edited 22 months ago by Lolint (previous) (diff)

comment:9 Changed 22 months ago by mcs

Cc: tbb-team added

comment:10 in reply to:  7 Changed 22 months ago by boklm

Replying to cypherpunks:

Additionally HTTPS Everywhere has stopped redirecting pages to HTTPS according to the rules. For example, http://imgur.com/ used to redirect to HTTPS, now it does not.

I can reproduce this problem (and the blank popup too) when security level is set to High or Medium. It is however still working when using the Low security level (after restarting the browser when changing the level).

comment:11 Changed 22 months ago by boklm

It seems the problem is related to noscript. Setting noscript.global to false and restarting the browser is enough to cause HTTPS Everywhere to stop working.

comment:12 Changed 22 months ago by boklm

Summary: HTTPS Everywhere auto updated and now there is a new icon in the toolbarHTTPS Everywhere is not working when noscript is enabled globally

comment:13 Changed 22 months ago by cypherpunks

Once again the developers messed the "stable" series without testing, even when they had been warned https://blog.torproject.org/comment/270389#comment-270389. This is unprofessional.

comment:14 in reply to:  2 Changed 22 months ago by dcf

Replying to dcf:

I guess I have the same on GNU/Linux with 7.5a4. I don't remember if the icon was on the toolbar before or not. In my case, the panel isn't blank, unlike Dbryrtfbcbhgf's bug.png.

comment:6, comment:7, comment:10 are correct as far as a I can tell. Yesterday I was on the Low security setting and the panel was not blank. Today I am on the Medium security level and the panel is blank. And typing "web.archive.org" into the URL bar does not automatically take me to https://web.archive.org/.

comment:15 Changed 22 months ago by gk

Cc: legind added; tbb-team removed
Component: HTTPS Everywhere/EFF-HTTPS EverywhereApplications/Tor Browser
Keywords: TorBrowserTeam201708 added
Owner: changed from legind to tbb-team
Status: needs_informationassigned

I guess what we can and maybe even should do is whitelist moz-extension: (see #21270). That solves the problem for me. However, it causes a startup delay of several seconds on my Linux debug build. So, hrm.

comment:16 in reply to:  15 ; Changed 22 months ago by gk

Replying to gk:

I guess what we can and maybe even should do is whitelist moz-extension: (see #21270). That solves the problem for me. However, it causes a startup delay of several seconds on my Linux debug build. So, hrm.

So, the startup delay is not related to the whitelisting but to the new HTTPS-E itself. :/

comment:17 Changed 22 months ago by yawning

comment:18 Changed 22 months ago by legind

@cypherpunks again, your comments are extremely unhelpful.

This is related to https://github.com/EFForg/https-everywhere/issues/11948#issuecomment-323147599.

comment:19 in reply to:  16 Changed 22 months ago by yawning

Replying to gk:

Replying to gk:

I guess what we can and maybe even should do is whitelist moz-extension: (see #21270). That solves the problem for me. However, it causes a startup delay of several seconds on my Linux debug build. So, hrm.

So, the startup delay is not related to the whitelisting but to the new HTTPS-E itself. :/

That's unfortunate especially for people that use amnesiac setups like Tails or the experimental sandbox option. What is HTTPSE doing during the startup delay? Is it something that can be accelerated by pregenerating things during the build process?

As far as I am aware, the plan is to disable automated updates (that aren't done via the Tor Browser update process, as in, via a MAR), so the delay could be eliminated...

comment:20 Changed 22 months ago by legind

I've identified the issue with cookies being disabled completely and have a fix for it, but this doesn't seem to solve the problem for Tor Browser on medium or high security settings, and I'm getting no useful output from the browser console.

comment:21 Changed 22 months ago by legind

Testing between Firefox 52 (ESR) and Firefox 55.

When JavaScript is disabled in 55, the extension behaves properly upon browser load.
When JavaScript is disabled in 52 ESR, the extension does not redirect URLs and the popup is loaded with blank fields.

So this seems to be a problem Firefox fixed between the two versions. I wonder if we can get this backported to ESR. I'm still not sure what the *exact* problem is, since Firefox does not throw any errors in the debugger.

comment:22 in reply to:  21 Changed 22 months ago by mcs

Replying to legind:

So this seems to be a problem Firefox fixed between the two versions. I wonder if we can get this backported to ESR. I'm still not sure what the *exact* problem is, since Firefox does not throw any errors in the debugger.

I do not have time to investigate further right now, but a quick bugzilla search turned up this possibility:

"Disabling JavaScript breaks Web Extensions"
https://bugzilla.mozilla.org/show_bug.cgi?id=1329731

comment:23 Changed 22 months ago by legind

Thanks @mcs, I'm guessing that's the issue. Can we make a new Tor Browser release, applying this patch? This is not something I can fix in HTTPS Everywhere.

comment:24 Changed 22 months ago by legind

I've just made a new release - use HTTPS Everywhere 2017.8.17 for new builds.

comment:25 Changed 22 months ago by legind

If backporting the fix in https://bugzilla.mozilla.org/show_bug.cgi?id=1329731 into Tor Browser isn't viable, you could keep the current build version 5.2.21 and disable the XPCOM update channel, which should be achievable via

sed -i 's/https:\/\/www.eff.org\/files\/https-everywhere-eff-update-2048.rddata:text\/plain,/g' src/install.rdf

Note that this will not get ruleset updates, since rulesets are updated with the extension itself.

This is a less than ideal solution since the upgrade to WebExtensions will have to be made in any case, and users who do not upgrade to the Embedded WebExtension (current HTTPS Everywhere) first will miss an important settings migration step in addition to having stale rulesets. Let's try to have Mozilla fix this in ESR.

Last edited 22 months ago by legind (previous) (diff)

comment:26 Changed 22 months ago by legind

For the latest rulesets you could add another line to the build process, either after or before removing the update channel:

git tag -v 2017.8.17 && git checkout 2017.8.17 src/chrome/content/rules/

The rulesets are backwards compatible with the XPCOM version of the extension.

Last edited 22 months ago by legind (previous) (diff)

comment:27 Changed 22 months ago by boklm

I think we can take this patch in the next release if that fix the issue. However the next planned release is on September 26, which is a long time. So we have to see if we want to make a release to fix that before, or if there is an other workaround we could use to avoid a new release.

Changed 21 months ago by Dbryrtfbcbhgf

Attachment: bug.2.png added

comment:28 Changed 21 months ago by Dbryrtfbcbhgf

The preference button on ad-on->extension->click on the blue more button next to https everywhere, is missing and shows a browse button instead. Here is a photo showing the bug. bug.2.png

comment:29 in reply to:  16 ; Changed 21 months ago by cypherpunks

Replying to gk:

So, the startup delay is not related to the whitelisting but to the new HTTPS-E itself. :/

Or maybe you were using Medium or High security settings and there's no JIT whitelist for webextensions just like it was the case with pdfjs?

comment:30 in reply to:  18 Changed 21 months ago by cypherpunks

Replying to legind:
If you really want to fix this, you can make a new release and put refreshed 5.2.21 in it, then postpone updates from EFF for TBB channels like it was before (ticket:10394#comment:7 if it's still available), and then make a new release with the latest updates for all clients which unaffected.

Last edited 21 months ago by cypherpunks (previous) (diff)

comment:31 in reply to:  29 Changed 21 months ago by gk

Replying to cypherpunks:

Replying to gk:

So, the startup delay is not related to the whitelisting but to the new HTTPS-E itself. :/

Or maybe you were using Medium or High security settings and there's no JIT whitelist for webextensions just like it was the case with pdfjs?

No, that's on level "low". Actually, getting NoScript to work on higher security slider settings shows the same effect. Thus, that's very likely due to the new HTTPS-Everywhere.

comment:32 Changed 21 months ago by boklm

#23290 is a duplicate.

comment:33 Changed 21 months ago by legind

With NoScript, much of the code is still in the XPCOM wrapper. That's why it continues to work.

On all pure WebExtensions I've tested in ESR with JavaScript disabled, I've seen the same result. I've tested with uBlock Origin, Disconnect, Ghostery, Privacy Badger. All the same: no functionality, unresponsive.

As I've mentioned the way to fix this is to pick one of the following:

  1. Backport the Firefox 54 fix to Tor Browser
  2. Wait for Firefox to backport the fix, which given their responsiveness lately I doubt will be very effective
  3. Make a new TB release with 5.2.21, removing the update channel and using the new rulesets in 2017.8.19.

Option 3 is out of my hands, but I've provided build instructions above. Of course I'm willing to help if you run into any issues.

comment:34 in reply to:  33 Changed 21 months ago by gk

Replying to legind:

With NoScript, much of the code is still in the XPCOM wrapper. That's why it continues to work.

On all pure WebExtensions I've tested in ESR with JavaScript disabled, I've seen the same result. I've tested with uBlock Origin, Disconnect, Ghostery, Privacy Badger. All the same: no functionality, unresponsive.

As I've mentioned the way to fix this is to pick one of the following:

  1. Backport the Firefox 54 fix to Tor Browser
  2. Wait for Firefox to backport the fix, which given their responsiveness lately I doubt will be very effective
  3. Make a new TB release with 5.2.21, removing the update channel and using the new rulesets in 2017.8.19.

Option 3 is out of my hands, but I've provided build instructions above. Of course I'm willing to help if you run into any issues.

Thanks. My current plan is to backport the patch and, after some testing, get a new stable release out (probably next week).

comment:35 Changed 21 months ago by gk

It turns out that the Firefox fix is not working for us as NoScript is blocking things differently. A Tor Browser with the fix for https://bugzilla.mozilla.org/show_bug.cgi?id=1329731 _and_ NoScript disabled works if javascript.enabled is set to false (contrary to a vanilla Firefox 52 ESR). However, once NoScript is enabled we are still stuck.

So I guess we go with an option 4 which I mentioned in comment:15.

comment:36 Changed 21 months ago by gk

Keywords: TorBrowserTeam201708R added; TorBrowserTeam201708 removed
Status: assignedneeds_review

comment:37 in reply to:  36 Changed 21 months ago by cypherpunks

Replying to gk:

Shipping HTTPS-Everywhere based on the new WebExtensions model broke its
functionality on higher security levels as our JavaScript blocking does
not whitelist moz-extension: which is needed.

Shipping NoScript as our JavaScript blocking on higher security levels does not whitelist moz-extension: which is needed and broke functionality of HTTPS-Everywhere based on the new WebExtensions model.

It turns out that vanilla Firefox 52 ESR is affected as well but differently:
Backorting https://bugzilla.mozilla.org/show_bug.cgi?id=1329731 does not help
us as NoScript is regulating JavaScript related things.

Remove as irrelevant.

comment:38 Changed 21 months ago by yawning

Closed #23321 as a duplicate.

comment:39 in reply to:  36 Changed 21 months ago by arthuredelstein

Status: needs_reviewmerge_ready

Replying to gk:

See: https://oniongit.eu/gk/tor-browser-bundle/merge_requests/4 for a patch up for review.

This patch looks good to me.

comment:40 Changed 21 months ago by gk

Resolution: fixed
Status: merge_readyclosed

Thanks. Applied to master and maint-7.0 (commit 32dc8cef283f739bdc045918adbd448c9d78a746 and 35ebe6d5e7f1e33f5cb1b67f0a4022814fd72a15).

cypherpunks: I wanted to keep the Mozilla related part of the commit message as it explains why we did not take that road. Thus, I kept the first part as well.

comment:41 in reply to:  16 Changed 21 months ago by cypherpunks

Replying to gk:

So, the startup delay is not related to the whitelisting but to the new HTTPS-E itself. :/

Confirmed. It creates

│   ├──27.41 MB (19.21%) -- https-everywhere-eff@eff.org
│   │  ├──27.38 MB (19.19%) ++ window-objects/top(moz-extension://06272878-3037-4aa2-bbd3-e0effb78113c/_generated_background_page.html, id=14)

at startup.

comment:42 in reply to:  29 Changed 21 months ago by cypherpunks

Replying to gk:

So, the startup delay is not related to the whitelisting but to the new HTTPS-E itself. :/

They have a couple ideas to resolve a bit that issue, you may want to follow https://github.com/EFForg/https-everywhere/issues/12232

Note: See TracTickets for help on using tickets.