Opened 8 years ago

Closed 6 years ago

#3190 closed defect (fixed)

Failures when rewriting request URLs generated by other extensions

Reported by: Tobu Owned by: pde
Priority: Very High Milestone:
Component: HTTPS Everywhere/EFF-HTTPS Everywhere Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

When I install a userscript that depends on a javascript library whose url HTTPS Everywhere rewrites, Scriptish can't download the dependency. With the Scriptish master branch and the development version of HTTPS Everywhere, I get:

Error loading dependency: http://ajax.googleapis.com/ajax/libs/jquery/1.3/jquery.min.js
Error! Server returned: nothing (timed out)

Greasemonkey 0.9.3 also fails, but silently.

First reported here: https://github.com/erikvold/scriptish/issues/282
The download code is over there: https://github.com/erikvold/scriptish/blob/master/extension/modules/script/scriptdownloader.js

Child Tickets

TicketStatusOwnerSummaryComponent
#3073closedpdeHTTPS everywhere fails to handle X-ICON resources.HTTPS Everywhere/EFF-HTTPS Everywhere
#5682closedpdehttps-everywhere-uri-rewrite event not being sent in 2.0.x?HTTPS Everywhere/EFF-HTTPS Everywhere
#5912closedgraffatcolmingovMyWOT extension breakageHTTPS Everywhere/EFF-HTTPS Everywhere
#6199closedpdeWrite docs for other extension developers on how to be HTTPS Everywhere compatibleHTTPS Everywhere/EFF-HTTPS Everywhere
#6499closedpdezotero ACM Digital Library translator fails when HTTPS Everywhere is enabledHTTPS Everywhere/EFF-HTTPS Everywhere
#7423closedpdeHTTPS Everywhere breaks the StumbleUpon pluginHTTPS Everywhere/EFF-HTTPS Everywhere
#7769closedpdeInteraction with Free Download ManagerHTTPS Everywhere/EFF-HTTPS Everywhere

Change History (30)

comment:1 Changed 8 years ago by Tobu

Summary: Breaks download of userscript dependencies by ScriptishBreaks download of userscript dependencies by Greasemonkey and Scriptish

comment:2 Changed 8 years ago by pde

Status: newaccepted

Anyone up for wading into this bug? I would start with the JavaScript debugger, as well as setting LogLevel to 1 or 2 in about:config and trying to work out where things are going wrong...

comment:3 Changed 8 years ago by effuser

This basically blocks the browser at least on Mac Snow Leopard/Scriptish Latest and the browser needs to be force killed. The priority for this should be raised.

comment:4 Changed 8 years ago by effuser

See https://github.com/scriptish/scriptish/issues/441 for more details on Scriptish end as the original 'First reported here' page has disappeared.

comment:5 Changed 8 years ago by pde

Even the more recent Scriptish bug report link is broken for me now.

comment:6 Changed 8 years ago by pde

Keywords: wont-fix-here added

The problem here is that we have to replace the channel that Scriptish / Greasemonkey is working with, but S/G doesn't realise the channel has been replaced out from underneath it.

Until Firefox has a sane, consistent request rewriting policy, I'm not sure whether there is an elegant solution to this problem. There is, however, an inelegant solution that S/G could implement, which was built by jsamuel for RequestPolicy:

https://www.requestpolicy.com/dev/ticket/115

comment:7 Changed 8 years ago by pde

s/The problem here/I believe the problem here/

(this is still just a hypothesis)

comment:8 Changed 8 years ago by pde

Keywords: workaround-available added
Priority: normalmajor
Summary: Breaks download of userscript dependencies by Greasemonkey and ScriptishFailures when rewriting request URLs generated by other extensions

comment:10 Changed 8 years ago by pde

Priority: majorcritical

comment:11 Changed 8 years ago by pde

A note for any other Extension developers that may be reading here:

The "https-everywhere-uri-rewrite" event that we currently send lists the (before,after) URLs from the rewrite. We could probably replace that with an event that sends the post-rewrite nsIHTTPChannel, which is much more elegant if you were trying to obtain data from the channel :).

If anyone lets us know that they would like such code from us, we can include it and push it to our userbase fairly quickly.

comment:12 Changed 8 years ago by pde

I think we have a fix in the works for this bug. Can someone give me some examples of UserScripts for Scriptish/Greasemonkey that try to fetch jquery from Google over http, so I can test it?

comment:13 Changed 8 years ago by cypherpunks

I have had issues with XMLHttpRequests from Greasemonkey with HTTPS Everywhere. My script https://www.userscripts.org/scripts/review/100840 did not work until I put the "https" into the gdata API URL. Feel free to test with that.

Additionally, I had issues with the NewsFox RSS reader extension (not fetching new articles anymore on some feeds) with HTTPS Everywhere enabled.

comment:14 in reply to:  12 ; Changed 8 years ago by mikeperry

Replying to pde:

I think we have a fix in the works for this bug. Can someone give me some examples of UserScripts for Scriptish/Greasemonkey that try to fetch jquery from Google over http, so I can test it?

pde: The fix the one Giorgio mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=677643#c75 right? Ie, we basically expect this bug to get knocked out by #4175, pending a new NoScript release, yes?

comment:15 in reply to:  14 Changed 8 years ago by pde

Replying to mikeperry:

pde: The fix the one Giorgio mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=677643#c75 right? Ie, we basically expect this bug to get knocked out by #4175, pending a new NoScript release, yes?

Yes, and I have that committed in a private repo but I just wanted to confirm that it actually fixes the bug before pushing it.

comment:16 Changed 8 years ago by pde

Thanks for the test case cypherpunks. Sadly, what it's telling me is that Giorgio's patch doesn't fix this bug (or possibly, doesn't fix all instances of this bug) :( :( :(

comment:17 Changed 8 years ago by pde

I just tried to reproduce this bug with either LastPass or StumbleUpon (to see if Giorgio's patch might fix either of those cases), but I just can't reproduce. Perhaps the bug reports caused these extensions to switch over to HTTPS themselves?

comment:18 Changed 8 years ago by pde

A version of HTTPS Everywhere with Giorgio's patch is in the extension-compatibility branch of my remote, git://gitweb.torproject.org/pde/https-everywhere.git

(https://gitweb.torproject.org/pde/https-everywhere.git/shortlog/refs/heads/extension-compatibility )

comment:19 Changed 8 years ago by pde

I can also get a clean reproduction with NewsFox, subscribing to an RSS feed like http://www.economist.com/feeds/print-sections/68/letters.xml with the Economist ruleset enabled.

Giorgio's IOUtils patch isn't helping in that case either :(

comment:20 in reply to:  18 Changed 8 years ago by mikeperry

Replying to pde:

A version of HTTPS Everywhere with Giorgio's patch is in the extension-compatibility branch of my remote, git://gitweb.torproject.org/pde/https-everywhere.git

(https://gitweb.torproject.org/pde/https-everywhere.git/shortlog/refs/heads/extension-compatibility )

Hrmm.. We probably want to just merge with NoScript 2.1.5.. 2.1.5 involves a file rename, and some other refactoring as well, so it might be nice to try to keep it as simple as possible to deal with. I believe it also includes this fix.

comment:21 Changed 7 years ago by pde

This commit is an workaround for much of the breakage experienced by Greasemonkey and Scriptish users.  However it leaves users installing these scripts over HTTP and I wonder whether we shouldn't just disable the ruleset altogether until we can get a more secure workaround into those extensions :(.

comment:22 Changed 7 years ago by pde

We hope that a lot of the extension incompatibility bugs will be fixed if/when Mozilla accepts our patches for this ticket: https://bugzilla.mozilla.org/show_bug.cgi?id=765934

At least, extensions that use XMLHTTPRequest or nsIHTTPChannel.asyncOpen() should be fixed that way... nsIHTTPChannel.open() will require more work.

comment:23 Changed 7 years ago by mikeperry

The next TBB 2.3.x-alpha should ship with that Firefox patch already applied, and it is on the HTTPS-Everywhere 3.0 dev channel as well. Might be a useful early testbed...

comment:24 Changed 6 years ago by pde

What is hopefully a fix for this bug has landed in mozilla-central and shipped in today's Firefox nightly build. We now need to test whether each of the sub-instances of this bug has been resolved from today's build of FF 21 onwards.

comment:25 Changed 6 years ago by pde

Keywords: wont-fix-here workaround-available removed

comment:26 Changed 6 years ago by schoen

I just did a test using 32-bit Firefox Nightly 2013-02-01 built from
http://hg.mozilla.org/mozilla-central/rev/50cf5bbcb180, with NewsFox 1.0.8.4.1.1 and HTTPS Everywhere 4.0development.5.

I subscribed to the insecure version of the Google News feed at http://news.google.com/news?pz=1&cf=all&ned=us&hl=en&topic=h&num=3&output=rss and refreshed the feed. The feed was successfully populated with articles, and Wireshark showed that the connection was being made over HTTPS, not HTTP. (I also checked that this URL would have been rewritten by HTTPS Everywhere, and that the rewritten and non-rewritten versions both contain RSS content.)

Looks good!

comment:27 Changed 6 years ago by pde

We got the fix into Firefox 20 (currently Aurora) but it seems like we won't get to Firefox 19 or earlier, including the ESR release.

comment:28 Changed 6 years ago by pde

I put my test case for this back online: http://reworld.org/old/bugs/5477-tst.html

I agree that we're good now. Time to close all of these bugs!

comment:29 Changed 6 years ago by pde

With this bug closing and FF 20 released, I think we could consider experimenting with reenabling rulesets for Scriptish, Greasemonkey, MyWOT, StumbleUpon and soforth. At least in the development branch...

comment:30 Changed 6 years ago by pde

Resolution: fixed
Status: acceptedclosed

Ding dong.

Note: See TracTickets for help on using tickets.