Opened 7 months ago

Closed 6 months ago

#25603 closed enhancement (fixed)

Update Orfox HTTPS-E Add-on

Reported by: sysrqb Owned by: sysrqb
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-mobile, TorBrowserTeam201805R
Cc: gk, igt0 Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Orfox bundles NoScript and HTTPS-Everywhere as distribution extensions. The HTTPS-E version is old. Orfox contains NoScript Anywhere (NSA) 3.5a15, but it seems like this is still the latest version. We'll start using NoScript when we start using the WebExtensions version.

Orfox uses HTTPS-Everywhere 5.2.20, so (at a minumum) we should bump this to 5.2.21 [0] (as gk mentioned in #19675 [1]). We should also test the WebExtensions version and decide if we should simply upgrade to version 2018.3.13 (like Desktop).

I think we should continue using the Orfox distribution, and we can move to a better solution during the rebase (and when we move to rbm).

Orfox includes the tor-browser-settings [2] add-on, too, but that hasn't changed.

[0] https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/versions/?page=1#version-5.2.21
[1] https://trac.torproject.org/projects/tor/ticket/19675#comment:20
[2] https://git.synz.io/synzvato/tor-browser-settings.git

Child Tickets

Change History (13)

comment:1 Changed 7 months ago by gk

Keywords: TorBrowserTeam201803 added

comment:2 Changed 6 months ago by gk

Keywords: TorBrowserTeam201804 added; TorBrowserTeam201803 removed

Moving our tickets to April.

comment:3 Changed 6 months ago by gk

Priority: HighVery High

comment:4 Changed 6 months ago by sysrqb

Owner: changed from tbb-team to sysrqb
Status: newassigned

Unfortunately this is not working as expected.

In addition to #25659, during testing I am seeing https-everywhere successfully load, but then it doesn't rewrite any URLs. I also tested using an older version of Orfox (based on 52.2.0), and it bundles https-everywhere 5.2.20 (tagged on 05 Jul, 2017). This combination of https-everywhere+Orfox works as expected, and the URLs are rewritten. If I leave Orfox open for some time, https-everywhere automatically updates to version 2018.4.11 (webextension). After restarting the app, I see the add-on loads, but it doesn't perform any URL rewrites. I see some messages in the log about an error processing the add-on's manifest, but it seems like the extension is loaded successfully.

I/Gecko   (24006): 1523904137200        addons.xpi      DEBUG   getModTime: Recursive scan of https-everywhere-eff@eff.org
I/Gecko   (24006): 1523904137200        DeferredSave.extensions.json    DEBUG   Starting timer
I/Gecko   (24006): 1523904137500        DeferredSave.extensions.json    DEBUG   Starting write
I/Gecko   (24006): 1523904137700        addons.webextension.https-everywhere-eff@eff.org        WARN    Loading extension 'https-everywhere-eff@eff.org': Reading manifest: Error processing permissions.3: Unknown permission "tabs"
I/Gecko   (24006): 1523904137700        addons.webextension.https-everywhere-eff@eff.org        WARN    Loading extension 'https-everywhere-eff@eff.org': Reading manifest: Error processing browser_action: An unexpected property was found in the WebExtension manifest.
I/Gecko   (24006): 1523904137700        addons.webextension.https-everywhere-eff@eff.org        WARN    Loading extension 'https-everywhere-eff@eff.org': Reading manifest: Error processing devtools_page: An unexpected property was found in the WebExtension manifest.
I/Gecko   (24006): 1523904137700        DeferredSave.extensions.json    DEBUG   Write succeeded

I see these same messages when I bundle https-everywhere 2018.03.13.

Next, I'll try bundling https-e 5.2.20 with the newest Orfox version, because I'm curious if that reliably loads and if it rewrites URLs. If we can't get https-everywhere loading or working reliably, I'd rather remove it from Orfox in the next release. I don't want to give the false impression https-everywhere is loaded and working - when it isn't working in reality.

comment:5 Changed 6 months ago by sysrqb

For reference, I am specifically testing whether loading http://deb.torproject.org is rewritten to https://deb.torproject.org. This does not happen automatically in Firefox because it is not forced on the server-side, but it does happen in Tor Browser. I also tested in Tor Browser with https-everywhere disabled, and the request is not redirected.

Also, I see similar (but not the same) load errors occur on the desktop, too.

1523974564000	addons.webextension.{73a6fe31-595d-460b-a920-fcc0f8843232}	WARN	Loading extension '{73a6fe31-595d-460b-a920-fcc0f8843232}': Reading manifest: Error processing permissions.1: Unknown permission "privacy"
1523974564000	addons.webextension.{73a6fe31-595d-460b-a920-fcc0f8843232}	WARN	Loading extension '{73a6fe31-595d-460b-a920-fcc0f8843232}': Reading manifest: Error processing permissions.4: Unknown permission "unlimitedStorage"
Apr 17 14:16:04.000 [notice] Bootstrapped 85%: Finishing handshake with first hop
1523974564400	addons.webextension.https-everywhere-eff@eff.org	WARN	Loading extension 'https-everywhere-eff@eff.org': Reading manifest: Error processing devtools_page: An unexpected property was found in the WebExtension manifest.

Both desktop and Android use the same extensions code for parsing and loading, so I'm not sure why there is a difference.

comment:6 Changed 6 months ago by gk

Cc: gk added

Does HTTPS-E work for you in a vanilla Firefox for Mobile?

comment:7 in reply to:  6 ; Changed 6 months ago by sysrqb

Replying to gk:

Does HTTPS-E work for you in a vanilla Firefox for Mobile?

Yes, 2018.4.11 works with Fennec 59.0.2. I don't have a vanilla Fennec 52.7.3 already built, so I'll need to build that and test this.

comment:8 in reply to:  7 Changed 6 months ago by sysrqb

Replying to sysrqb:

Replying to gk:

Does HTTPS-E work for you in a vanilla Firefox for Mobile?

Yes, 2018.4.11 works with Fennec 59.0.2. I don't have a vanilla Fennec 52.7.3 already built, so I'll need to build that and test this.

It doesn't work with a fresh Fennec 52.7.3. I installed https-everywhere 2018.4.11 from AMO, and it does not rewrite the URL. I'll focus on testing Orfox with https-everywhere 5.2.21. Maybe we can ship it with an updated ruleset.

comment:9 Changed 6 months ago by sysrqb

Oh, and note-to-self. We should disable auto-update for this extension, else Orfox will download the new version and it won't rewrite any URLs.

comment:10 Changed 6 months ago by sysrqb

Process for creating new https-everywhere xpi:

1) Checkout tag 2018.4.11
2) Copy src/chrome/content/rules into another directory outside working directory (cp -r src/chrome/content/rules ../)
3) Checkout tag 5.2.21
4) Edit src/install.rdf - clear updateURL and updateKey [0] (this prevents auto-updating)
5) Delete directory src/META-INF/ (rm -r src/META-INF)
6) Delete all current rules (rm -r src/chrome/content/rules/)
7) Copy new rules into working directory (cp -r ../rules src/chrome/content/)
8) ./makexpi.sh
9 sha256sum pkg/https-everywhere-5.2.21~4c7803208b-dirty-eff.xpi is

7a33c13dbd80fd881b1508fca6dc10fca787f8eb4da754104321537240ffb866  pkg/https-everywhere-5.2.21~4c7803208b-dirty-eff.xpi

10) Copy pkg/https-everywhere-5.2.21~4c7803208b-dirty-eff.xpi into tor-browser/mobile/android/orfox/distribution/assets/distribution/extensions/https-everywhere-eff@eff.org.xpi

[0] Diff from step (4)

diff --git a/src/install.rdf b/src/install.rdf
index fe321f48b0..6fa8ece9c7 100644
--- a/src/install.rdf
+++ b/src/install.rdf
@@ -15,8 +15,8 @@
         <em:optionsURL>chrome://https-everywhere/content/observatory-preferences.xul</em:optionsURL>
         <em:iconURL>chrome://https-everywhere/skin/icon-active-48.png</em:iconURL>
         <em:unpack>false</em:unpack>
-        <em:updateURL>https://www.eff.org/files/https-everywhere-eff-update-2048.rdf</em:updateURL> <!-- 2015-08-14: New update URL to go with new id (https-everywhere-eff@ef.org) -->
-        <em:updateKey>MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6MR8W/galdxnpGqBsYbqOzQb2eyW15YFjDDEMI0ZOzt8f504obNs920lDnpPD2/KqgsfjOgw2K7xWDJIj/18xUvWPk3LDkrnokNiRkA3KOx3W6fHycKL+zID7zy+xZYBuh2fLyQtWV1VGQ45iNRp9+Zo7rH86cdfgkdnWTlNSHyTLW9NbXvyv/E12bppPcEvgCTAQXgnDVJ0/sqmeiijn9tTFh03aM+R2V/21h8aTraAS24qiPCz6gkmYGC8yr6mglcnNoYbsLNYZ69zF1XHcXPduCPdPdfLlzVlKK1/U7hkA28eG3BIAMh6uJYBRJTpiGgaGdPd7YekUB8S6cy+CQIDAQAB</em:updateKey>
+        <em:updateURL></em:updateURL> <!-- 2015-08-14: New update URL to go with new id (https-everywhere-eff@ef.org) Deleted for Orfox -->
+        <em:updateKey></em:updateKey> <!-- Deleted for Orfox -->
         <!-- Firefox -->
         <em:targetApplication>
             <Description>

comment:11 Changed 6 months ago by sysrqb

Cc: igt0 added
Status: assignedneeds_review

After #25980 is merged, I have a branch ready for this.

The top commit on branch bug25980+bug25603 is the update, the branch is based on the fix for #25980.

I wish we had a better way of providing the source for the bundled https-everywhere xpi in Orfox. I can create a public repo somewhere. The steps documented in comment:10 are exactly how I created this xpi. The most important step is deleting the updateURL (but leaving the <em:updateURL> tags). We must delete the manifest because the signature is no longer valid. I know, this is sad.

https://git.torproject.org/user/sysrqb/tor-browser.git

comment:12 Changed 6 months ago by gk

Keywords: TorBrowserTeam201805R added; TorBrowserTeam201804 removed

comment:13 in reply to:  11 Changed 6 months ago by gk

Resolution: fixed
Status: needs_reviewclosed

Replying to sysrqb:

After #25980 is merged, I have a branch ready for this.

The top commit on branch bug25980+bug25603 is the update, the branch is based on the fix for #25980.

I wish we had a better way of providing the source for the bundled https-everywhere xpi in Orfox. I can create a public repo somewhere. The steps documented in comment:10 are exactly how I created this xpi. The most important step is deleting the updateURL (but leaving the <em:updateURL> tags). We must delete the manifest because the signature is no longer valid. I know, this is sad.

I think we should not spend our time fixing this problem as this is supposed to go away with transitioning to Tor Browser for Android.

The patch looks good. I cherry-picked it onto tor-browser-52.7.3esr-8.0-1 (commit e3effb933616512282e0047d56ed0087cce702a6).

Note: See TracTickets for help on using tickets.