Updates of HTTPS-Everywhere we ship do not seem to update the rulesets
A user on the blog (https://blog.torproject.org/comment/279696#comment-279696) noticed that updates we ship do not seem to update the rulesets. While the HTTPS-Everywhere version is updated, the rulesets version shown by clicking on the HTTPS-Everywhere icon is still the old one.
While this issue goes away with the regular rulesets update in the background, it seems weird that the rulesets do not seem to get updated in the first place.
Activity
gk: The extension checks for new rulesets when the browser starts, and subsequently at 24 hour intervals. My guess is that the tor daemon has not finished bootstrapping when the initial check takes place, and thus it fails for the first try.
Changing the periodicity to 30 seconds instead of every day, after a minute or so the rulesets do update to the latest version:
cd tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default/extensions mkdir https-everywhere unzip -qd https-everywhere https-everywhere-eff@eff.org.xpi sed -i 's/periodicity = 86400;/periodicity = 30;/g' https-everywhere/background-scripts/update.js rm https-everywhere-eff@eff.org.xpi cd https-everywhere/ zip -qr ../https-everywhere-eff@eff.org.xpi ./* cd .. && rm -rf https-everywhereKeep in mind that rulesets are still bundled with the extension itself, so Tor Browser is still protected with whatever rulesets were present at the time of the last full extension release.
Is there any way that you can notify HTTPS Everywhere when the tor bootstrapping process has completed? I could create a hook for this event.
Replying to legind:
gk: The extension checks for new rulesets when the browser starts, and subsequently at 24 hour intervals. My guess is that the tor daemon has not finished bootstrapping when the initial check takes place, and thus it fails for the first try.
Changing the periodicity to 30 seconds instead of every day, after a minute or so the rulesets do update to the latest version:
{{{ cd tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default/extensions mkdir https-everywhere unzip -qd https-everywhere https-everywhere-eff@eff.org.xpi sed -i 's/periodicity = 86400;/periodicity = 30;/g' https-everywhere/background-scripts/update.js rm https-everywhere-eff@eff.org.xpi cd https-everywhere/ zip -qr ../https-everywhere-eff@eff.org.xpi ./* cd .. && rm -rf https-everywhere }}}
Keep in mind that rulesets are still bundled with the extension itself, so Tor Browser is still protected with whatever rulesets were present at the time of the last full extension release.
Is there any way that you can notify HTTPS Everywhere when the tor bootstrapping process has completed? I could create a hook for this event.
Okay, to be clear the "Rulesets version for EFF (full)" item behind the HTTPS Everywhere button is bound to the ruleset update done by the extension? That is, as long as no update check (successfully) happened that UI piece does not update even though all the rulesets are there due to the extensions update?
Regarding the notification I have to think a bit about that. That might get easier after we have transitioned away from our legacy extensions and included both Tor Launcher and Torbutton into the browser itself.
Okay, to be clear the "Rulesets version for EFF (full)" item behind the HTTPS Everywhere button is bound to the ruleset update done by the extension? That is, as long as no update check (successfully) happened that UI piece does not update even though all the rulesets are there due to the extensions update?
Yes, this is correct. We do extension updates less often than ruleset updates, so the rulesets bundled with the extension may be a little out of date, but even if there is no "Rulesets version for EFF (full)" item displayed, there will be rulesets in effect.
-
Replying to legind:
Okay, to be clear the "Rulesets version for EFF (full)" item behind the HTTPS Everywhere button is bound to the ruleset update done by the extension? That is, as long as no update check (successfully) happened that UI piece does not update even though all the rulesets are there due to the extensions update?
Yes, this is correct. We do extension updates less often than ruleset updates, so the rulesets bundled with the extension may be a little out of date, but even if there is no "Rulesets version for EFF (full)" item displayed, there will be rulesets in effect.
Okay, so the item is meant to display the last date when those rulesets got fetched? Or is it meant to show the version of the currently active rulesets? My intuition (and that of the user who reported that bug) is the latter, because that's what I as a user care about. It seems to me then that the item should not "lie" to the user regardless whether the check on start-up is succeeding or not (it might in fact even fail outside of Tor Browser/unrelated to it for various reasons).
Extension update checks are not on the same schedule as ruleset update checks, so what happens if you update your extension (because there is a new version available) but you already had newer rulesets installed? Do those newer rulesets get overwritten by the ones shipped with the extension?
Okay, so the item is meant to display the last date when those rulesets got fetched? Or is it meant to show the version of the currently active rulesets? My intuition (and that of the user who reported that bug) is the latter, because that's what I as a user care about. It seems to me then that the item should not "lie" to the user regardless whether the check on start-up is succeeding or not (it might in fact even fail outside of Tor Browser/unrelated to it for various reasons).
Yes, it's the latter. I don't quite understand your point about the extension lieing to the user. If the rulesets have been downloaded and applied, it displays it. If not, it will not display the ruleset line.
Extension update checks are not on the same schedule as ruleset update checks, so what happens if you update your extension (because there is a new version available) but you already had newer rulesets installed? Do those newer rulesets get overwritten by the ones shipped with the extension?
Out-of-band downloaded rulesets, if present, will always take precedence over extension-bundled rulesets. We also always make a ruleset release on the same day as an extension release, to minimize the cases where a user has an old out-of-band ruleset and a newer extension.
-
Replying to legind:
Okay, so the item is meant to display the last date when those rulesets got fetched? Or is it meant to show the version of the currently active rulesets? My intuition (and that of the user who reported that bug) is the latter, because that's what I as a user care about. It seems to me then that the item should not "lie" to the user regardless whether the check on start-up is succeeding or not (it might in fact even fail outside of Tor Browser/unrelated to it for various reasons).
Yes, it's the latter. I don't quite understand your point about the extension lieing to the user. If the rulesets have been downloaded and applied, it displays it. If not, it will not display the ruleset line.
Yes, but lying in the sense that the date is wrong as in the bug report. The user opened an old Tor Browser, updated and was confused that the old date was still shown in the display suggesting that the rules from that date were used while the update brought a new HTTPS-Everywhere version with the new rules. I think the expected behavior would be: show the date of applied rulesets either those that got downloaded (as is the case now) OR those that we got by a new HTTPS Everywhere version (which is not happening right now). The request on start-up might not get through for whatever reason...
Yes, but lying in the sense that the date is wrong as in the bug report. The user opened an old Tor Browser, updated and was confused that the old date was still shown in the display suggesting that the rules from that date were used while the update brought a new HTTPS-Everywhere version with the new rules. I think the expected behavior would be: show the date of applied rulesets either those that got downloaded (as is the case now) OR those that we got by a new HTTPS Everywhere version (which is not happening right now). The request on start-up might not get through for whatever reason...
To avoid the problem of old rulesets still applying even when a new extension version is released, the simplest thing to do is probably to just clear the out-of-band rulesets from storage upon first load of any new version. This will ensure that the extension-bundled rulesets are used upon each extension upgrade. How does this sound?
-
Replying to legind:
Yes, but lying in the sense that the date is wrong as in the bug report. The user opened an old Tor Browser, updated and was confused that the old date was still shown in the display suggesting that the rules from that date were used while the update brought a new HTTPS-Everywhere version with the new rules. I think the expected behavior would be: show the date of applied rulesets either those that got downloaded (as is the case now) OR those that we got by a new HTTPS Everywhere version (which is not happening right now). The request on start-up might not get through for whatever reason...
To avoid the problem of old rulesets still applying even when a new extension version is released, the simplest thing to do is probably to just clear the out-of-band rulesets from storage upon first load of any new version. This will ensure that the extension-bundled rulesets are used upon each extension upgrade. How does this sound?
Sounds good to me. What happens to the shown date after "Rulesets version for EFF (full)" in that case? I guess, ideally, it would reflect the extension release date as that's the date of the active rulesets (until a new out-of-band ruleset update happens).
Sounds good to me. What happens to the shown date after "Rulesets version for EFF (full)" in that case? I guess, ideally, it would reflect the extension release date as that's the date of the active rulesets (until a new out-of-band ruleset update happens).
In that case only the "Version: " line would appear. The ruleset line would not appear, since no ruleset bundle would have been downloaded separately.
I understand this is kind of confusing, since it makes it appear as though users aren't being protected by the extension when they actually are. We may want to change the logic to display this line in any case, even if no ruleset bundle has been downloaded yet.
Here's a more detailed explainer on ruleset updates: https://github.com/EFForg/https-everywhere/blob/master/docs/en_US/ruleset-update-channels.md
-
Replying to legind:
Sounds good to me. What happens to the shown date after "Rulesets version for EFF (full)" in that case? I guess, ideally, it would reflect the extension release date as that's the date of the active rulesets (until a new out-of-band ruleset update happens).
In that case only the "Version: " line would appear. The ruleset line would not appear, since no ruleset bundle would have been downloaded separately.
I understand this is kind of confusing, since it makes it appear as though users aren't being protected by the extension when they actually are. We may want to change the logic to display this line in any case, even if no ruleset bundle has been downloaded yet.
Yeah, that's exactly the issue. And, I agree, your idea would be a good solution (much better than just the "Version: " line appearing as this could confuse users again ("Huh, where are my rulesets gone after the latest extension update? The rulesets version is not shown anymore now!1!")).
This has been fixed in
https://github.com/EFForg/https-everywhere/pull/17553/ https://github.com/EFForg/https-everywhere/pull/17551/
Trac:
Resolution: N/A to fixed
Status: new to closed