HTTPS Everywhere is not working in Tor Browser nightly

Example URLs:

• http://www.freedomboxfoundation.org/
• http://17th.co/aa

When loading one of those URLs in Tor Browser nightly, we are not redirected to their https version. However, the HTTPS Everywhere add-on can be seen in the list of enabled add-ons.

added component::applications/tor browser owner::legind priority::medium resolution::fixed severity::normal status::closed type::defect labels

After updating the add-on and restarting the browser, it is working again.

The nightly is using the master branch from https://git.torproject.org/https-everywhere.git, however it looks like the master branch on this repository is no longer updated: https://gitweb.torproject.org/https-everywhere.git/ Contrary to the master branch on github: https://github.com/EFForg/https-everywhere

legind: what are your plans here? Do you plan to abandon the Tor Project hosted git repo and use solely Github? Or keep the former just supported with tags as done right now? Or would you use both and keep them in sync?

Trac:
Status: new to needs_information
Cc: N/A to legind

gk: our current plans are to continue pushing the signed tags and developing from master on GitHub. Is the above process of fetching from master part of an overall push to staple the version of HTTPS Everywhere within Tor Browser? If so, a greater degree of coordination is required. We have a plan to support TB moving in this direction, but there are lateral issues that require some cooperative planning.

Well, we only use master for our nightly builds as we thought it would be good to detect HTTPS-E issues (in our context) as early as possible. But that of course is kind of pointless with master on tpo not being in sync with the one on Github. So, I think there are basically three ways to solve this bug:

1. Sync HTTPS-E master on Github with the counterpart on torproject.org
2. Use the latest HTTPS-E release (i.e. the .xpi) in our nightly builds as well
3. Use the HTTPS-E repository on Github for all Tor Browser needs (just using Github for the nightlies is not easily possible)

I'd prefer 1) and am not so happy with 3) while 2) would somehow work for us (although it feels like we could/should do better than that).

What do you think?

At a high level, it's important to know that the rulesets are bundled with the extension itself and not downloaded in some out-of-band manner. If option 3 is used, will the regular extension update mechanism still be operational? I think the priority should be keeping rulesets up to date within Tor Browser, especially since there can be long stretches of time betweeen TB releases. Ideally, HTTPS Everywhere has a new release every two weeks to a month. So we should avoid any of the above options which preclude the possibility of regular updates. Option 2 seems like the least-friction option with regards to this concern, but maybe I'm missing something.

A high priority for us is to move ruleset download and validation towards an out-of-band mechanism that can be implemented within the extension itself. When this transition is complete, there should be no barriers to stapling HTTPS Everywhere versions to the TB.

This ticket is only about fixing the nightly builds, so I think that in options 1 and 3 gk meant using the master branch in the nightly builds, and the tag of the latest version in regular releases. For our regular releases, we want to keep using the latest HTTPS Everywhere release, with extension update mechanism operational (at least concerning the issue we are talking about in this ticket).

We only want to use the master branch for our nightly builds. Those builds are not intended to be used for more than a few days after they are built, so keeping the extension or its rulesets updated does not really matter here. The main use of the nightly build is for testing. We can use the latest HTTPS Everywhere release for that (option 2), but using the master branch would allow us to detect in advance possible issues before the next HTTPS Everywhere version is released.

So I think the main question is whether it's possible to push the master branch to git.torproject.org in addition to github. If it's not possible then we should either switch to github, or stop using the master branch in the nightly and use the latest tag instead.

boklm: unfortunately it wouldn't be possible to keep GitHub and Tor repos' master consistently in sync by pushing to both, since ruleset maintainers often merge PRs and have no access to push to Tor. I could add a step in my release process to push to Tor, but in this case master would just have the same code as an actual HTTPS Everywhere release, and that would kind of defeat the purpose. Alternatively, I could possibly push the latest state of master to the Tor repo on some regular basis (either with cron or manually). Or we could set up a piece of infrastructure that receives a notification via Webhooks https://developer.github.com/webhooks/ upon push, which is tasked with syncing. So there's a few options. I think the easiest right now is for me to manually push to the Tor repo master at some regular interval, and come up with a longer-term solution at some point later.

Replying to legind: syncing. So there's a few options. I think the easiest right now is for me to manually push to the Tor repo master at some regular interval, and come up with a longer-term solution at some point later.

That would be totally fine, thanks!

Trac:
Owner: tbb-team to legind
Status: needs_information to assigned

(To make it clear this ticket would be done if Github HTTPS-E master got pushed to Tor HTTPS-E master once at least as this should unbreak our nightly builds/tests)

Just pushed to Tor HTTPSE master - care to verify the build is passing and close if so?

Looks good, thank!

Trac:
Status: assigned to closed
Resolution: N/A to fixed

closed

mentioned in issue #24179 (moved)

moved to tpo/applications/tor-browser#20585 (closed)