Officially support onions in HTTPS-Everywhere

added TorBrowserTeam202004R actualpoints::18.5 component::applications/tor browser https-everywhere network-team-roadmap-2020Q1 network-team-roadmap-november owner::tbb-team parent::30029 points::20 priority::medium resolution::fixed reviewer::antonela reviewer::mcs reviewer::sysrqb severity::normal sponsor::27-must status::closed tor-hs type::defect ux-team labels

Trac:

Trac:
Sponsor: N/A to Sponsor27-must

Closing #1670 (moved) and #19812 (moved) as duplicate of this ticket. I think the onion rulesets idea fits nicely with their requests.

Trac:
Cc: N/A to ilf, bee

Trac:
Cc: ilf, bee to ilf, bee, fdsfgs@krutt.org

Trac:
Points: N/A to 20

Trac:
Parent: N/A to #30029 (moved)

Trac:
Description: Onion rulesets can currently be encoded in HTTPS-Everywhere, but this is not really officially supported by the UX.

We should work forward into providing onion rulesets as update channels for HTTPS-Everywhere.

to

The plan:

A major UX issue for onion services is their huge addresses. We want to fix this issue because an address with 56 random characters confuses people, it makes it harder to pass the address around, and it also makes it much harder to verify it.

There is a field of literature called "secure name systems" but none of the candidates are good enough for us right now. Hence, we present a hotfix that might offer a situational relief for users for the medium-term future, until we come up with something better, or while we experiment with more solutions. I suggest we keep this ticket focused to this idea, instead of debating why this and not that since we've already been doing this for far too long.

The plan is to use the HTTPS-Everywhere extension that we already have in Tor Browser, and encourage people to write their own rulesets for onions. We are talking about community-maintained rulesets and nothing that is officially maintained by The Tor Project or by HTTPS-Everywhere. This ticket is about making it easier for people to create, import and use this rulesets. We are talking about UI/UX improvements, writing blog posts and doing Q&A.

Here are some example of community rulesets we can imagine:

The SecureDrop ruleset: where securedrop makes a ruleset with their whole directory. People can download that to quickly visit securedrop destinations, by going to securedrop-nyt.tor.onion .
The Torproject ruleset: where torproject makes a ruleset with all their onions. We developers can use that to quickly visit Tor sites over onion, by going to tor-trac.tor.onion instead of remembering the onion.
The Bitcoin ruleset: where a "trusted" bitcoin entity publishes a ruleset with various cryptocurrency-related rules that allow people to quickly visit them.

This approach has both positives and negatives (I assure you this is the case with every "secure naming" project out there):

Positives: Good security if the ruleset is taken from a trusted source. No state keeping. Reachable engineering effort. No global names, hence no fear of name squatting. Easy to understand tradeoffs.
Negatives: Terrible security if the ruleset is evil. No global names: If you want people to use your shorten onion name, you need to persuade them to use your ruleset.

Here are some HTTPS-Everywhere issues we need to solve based on my Mexico notes:

Be able to stop update channels per-channel.
Need good UI to easily look and understand rules.
Need to implement file extension to install ruleset with one-click from web button.

Here are some issues we need to think about:

We need good user text to make sure that people don't shoot themselves in the foot too often by installing bad rulesets and whatnot (they already do it daily when they open onions from "search enginers" or reddit).
Which tld to use? If we use .tor we open ourselves to DNS leaks in normal browsers. If we use .tor.onion that might be confusing to people.
Are there any issues with SSL?

More resources:

https://trac.torproject.org/projects/tor/wiki/org/meetings/2018MexicoCity/Notes/OnionV3ux https://trac.torproject.org/projects/tor/wiki/org/meetings/2018MexicoCity/Notes/HTTPSEverywhereNotes https://blog.torproject.org/cooking-onions-names-your-onions

Trac:
Cc: ilf, bee, fdsfgs@krutt.org to ilf, bee, fdsfgs@krutt.org, gaba

Trac:
Keywords: tor-hs https-everywhere tor-ux deleted, tor-hs https-everywhere tor-ux network-team-roadmap-november added

Here are some notes from the plans we made in Stockholm in the meeting between me, antonela, sysrqb, redshiftzero, geko and dgoulet:

Scope of work:

First iteration will include onion rules for securedrop websites (e.g. nytimes.securedrop.tor.onion -> nyttips4bmquxfzw.onion)
Need to add a toolbar button in the ffox UI to show that a redirect happened
Rewrite URL in URL bar (only show the human-readable url)
Add support for viewing rulesets (?)
See how update channels work and whether we should disable them or not.

Out of scope:

First iteration will not allow people to easily add their own rules

TLD scheme:

Three options for tld scheme:

a) nytimes.securedrop.onion (ambiguous and probably unsafe) b) nytimes.securedrop.tor.onion (safe but bad UX) c) nytimes.securedrop.tor (good UX but DNS leaks in other browsers)

We decided to ditch (a) from our options and do either (b) or (c). (b) is the safest and we should probably roll with that (?).

FPF plan:

FPF will change their securedrop directory to include ".tor.onion" links for their various instances.

Metadata:

3 months of work are enough

Putting this on the browser team's plate.

Trac:
Keywords: tor-hs https-everywhere tor-ux network-team-roadmap-november deleted, tor-hs https-everywhere tor-ux network-team-roadmap-november TorBrowserTeam202001 added

Some thoughts after thinking a bit how to implement this.

Supporting local redirects from human-memorable addresses (.tor.onion -> .onion) can be done with https-everywhere extension as is, either by adding a 3rd-party update channel or by adding custom rulesets in debug mode (the latter is less usable I assume). With respect to update channels, it should not be difficult to modify https-everywhere to allow enabling/disabling automatic updates per channel and not globally as it is currently implemented, and this is something that might be interesting also for non Tor Browser https-everywhere users (although I don't think many people have more update channels than the default one).

Then we need to show the human-memorable URL in the urlbar, plus possibly some visual feedback indicating that there was some kind of local redirect (I guess the concrete UX solution still needs to be decided). This has to be implemented in Tor Browser, since AFAIK there is no way to achieve that using a WebExtension like https-everywhere only. There are several ways this can be implemented from the Tor Browser side, but at the end of the day we need some way to know whether for the current .onion page we have some .tor.onion human-memorable "alias" that the user trusts.

With the current implementation, it would be difficult from Tor Browser side to ask https-everywhere questions like: is there any .onion mapping for this .tor.onion domain the user just entered? So I think the simplest is to observe .tor.onion -> .onion redirects and assume that when this happens it's because the original .tor.onion is a human-memorable alias for the corresponding .onion that the user trusts. Observing http requests like that should be quite simple to do from Tor Browser side, and this implementation would work independently of https-everywhere: we just observe .tor.onion -> .onion redirects, but we don't need to know how or by whom these redirects were done.

A few questions/issues regarding showing short .tor.onion in urlbar:

Whenever there is a .tor.onion -> .onion redirect, should we display the full original URL in the urlbar, or just replace the hostname for the human-memorable one in the final URL? For example, let's say we observe a redirect from http://nytimes.securedrop.tor.onion to https://www.nyttips4bmquxfzw.onion (upgrade to https + translate from .tor.onion + adding www). In this case I think it would not make much sense to show the original URL, but just display https://www.[nytimes.securedrop.tor.onion] (and if we follow the solution I mentioned, show https://www.nyttips4bmquxfzw.onion when user clicks on it).
What should we do when user keeps navigating in subpages of some .onion, given the fact that the .tor.onion -> .onion is only observed in the first navigation? Should we show the human-memorable .tor.onion in the urlbar for that tab until the user navigates away from the underlying .onion? Do we also need to show .tor.onion for links opened in a new tab from there?
Do we need to show .tor.onion when user navigates directly to a .onion page for which we have some .tor.onion "rule"? (mentioned by sysrqb in a IRC conversation)

If we were to implement 3) I think we would need to be able to ask https-everywhere for some kind of reverse lookup to obtain a .tor.onion for a given .onion hostname, so the current approach of just observing http redirects from Tor Browser would not work.

Note that if we had an explicit list of .tor.onion -> .onion pairs indicating this mapping (like a /etc/hosts file) we would not need to be looking for redirects to infer the .tor.onion -> .onion relationship, and doing a "reverse lookup" (finding a .tor.onion that corresponds to some .onion) would be trivial. I also think that designing a UI to view/edit .tor.onion rules/mappings would be much easier than doing one for https-everywhere rules, since https-everywhere rulesets are more powerful (and more complicated) than just .tor.onion -> .onion mappings. So, depending on what we want to support, especially if we want to do UI to view/edit rules, I would consider implementing the full feature in Tor Browser directly instead of partially using https-everywhere.

So I would suggest discussing/deciding about the Add support for viewing rulesets (?) feature, as I'm not sure how it could look like if we are going to do this with https-everywhere.

== Prototype

For now to move forward my plan is to implement some UI to replace .onion by .tor.onion in the urlbar, by observing request redirects as I mentioned before. I know the concrete UX still needs to be decided, but I'll try to implement the urlbar logic and I think most of this code will be needed independently of the final solution. I'll try to show .tor.onion in the urlbar whenever there is a redirect from .tor.onion and as long as the user does not navigate away from the real .onion. As I said, I think this would be easier if we had something like an explicit hosts-like .tor.onion -> .onion mapping instead of delegating this to https-everywhere, but we can still do a good job by observing redirects.

Then, it should be possible to test the feature by adding some custom update channel to https-everywhere that has .tor.onion mappings in it. For example: {{{ Path Prefix: https://people.torproject.org/~acat/misc/rulesets/

JWK: {"kty":"RSA","e":"AQAB","n":"2IGvrYj_UwpO8EMNeEiitb8imzuJA-BGngLYtMkY72wx6aPhuWWvGs-Dlh4lHB4fi8rxfeb0N6ZwCyfCKBeHgfuTgbH14HB-ZSA47JHKEO4fSiay1jjhsGDzQlZDVdn_Wfyxi5je80pizMsOulHKEKzRx4HIcrXY1nf8iiOHEWaB0GX8H1pJjyIlqTxN9Pm5Q4h7kRLUoq3O9EE3o7q1dVeaxYM221WPqaSq9mawAhih4wo_79mb6JinG--s9F3jfqmnOvQk-xAoSTA-XNqTvO-7Mg_WajoA7Lnx1pJbGuvhqNOdodWOFxdMMNaPL8JXmPywl6zAqMESU1rXcaVKVdx1LpcmTyz8dGi3u_GTb6fo5GqXUgByvvRcOXA6DAFC7tbLEy1QinU0q4cRZJYf6s4QxgyRsCgxrcJ9kuDwDHviAm9Yn3eEFRbD2e3hRFfZyvkkLepEWywEfBGdBjQ_Kz9gkQTzmpVef1J-sSD6dnW5OEVEXAPO0sEdr5o-Ng9NSvDGZ3Sw-4AgFO6aynLnpvbVOYneppLF7MKwGVQv0tQ8XY3zBEsxidTIkvmpzKZp6QElpfCwbYnl9aQ9hQ3BmOPIhM2VunP47MPOgyAp4s2m3knwCWbPSR5Gm8agDwIGA1Va1eFAtS-YAYk8v-J20iTyuXrpWqrQmFjEnVLsav8"}

}}}

This should not take long, I would say at most 5 points. From there we could discuss about the .tor.onion urlbar replacement UX, and decide about the UI to view rules/mappings. Depending on that, we could still go for https-everywhere or implement everything in Tor Browser.

Trac:
Cc: ilf, bee, fdsfgs@krutt.org, gaba to ilf, bee, fdsfgs@krutt.org, gaba, brade, mcs

Move s27 must tickets into new roadmap.

Trac:
Keywords: tor-hs https-everywhere tor-ux network-team-roadmap-november TorBrowserTeam202001 deleted, TorBrowserTeam202001, tor-ux, network-team-roadmap-2020Q1, tor-hs, https-everywhere, network-team-roadmap-november added

I was not sure whether it was worth implementing this in Tor Browser directly instead of using https-everywhere, so as suggested by asn I made some list of pros/cons and time estimates for each option.

TLDR: I think it's ok to stick to https-everywhere.

== Features to be implemented

Listing (sub)features so that these can be referenced later.

Have some local knowledge of mappings from memorable *.tor.onion to *.onion in Tor Browser.
Have a way to subscribe to sources of .tor.onion -> .onion mappings, in a way that these can be automatically updated (update channels).
Automatically redirect known human-memorable *.tor.onion addresses to the corresponding .onion.
Via "some UX solution", show the human-memorable .tor.onion in the urlbar even though the actual location is a "long" .onion.
(Maybe) also do 3) when the user navigates to some .onion directly for which we know some *.tor.onion ("reverse lookup").
(Maybe) allow users to view the local .tor.onion -> .onion mappings.
(Out of scope, but maybe later) allow users to edit the local .tor.onion -> .onion mappings.

== Pros/cons of using https-everywhere

Pros:

Update channels already implemented (feature 2).
Automatic redirect already implemented (feature 3).
Might be easier to get third-parties to maintain .tor.onion update channels than if we implement our own update channel mechanism.

Cons:

We might need to do a custom build of the extension if we can't get the changes we need upstreamed to https-everywhere, and that would mean dealing with [1].
With the current rules format, it's not obvious how to extract reliable, explicit .tor.onion -> .onion mappings (see [3] for an example of a torproject.org ruleset extracted from [4]). I think a list of explicit .tor.onion -> .onion mappings would be more desireable, as https-everywhere can do arbitrary url rewrites, and depending on how a rule is written it might not be obvious whether it's a .tor.onion -> .onion mapping or something else.
If for some reason one day https-everywhere is not maintained anymore, we will probably have to reimplement it in Tor Browser anyway.

== Work needed

I'll list the concrete changes that would need to be done if we were to implement it with https-everywhere, with Tor Browser, and finally the ones that need to be done in any case in Tor Browser (common), with time estimation.

=== HTTPS-Everywhere:

Initially I was considering trying to make https-everywhere support a new kind of rules which would explicitly express a .tor.onion -> .onion mapping. However, after thinking a while I think it's too unclear how much time this change would take or if a possible PR would be accepted at all. So, trying to be pragmatic here and suggesting to leave the rules as they are now, and somehow implement a function in Tor Browser side that receives a .tor.onion https-everywhere ruleset and returns a (conservative) .tor.onion -> .onion mapping, so that we can do urlbar .tor.onion rewriting and rules viewing in a simple way.

3 points: Implement message passing in https-everywhere similar to the one we use for NoScript, so that from Tor Browser side we can receive .tor.onion rulesets and features 4, 5, 6 and possibly 7 can be implemented.
1 point: Implement message passing to be able to setup/control update channels programatically from Tor Browser side (feature 1). This should allow us to install an update channel without needing a custom build and having to deal with [1], which I'm not sure how difficult would be.

=== Everything in Tor Browser:

5 points: Implement update channels feature.
2 points: Implement automatic redirects based on .tor.onion mappings.

=== Common items (need to be implemented in Tor Browser in any case):

2 points: UI to view .tor.onion -> .onion mappings (I'm assuming we don't want this in https-everywhere).
3 points: Replacing .onion by human-memorable .tor.onion in urlbar.

According to these estimates, https-everywhere path would be 4 + 5 = 9 points, and doing everything in Tor Browser 7 + 5 = 12 points.

I think it's preferrable to keep the plan and implement it using https-everywhere, and only switch to doing everything in Tor Browser if, at some point, we are forced for technical reasons (which I hope will not be the case).

[1] https://trac.torproject.org/projects/tor/ticket/10394 [2] https://github.com/EFForg/https-everywhere/blob/83642b8e7024a7c4ddc8c6f062c2306882f7f3c0/chromium/background-scripts/update.js [3]

<ruleset name="TorProject Onion" default_off="See https://trac.torproject.org/projects/tor/ticket/11567">
	<target host="torproject.org" />
	<target host="www.torproject.org" />
	<target host="archive.torproject.org" />
	<target host="media.torproject.org" />
	<target host="trac.torproject.org" />
	<rule from="^https?://trac\.torproject\.org/" to="http://vwp5zrdfwmw4avcq.onion/" />
	<rule from="^https?://media\.torproject\.org/" to="http://p3igkncehackjtib.onion/" />
	<rule from="^https?://archive\.torproject\.org/" to="http://j6im4v42ur6dpic3.onion/" />
	<rule from="^https?://(www\.)?torproject\.org/" to="http://idnxcnkne4qt76tg.onion/" />
</ruleset>

[4] https://github.com/chris-barry/darkweb-everywhere

Replying to acat:

Some thoughts after thinking a bit how to implement this.

thanks for sharing this walking through, acat :)

A few questions/issues regarding showing short .tor.onion in urlbar:

Whenever there is a .tor.onion -> .onion redirect, should we display the full original URL in the urlbar, or just replace the hostname for the human-memorable one in the final URL? For example, let's say we observe a redirect from http://nytimes.securedrop.tor.onion to https://www.nyttips4bmquxfzw.onion (upgrade to https + translate from .tor.onion + adding www). In this case I think it would not make much sense to show the original URL, but just display https://www.[nytimes.securedrop.tor.onion] (and if we follow the solution I mentioned, show https://www.nyttips4bmquxfzw.onion when user clicks on it).

Right. The urlbar should display the onion icon + the .onion name: [⊚] https://www.[nytimes.securedrop.tor.onion]

What should we do when user keeps navigating in subpages of some .onion, given the fact that the .tor.onion -> .onion is only observed in the first navigation? Should we show the human-memorable .tor.onion in the urlbar for that tab until the user navigates away from the underlying .onion? Do we also need to show .tor.onion for links opened in a new tab from there?

Maybe, ideally, yes. I wonder what sysrqb or pospeselr think about this. It is hard for me to map all the problems it could carry.

Do we need to show .tor.onion when user navigates directly to a .onion page for which we have some .tor.onion "rule"? (mentioned by sysrqb in a IRC conversation)

This is a good question and I think we should define it before anything because it can be very weird. If we have a strong communication about onion names (via trusted parties, like SecureDrop), then showing an unmemorable onion address to end-users is not smart. If a user writes/copypaste the .onion we can think about how to communicate the .tor.onion redirect for first time users with any UI fashion (a doorhanger can do this work very nicely)

So I would suggest discussing/deciding about the Add support for viewing rulesets (?) feature, as I'm not sure how it could look like if we are going to do this with https-everywhere.

Maybe this is out of scope. We will not allow users to add nor remove .onion related rulesets. IMO showing them at the UI is not needed on this sprint.

Replying to acat:

Via "some UX solution", show the human-memorable .tor.onion in the urlbar even though the actual location is a "long" .onion.

Yes. We should show the memorable .tor.onion in the url bar and we should rely on the circuit display for showing the long onion. I made some props here. Look at the image 4.0.

(Maybe) also do 3) when the user navigates to some .onion directly for which we know some *.tor.onion ("reverse lookup").

I'm thinking about this flow.

(Maybe) allow users to view the local .tor.onion -> .onion mappings.

I don't think this is prior to this release.

Replying to antonela:

Replying to acat:

Via "some UX solution", show the human-memorable .tor.onion in the urlbar even though the actual location is a "long" .onion.

Yes. We should show the memorable .tor.onion in the url bar and we should rely on the circuit display for showing the long onion. I made some props here. Look at the image 4.0.

What if the user wants to copy-paste or inspect the real address (the .onion one)? Or is this not a use case we would need to support?

Besides, I was thinking that perhaps we should make it more noticeable that we are displaying (just) a short .tor.onion local "alias" instead of the actual .onion in the urlbar. In the current design, unless I'm missing something, we would show it exactly as if the .tor.onion was the real page address. I think that might lead to surprises for some users, especially the more technical ones. For example, I would expect that the hostname displayed in the urlbar is the actual origin of the page (e.g. the one that is sent as Host: header in the page HTTP requests, etc.). But that would not be the case here, since we would show .tor.onion in the urlbar the but the true origin of the page would be .onion (the .tor.onion urlbar replacement would be just "cosmetic").

I'm not completely sure how this could be done in terms of UI. Note that now we already show the domain in a highlighted color in the urlbar (Firefox change). Perhaps in these cases we could show the .tor.onion domain in a different color, and when user clicks on the urlbar show the real .onion so that it can be copy-pasted if needed? I'm not sure.

I think the changes here belong more to Tor Browser than to HTTPS Everywhere component.

Trac:
Owner: legind to tbb-team
Component: HTTPS Everywhere/EFF-HTTPS Everywhere to Applications/Tor Browser

Officially support onions in HTTPS-Everywhere

Child items ...

Activity