Investigate single-locale language repacks for Tor Browser on desktop
Right now we ship in every non en-US bundle both the en-US locale and a language pack for the respective locale. We can do better here by doing the "single-locale language repacks-trick" (https://searchfox.org/mozilla-central/source/build/docs/locales.rst). This would not only save some bundle size but would help us as well routing around the signature problem we have with the signed language packs (see: #26465 (moved)).
Activity
See: #26843 (moved) for the multi-locale support for Tor Browser for Android.
-
This seems to be requiring mercurial and not git to get the l10n sources. I wonder if we could cheat here (or do the whole transition in a multi-step approach) and use the downloaded and verified language packs pointing the repackaging machinery to the unzipped strings instead of a mercurial repo...
-
Replying to gk:
This seems to be requiring mercurial and not git to get the l10n sources. I wonder if we could cheat here (or do the whole transition in a multi-step approach) and use the downloaded and verified language packs pointing the repackaging machinery to the unzipped strings instead of a mercurial repo...
This is actually not a problem. We are about to do this for the mobile multi language bundle (see: #26843 (moved)). In fact, looking at the related documentation there should be an option available to provide such a one-bundle-including-all-locale for desktop as well. However, I think we should not make that move for our desktop platforms but rather stick to what Mozilla does, which is the topic of this bug.
The repackaging is happening during the regular Firefox build and the logic around it is quite complex. So, I am still thinking about how to do this best in our reproducible builds setup. Exporting all the neceassary tools and doing the rebundling in the
tor-browserstep (where we are currently kind of rebundling by adding our own search engines etc.) is likely not easily feasible.Trac:
Cc: intrigeri to intrigeri, boklm -
Note: #28196 (moved) depends on this ticket.
Trac:
Parent: N/A to #28196 (moved) 
Trac:
Username: CyrilBrulebois
Cc: intrigeri, boklm to intrigeri, boklm, ckb@riseup.net
I've taken a look again after trying to find a good solution for Tails if/when Tor Browser drops the langpacks entirely and does not start providing a multilingual tarball. I think there's a workaround that should hopefully work for Tails but it's ugly (basically: download the langpacks from Mozilla ourselves, patch them ourselves to apply Tor Browser's search plugins and bookmarks customization, and turn
extensions.langpacks.signatures.requiredoff ourselves; i.e. essentially do on our side whatprojects/firefox-langpacks/configandprojects/tor-browser/buildcurrently do when building Tor Browser); for somewhat silly reasons you're probably not particularly interested in, it also requires repetitive busywork for us. Let's say that's… not my idea of "fun".So I've taken a look at the related bugs (#26465 (moved), #18915 (moved), #21879 (moved), #11236 (moved)) and some of the linked Mozilla bugs to try to understand what's the big deal here. My understanding is that we want to do this because:
- It would be nice to make the tarball and upgrade deltas other than en-US smaller, because they would not include en-US strings anymore. Each langpack is currently 500-600KB large and each localized tarball is 72MB large, so the single-locale repacking technique could potentially save up to 0.8% on the size of a tarball. I suspect there are lower-hanging fruits, with a better cost/benefit, if we want to shrink the bundles (but I would be happy to stand corrected if you tell me that all those have been fixed already so that <1% optimizations are now a sensible goal :).
- Mozilla does not support overriding bookmarks and search plugins in any suitable way.
- So we currently have to patch every langpack.
- It follows that we need to turn
extensions.langpacks.signatures.requiredoff. - But that's not nice and we would prefer to enable
extensions.langpacks.signatures.required. - So the only way to fix that is to get rid of the langpacks entirely, with this single-locale repack technique.
- Ship something closer to what Mozilla is shipping (I'm assuming they're applying this technique to their own localized Firefox desktop downloadable products). Note that this is only relevant for Windows and macOS: AFAIK most Linux users get Firefox from their distros and I doubt distro packages use this technique; at least on Debian, one gets an en-US Firefox and installs the langpack via the
firefox-l10n-$LOCALEpackage. So I'm quite confident that Mozilla will keep supporting "en-US + langpack" for the foreseeable future. - Any other reason?
Zooming into the reason that seems the most valid to me, I guess we want to enable
extensions.langpacks.signatures.requiredbecause:- Firefox may simply stop supporting this pref some day and then we might have to deal with this with only a few months notice.
- For defense in depth reasons: if the Firefox add-on updating code is buggy or AMO is compromised, and our code that disables updating most add-ons is buggy, then presumably an attacker could force an update to a langpack they control, and then all kinds of undesirable things can happen.
If this analysis of our motivations is about right, then I would argue that sure, it would be very nice to have, but perhaps not urgent to the point of rushing this just a few weeks before 8.5. If it's an option to relax the timeline a bit, then we can perhaps do some of:
- Revisit some of the other options you've mentioned in https://trac.torproject.org/projects/tor/ticket/18915#comment:3 for search plugins.
- Check if Mozilla would be open to adding a pref that disables looking in langpacks for bookmarks and search engines. Then we don't need to patch them, we can enable
extensions.langpacks.signatures.required, and the main reason to do what this ticket is about is gone. - Block this on having one-bundle-including-all-locale for desktop first.
Now, if there's a cheap, possibly dirty but good enough workaround that Tails can apply in time for 8.5, I'll happily let you handle this in this timeframe and I'll shut up :)
-
Replying to intrigeri:
I've taken a look again after trying to find a good solution for Tails if/when Tor Browser drops the langpacks entirely and does not start providing a multilingual tarball. I think there's a workaround that should hopefully work for Tails but it's ugly (basically: download the langpacks from Mozilla ourselves, patch them ourselves to apply Tor Browser's search plugins and bookmarks customization, and turn
extensions.langpacks.signatures.requiredoff ourselves; i.e. essentially do on our side whatprojects/firefox-langpacks/configandprojects/tor-browser/buildcurrently do when building Tor Browser); for somewhat silly reasons you're probably not particularly interested in, it also requires repetitive busywork for us. Let's say that's… not my idea of "fun".So I've taken a look at the related bugs (#26465 (moved), #18915 (moved), #21879 (moved), #11236 (moved)) and some of the linked Mozilla bugs to try to understand what's the big deal here. My understanding is that we want to do this because:
- It would be nice to make the tarball and upgrade deltas other than en-US smaller, because they would not include en-US strings anymore. Each langpack is currently 500-600KB large and each localized tarball is 72MB large, so the single-locale repacking technique could potentially save up to 0.8% on the size of a tarball. I suspect there are lower-hanging fruits, with a better cost/benefit, if we want to shrink the bundles (but I would be happy to stand corrected if you tell me that all those have been fixed already so that <1% optimizations are now a sensible goal :).
- Mozilla does not support overriding bookmarks and search plugins in any suitable way.
- So we currently have to patch every langpack.
- It follows that we need to turn
extensions.langpacks.signatures.requiredoff. - But that's not nice and we would prefer to enable
extensions.langpacks.signatures.required. - So the only way to fix that is to get rid of the langpacks entirely, with this single-locale repack technique.
- Ship something closer to what Mozilla is shipping (I'm assuming they're applying this technique to their own localized Firefox desktop downloadable products). Note that this is only relevant for Windows and macOS: AFAIK most Linux users get Firefox from their distros and I doubt distro packages use this technique; at least on Debian, one gets an en-US Firefox and installs the langpack via the
firefox-l10n-$LOCALEpackage. So I'm quite confident that Mozilla will keep supporting "en-US + langpack" for the foreseeable future. - Any other reason?
This list is good, thanks for that. Yes, there is another benefit: we could get our security bugfixes out earlier as we don't need to wait on the langpacks being ready to start the build. We are talking here about a couple of hours which seems like a good win in case of emergency releases.
Zooming into the reason that seems the most valid to me, I guess we want to enable
extensions.langpacks.signatures.requiredbecause:- Firefox may simply stop supporting this pref some day and then we might have to deal with this with only a few months notice.
- For defense in depth reasons: if the Firefox add-on updating code is buggy or AMO is compromised, and our code that disables updating most add-ons is buggy, then presumably an attacker could force an update to a langpack they control, and then all kinds of undesirable things can happen.
If this analysis of our motivations is about right,
Yeah, I think that's the case.
then I would argue that sure, it would be very nice to have, but perhaps not urgent to the point of rushing this just a few weeks before 8.5. If it's an option to relax the timeline a bit, then we can perhaps do some of:
- Revisit some of the other options you've mentioned in https://trac.torproject.org/projects/tor/ticket/18915#comment:3 for search plugins.
- Check if Mozilla would be open to adding a pref that disables looking in langpacks for bookmarks and search engines. Then we don't need to patch them, we can enable
extensions.langpacks.signatures.required, and the main reason to do what this ticket is about is gone. - Block this on having one-bundle-including-all-locale for desktop first.
Now, if there's a cheap, possibly dirty but good enough workaround that Tails can apply in time for 8.5, I'll happily let you handle this in this timeframe and I'll shut up :)
No need to shut up. That's been a good summary and analysis. Now, alas, I won't have time to work on that for 8.5 as I need to help out to get the browser for Android into a shippable state. Thus, this won't come before 9.0. And, yes, this ticket is about an investigation, so we might at the end find we achieve our goals better by other means (but I am skeptical so far). Right now the most important one is getting the signature pref enabled, followed by starting builds earlier, followed by being closer to what Mozilla ships, followed by saving several hundred kBytes (Note, though, it will actually be quite a bit more we could save, given that we would not need to ship all Torbutton locales and Tor Launcher locales either anymore which is tracked in #5184 (moved)).
-
Replying to gk:
Replying to intrigeri: This list is good, thanks for that.
/me blushes
Yes, there is another benefit: we could get our security bugfixes out earlier as we don't need to wait on the langpacks being ready to start the build. We are talking here about a couple of hours which seems like a good win in case of emergency releases.
This would be great! It would also improve Tails' "time to remediation", so I now see there's something in it for Tails too, which is motivating :)
That's been a good summary and analysis. Now, alas, I won't have time to work on that for 8.5 as I need to help out to get the browser for Android into a shippable state. Thus, this won't come before 9.0.
OK. With my Tails hat on, this is reassuring wrt. the timing of upcoming changes. To make sure I'm on the right page, 9.0 == Firefox 68esr = Oct 2019?
And, yes, this ticket is about an investigation, so we might at the end find we achieve our goals better by other means (but I am skeptical so far).
ACK. I'm personally interested in helping find a solution that avoids ugly workarounds on Tails' side, but alas, at the moment I can't reasonably commit to anything more. (My evil plans for early retirement take more time than I would have hoped ;) Now, if we find a couple spare hours together at a Tor meeting — unlikely —, I'd be happy to do whatever I can there.
Right now the most important one is […]
Thanks for spelling out the goals and their priorities this clearly. I think it's invaluable info to rank this in a roadmap and it helps me understand better what's going on.
-
Replying to intrigeri:
Replying to gk:
That's been a good summary and analysis. Now, alas, I won't have time to work on that for 8.5 as I need to help out to get the browser for Android into a shippable state. Thus, this won't come before 9.0.
OK. With my Tails hat on, this is reassuring wrt. the timing of upcoming changes. To make sure I'm on the right page, 9.0 == Firefox 68esr = Oct 2019?
Yes.
-
Trac:
Parent: #28196 (moved) to N/A -
Alright, to give an update here. There is no need anymore to include search engines in the lang packs for non-macOS bundles from our side starting with Tor Browser 9. I've opened #31942 (moved) for re-enabling the signature check there at least.
mentioned in issue #28196 (moved)
mentioned in issue #32475 (moved)
