Opened 15 months ago

Last modified 6 weeks ago

#27466 new task

Investigate single-locale language repacks for Tor Browser on desktop

Reported by: gk Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: AffectsTails, TorBrowserTeam201904, GeorgKoppen201904
Cc: intrigeri, boklm, ckb@… Actual Points:
Parent ID: Points:
Reviewer: Sponsor: Sponsor44-can

Description

Right now we ship in every non en-US bundle both the en-US locale and a language pack for the respective locale. We can do better here by doing the "single-locale language repacks-trick" (https://searchfox.org/mozilla-central/source/build/docs/locales.rst). This would not only save some bundle size but would help us as well routing around the signature problem we have with the signed language packs (see: #26465).

Child Tickets

Change History (30)

comment:1 Changed 15 months ago by gk

See: #26843 for the multi-locale support for Tor Browser for Android.

Last edited 15 months ago by gk (previous) (diff)

comment:2 Changed 15 months ago by gk

This seems to be requiring mercurial and not git to get the l10n sources. I wonder if we could cheat here (or do the whole transition in a multi-step approach) and use the downloaded and verified language packs pointing the repackaging machinery to the unzipped strings instead of a mercurial repo...

comment:3 Changed 15 months ago by intrigeri

Cc: intrigeri added

comment:4 Changed 14 months ago by gk

Keywords: TorBrowserTeam201810 added; TorBrowserTeam201809 removed

Moving tickets to October

comment:5 Changed 13 months ago by gk

Keywords: GeorgKoppen201810 added; GeorgKoppen201809 removed

Moving my tickets to October.

comment:6 Changed 12 months ago by gk

Keywords: TorBrowserTeam201811 added; TorBrowserTeam201810 removed

Moving our tickets to November.

comment:7 Changed 12 months ago by gk

Keywords: GeorgKoppen201811 added; GeorgKoppen201810 removed

comment:8 Changed 12 months ago by gk

Keywords: GeorgKoppen201812 added; GeorgKoppen201811 removed

Moving my tickets to December.

comment:9 in reply to:  2 Changed 12 months ago by gk

Cc: boklm added

Replying to gk:

This seems to be requiring mercurial and not git to get the l10n sources. I wonder if we could cheat here (or do the whole transition in a multi-step approach) and use the downloaded and verified language packs pointing the repackaging machinery to the unzipped strings instead of a mercurial repo...

This is actually not a problem. We are about to do this for the mobile multi language bundle (see: #26843). In fact, looking at the related documentation there should be an option available to provide such a one-bundle-including-all-locale for desktop as well. However, I think we should not make that move for our desktop platforms but rather stick to what Mozilla does, which is the topic of this bug.

The repackaging is happening during the regular Firefox build and the logic around it is quite complex. So, I am still thinking about how to do this best in our reproducible builds setup. Exporting all the neceassary tools and doing the rebundling in the tor-browser step (where we are currently kind of rebundling by adding our own search engines etc.) is likely not easily feasible.

Last edited 9 months ago by intrigeri (previous) (diff)

comment:10 Changed 11 months ago by gk

Parent ID: #28196

Note: #28196

Version 0, edited 11 months ago by gk (next)

comment:11 Changed 11 months ago by gk

Keywords: TorBrowserTeam201812 added; TorBrowserTeam201811 removed

Moving our tickets to December.

comment:12 Changed 10 months ago by gk

Keywords: GeorgKoppen201901 added; GeorgKoppen201812 removed

Moving my tickets.

comment:13 Changed 10 months ago by gk

Keywords: TorBrowserTeam201901 added; TorBrowserTeam201812 removed

Moving tickets to Jan 2019.

comment:14 Changed 9 months ago by gk

Keywords: GeorgKoppen201902 added; GeorgKoppen201901 removed

Moving my tickets to Feb

comment:15 Changed 9 months ago by gk

Keywords: TorBrowserTeam201902 added; TorBrowserTeam201901 removed

Moving tickets to February.

comment:16 Changed 9 months ago by CyrilBrulebois

Cc: ckb@… added

comment:17 Changed 9 months ago by intrigeri

Keywords: AffectsTails added

comment:18 Changed 9 months ago by intrigeri

I've taken a look again after trying to find a good solution for Tails if/when Tor Browser drops the langpacks entirely and does not start providing a multilingual tarball. I think there's a workaround that should hopefully work for Tails but it's ugly (basically: download the langpacks from Mozilla ourselves, patch them ourselves to apply Tor Browser's search plugins and bookmarks customization, and turn extensions.langpacks.signatures.required off ourselves; i.e. essentially do on our side what projects/firefox-langpacks/config and projects/tor-browser/build currently do when building Tor Browser); for somewhat silly reasons you're probably not particularly interested in, it also requires repetitive busywork for us. Let's say that's… not my idea of "fun".

So I've taken a look at the related bugs (#26465, #18915, #21879, #11236) and some of the linked Mozilla bugs to try to understand what's the big deal here. My understanding is that we want to do this because:

  • It would be nice to make the tarball and upgrade deltas other than en-US smaller, because they would not include en-US strings anymore. Each langpack is currently 500-600KB large and each localized tarball is 72MB large, so the single-locale repacking technique could potentially save up to 0.8% on the size of a tarball. I suspect there are lower-hanging fruits, with a better cost/benefit, if we want to shrink the bundles (but I would be happy to stand corrected if you tell me that all those have been fixed already so that <1% optimizations are now a sensible goal :).
  • Mozilla does not support overriding bookmarks and search plugins in any suitable way.
    • So we currently have to patch every langpack.
    • It follows that we need to turn extensions.langpacks.signatures.required off.
    • But that's not nice and we would prefer to enable extensions.langpacks.signatures.required.
    • So the only way to fix that is to get rid of the langpacks entirely, with this single-locale repack technique.
  • Ship something closer to what Mozilla is shipping (I'm assuming they're applying this technique to their own localized Firefox desktop downloadable products). Note that this is only relevant for Windows and macOS: AFAIK most Linux users get Firefox from their distros and I doubt distro packages use this technique; at least on Debian, one gets an en-US Firefox and installs the langpack via the firefox-l10n-$LOCALE package. So I'm quite confident that Mozilla will keep supporting "en-US + langpack" for the foreseeable future.
  • Any other reason?

Zooming into the reason that seems the most valid to me, I guess we want to enable extensions.langpacks.signatures.required because:

  • Firefox may simply stop supporting this pref some day and then we might have to deal with this with only a few months notice.
  • For defense in depth reasons: if the Firefox add-on updating code is buggy or AMO is compromised, and our code that disables updating most add-ons is buggy, then presumably an attacker could force an update to a langpack they control, and then all kinds of undesirable things can happen.

If this analysis of our motivations is about right, then I would argue that sure, it would be very nice to have, but perhaps not urgent to the point of rushing this just a few weeks before 8.5. If it's an option to relax the timeline a bit, then we can perhaps do some of:

  • Revisit some of the other options you've mentioned in https://trac.torproject.org/projects/tor/ticket/18915#comment:3 for search plugins.
  • Check if Mozilla would be open to adding a pref that disables looking in langpacks for bookmarks and search engines. Then we don't need to patch them, we can enable extensions.langpacks.signatures.required, and the main reason to do what this ticket is about is gone.
  • Block this on having one-bundle-including-all-locale for desktop first.

Now, if there's a cheap, possibly dirty but good enough workaround that Tails can apply in time for 8.5, I'll happily let you handle this in this timeframe and I'll shut up :)

comment:19 in reply to:  18 ; Changed 9 months ago by gk

Replying to intrigeri:

I've taken a look again after trying to find a good solution for Tails if/when Tor Browser drops the langpacks entirely and does not start providing a multilingual tarball. I think there's a workaround that should hopefully work for Tails but it's ugly (basically: download the langpacks from Mozilla ourselves, patch them ourselves to apply Tor Browser's search plugins and bookmarks customization, and turn extensions.langpacks.signatures.required off ourselves; i.e. essentially do on our side what projects/firefox-langpacks/config and projects/tor-browser/build currently do when building Tor Browser); for somewhat silly reasons you're probably not particularly interested in, it also requires repetitive busywork for us. Let's say that's… not my idea of "fun".

So I've taken a look at the related bugs (#26465, #18915, #21879, #11236) and some of the linked Mozilla bugs to try to understand what's the big deal here. My understanding is that we want to do this because:

  • It would be nice to make the tarball and upgrade deltas other than en-US smaller, because they would not include en-US strings anymore. Each langpack is currently 500-600KB large and each localized tarball is 72MB large, so the single-locale repacking technique could potentially save up to 0.8% on the size of a tarball. I suspect there are lower-hanging fruits, with a better cost/benefit, if we want to shrink the bundles (but I would be happy to stand corrected if you tell me that all those have been fixed already so that <1% optimizations are now a sensible goal :).
  • Mozilla does not support overriding bookmarks and search plugins in any suitable way.
    • So we currently have to patch every langpack.
    • It follows that we need to turn extensions.langpacks.signatures.required off.
    • But that's not nice and we would prefer to enable extensions.langpacks.signatures.required.
    • So the only way to fix that is to get rid of the langpacks entirely, with this single-locale repack technique.
  • Ship something closer to what Mozilla is shipping (I'm assuming they're applying this technique to their own localized Firefox desktop downloadable products). Note that this is only relevant for Windows and macOS: AFAIK most Linux users get Firefox from their distros and I doubt distro packages use this technique; at least on Debian, one gets an en-US Firefox and installs the langpack via the firefox-l10n-$LOCALE package. So I'm quite confident that Mozilla will keep supporting "en-US + langpack" for the foreseeable future.
  • Any other reason?

This list is good, thanks for that. Yes, there is another benefit: we could get our security bugfixes out earlier as we don't need to wait on the langpacks being ready to start the build. We are talking here about a couple of hours which seems like a good win in case of emergency releases.

Zooming into the reason that seems the most valid to me, I guess we want to enable extensions.langpacks.signatures.required because:

  • Firefox may simply stop supporting this pref some day and then we might have to deal with this with only a few months notice.
  • For defense in depth reasons: if the Firefox add-on updating code is buggy or AMO is compromised, and our code that disables updating most add-ons is buggy, then presumably an attacker could force an update to a langpack they control, and then all kinds of undesirable things can happen.

If this analysis of our motivations is about right,

Yeah, I think that's the case.

then I would argue that sure, it would be very nice to have, but perhaps not urgent to the point of rushing this just a few weeks before 8.5. If it's an option to relax the timeline a bit, then we can perhaps do some of:

  • Revisit some of the other options you've mentioned in https://trac.torproject.org/projects/tor/ticket/18915#comment:3 for search plugins.
  • Check if Mozilla would be open to adding a pref that disables looking in langpacks for bookmarks and search engines. Then we don't need to patch them, we can enable extensions.langpacks.signatures.required, and the main reason to do what this ticket is about is gone.
  • Block this on having one-bundle-including-all-locale for desktop first.

Now, if there's a cheap, possibly dirty but good enough workaround that Tails can apply in time for 8.5, I'll happily let you handle this in this timeframe and I'll shut up :)

No need to shut up. That's been a good summary and analysis. Now, alas, I won't have time to work on that for 8.5 as I need to help out to get the browser for Android into a shippable state. Thus, this won't come before 9.0. And, yes, this ticket is about an investigation, so we might at the end find we achieve our goals better by other means (but I am skeptical so far). Right now the most important one is getting the signature pref enabled, followed by starting builds earlier, followed by being closer to what Mozilla ships, followed by saving several hundred kBytes (Note, though, it will actually be quite a bit more we could save, given that we would not need to ship all Torbutton locales and Tor Launcher locales either anymore which is tracked in #5184).

comment:20 in reply to:  19 ; Changed 9 months ago by intrigeri

Replying to gk:

Replying to intrigeri:
This list is good, thanks for that.

/me blushes

Yes, there is another benefit: we could get our security bugfixes out earlier as we don't need to wait on the langpacks being ready to start the build. We are talking here about a couple of hours which seems like a good win in case of emergency releases.

This would be great! It would also improve Tails' "time to remediation", so I now see there's something in it for Tails too, which is motivating :)

That's been a good summary and analysis. Now, alas, I won't have time to work on that for 8.5 as I need to help out to get the browser for Android into a shippable state. Thus, this won't come before 9.0.

OK. With my Tails hat on, this is reassuring wrt. the timing of upcoming changes.
To make sure I'm on the right page, 9.0 == Firefox 68esr = Oct 2019?

And, yes, this ticket is about an investigation, so we might at the end find we achieve our goals better by other means (but I am skeptical so far).

ACK. I'm personally interested in helping find a solution that avoids ugly workarounds on Tails' side, but alas, at the moment I can't reasonably commit to anything more. (My evil plans for early retirement take more time than I would have hoped ;) Now, if we find a couple spare hours together at a Tor meeting — unlikely —, I'd be happy to do whatever I can there.

Right now the most important one is […]

Thanks for spelling out the goals and their priorities this clearly. I think it's invaluable info to rank this in a roadmap and it helps me understand better what's going on.

comment:21 in reply to:  20 Changed 9 months ago by gk

Replying to intrigeri:

Replying to gk:

That's been a good summary and analysis. Now, alas, I won't have time to work on that for 8.5 as I need to help out to get the browser for Android into a shippable state. Thus, this won't come before 9.0.

OK. With my Tails hat on, this is reassuring wrt. the timing of upcoming changes.
To make sure I'm on the right page, 9.0 == Firefox 68esr = Oct 2019?

Yes.

comment:22 Changed 8 months ago by gk

Keywords: TorBrowserTeam201903 added; TorBrowserTeam201902 removed

Moving my tickets to March.

comment:23 Changed 8 months ago by gk

Keywords: GeorgKoppen201903 added; GeorgKoppen201902 removed

Now for my keyword.

comment:24 Changed 7 months ago by gk

Keywords: TorBrowserTeam201904 added; TorBrowserTeam201903 removed

Moving tickets to April.

comment:25 Changed 7 months ago by gk

Keywords: GeorgKoppen201904 added; GeorgKoppen201903 removed

Moving my tickets for April

comment:26 Changed 3 months ago by acat

Keywords: ff68-esr added

comment:27 Changed 3 months ago by pili

Sponsor: Sponsor44-can

Tagging with Sponsor 44

comment:28 Changed 7 weeks ago by gk

Keywords: ff68-esr removed

comment:29 Changed 7 weeks ago by gk

Parent ID: #28196

comment:30 Changed 6 weeks ago by gk

Alright, to give an update here. There is no need anymore to include search engines in the lang packs from our side starting with Tor Browser 9. I've opened #31942 for re-enabling the signature check at least.

Note: See TracTickets for help on using tickets.