TBA: Investigate how Mozilla Fennec provides localization

Fennec is shipped with multi-locale support. Under the Settings menu option, General -> Language -> Browser Language there are tens of languages and locales available. How are these included and shipped?

added GeorgKoppen201812 TBA-a2 TorBrowserTeam201812 component::applications/tor browser owner::tbb-team parent::26782 priority::high resolution::fixed severity::normal status::closed tbb-mobile type::task labels

Trac:
Parent: #5709 (moved) to #26782 (moved)

Okay. So, there's a doc for this. Specifically, the mozilla build system clones l10n-central and creates the localization package from there. I haven't look at the logic used for this, yet.

We'll want to clone l10n-central separately, and add the location into .mozconfig-android:

ac_add_options --with-l10n-base=/make/this/a/absolute/path

The specific locale is repackaged using:

      ./mach build installers-$AB_CD

The doc says:

You should find a re-packaged build at ``OBJDIR/dist/``, and a
runnable binary in ``OBJDIR/dist/l10n-stage/``.

Running the installers- build target (by default):

The ``installers`` target runs quite a few things for you, including getting
the repository for the requested locale from
https://hg.mozilla.org/l10n-central/. It will clone them into
``~/.mozbuild/l10n-central``.

Multilocale packages are created using:

      export MOZ_CHROME_MULTILOCALE="de it zh-TW"
      for AB_CD in $MOZ_CHROME_MULTILOCALE; do
         ./mach build chrome-$AB_CD
      done

      AB_CD=multi ./mach package

This sounds like the mechanism we eventually want to use for getting rid of our bundle-language-packs-approach. So, we want that for #26465 (moved) as well which makes me think we should keep desktop in mind once we file follow-up bugs to implement the things we learned from the investigation in this ticket.

FWIW: I plan to work on the repack idea for desktop in #27466 (moved) and might think about how to integrate the multi-locale support for Tor Browser on Android into our build system while I am at it.

Trac:
Keywords: N/A deleted, TorBrowserTeam201811, TBA-a2, GeorgKoppen201811 added

Adding some notes:

https://hg.mozilla.org/l10n-central is the repo to use (not something like https://hg.mozilla.org/releases/l10n/mozilla-release); changesets that are shipping can be found at URLs like

https://product-details.mozilla.org/1.0/l10n/Firefox-60.3.0esr-build1.json

(there are no tags in repos anymore).

Adding some more notes.

1. mkdir l10n-central
2. pushd l10n-central
3. clone target locale (such as es-ES, hg clone https://hg.mozilla.org/l10n-central/es-ES)
4. popd
5. Add --with-l10n-base= in mozconfig
6. ./mach configure ...
7. ./mach build && ./mach package
8. ./mach build installers-es-ES

This results in a localized apk. There is an inconsistency when built with #28051 (moved) because Orbot doesn't learn about the new locale until later, but we can correct that. Overall, Fennec, torbutton, and Orbot use the correct localized strings.

Okay. comment:7 is correct, but this becomes more complicated with Orbot. When we build a single-locale package, it replaces the localizable strings - meaning each English string that is localized is replaced with the localized string, so the English string is not available within the app. This is good - but it becomes more confusing.

Because all the English strings are substituted with localized strings, the user cannot choose between the localized string and the English strings. This usually wouldn't be a problem, except within Fennec's preferences menu "English (United States)" is always shown. It seems Mozilla thought it important to hard-code "en-US" as an option, even when it is not really available. sigh. But that is relatively easy to fix.

The difficulty comes when Orbot is added into the app. The single-locale re-package process only replaces the Fennec strings, it doesn't touch the Orbot strings, so Orbot uses the system default locale when it starts. This isn't a problem if the user downloaded Tor Browser for the same locale as their system, but if the user has (as an example) en-US as their system default locale but downloads the es-ES localized Tor Browser, then it becomes weird. In this situation, Orbot shows everything in English and Tor Browser shows everything in es-ES. Orbot allows the user switch their locale, but this is confusing.

I think we should add a mechanism within Orbot so it automatically chooses the correct locale when it starts. We can likely add this in the logic when Tor Browser runs Orbot the first time. I prefer having the user control their locale on the Tor Browser-side of the app, rather than the Orbot-side - but others can change my mind :)

I built a multi-locale package, including the 28 locales provided by the desktop alpha builds. It is as simple as the documentation says. The trick is you must do it exactly as the doc says, and don't change the name of the environment variable it says :) (specifically, MOZ_CHROME_MULTILOCALE must be used and set correctly)

Follow the same steps as comment:7, except substitute the last step (./mach build installers-es-ES) with the following three steps:

$ export MOZ_CHROME_MULTILOCALE="ar ca cs da de el es-ES fa fr ga-IE he hu id is it ja ka ko nb-NO nl pl pt-BR ru sv-SE tr vi zh-CN zh-TW"
$ for AB_CD in $MOZ_CHROME_MULTILOCALE; do
    ./mach build chrome-$AB_CD;
done
$ AB_CD=multi ./mach package

The resulting apk is ~48MB, including Orbot and the addons. The filename contains en-US:

$ # before
$ du -sh ~/tor-browser/obj-x86-linux-android/dist/fennec-60.3.0.en-US.android-i686.apk
46M     /home/android/tor-browser/obj-x86-linux-android/dist/fennec-60.3.0.en-US.android-i686.apk
$ # after
$ du -sh ~/tor-browser/obj-x86-linux-android/dist/fennec-60.3.0.en-US.android-i686.apk
48M     /home/android/tor-browser/obj-x86-linux-android/dist/fennec-60.3.0.en-US.android-i686.apk

Comparing the mutli-locale package with the single-locale package, the multi-locale package provides a better user experience. Orbot and Fennec use the system default locale on first-run, so they should use the same locale when the user opens the app. When a different locale is chosen within Fennec, Orbot uses the new locale, too. It seems like this mitigates the major problems in the single-locale package.

Branch bug_26843_v2 has a patch adding projects/firefox-locale-bundle: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26843_v2&id=85086602ebe3c25c6a3d81d7efeb7d0f6f905605

With this we are getting a directory containing one tarball for each locale.

Trac:
Status: new to needs_review
Keywords: TorBrowserTeam201811 deleted, TorBrowserTeam201811R added

Branch bug_26843_v3 has a new revision of the patch, creating one tarball with all locales instead of one for each locale: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26843_v3&id=6b77a6908c798304e6539c66ed4889b5aae4c875

Replying to boklm:

Branch bug_26843_v3 has a new revision of the patch, creating one tarball with all locales instead of one for each locale: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26843_v3&id=6b77a6908c798304e6539c66ed4889b5aae4c875

Looks mostly good, thanks. One thing we should fix:

exite_error("Error reading $file", 2) unless defined $json_text;

There is no exite_error function.

Trac:
Keywords: TorBrowserTeam201811R deleted, TorBrowserTeam201811 added
Status: needs_review to needs_revision

Trac:
Cc: igt0, gk to igt0, gk, emmapeel

Replying to gk:

Looks mostly good, thanks. One thing we should fix: {{{ exite_error("Error reading $file", 2) unless defined $json_text; }}} There is no exite_error function.

This typo is fixed in bug_26843_v4: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26843_v4&id=43c9452946313a5ab3dd064501daa05096db86eb

Trac:
Status: needs_revision to needs_review
Keywords: TorBrowserTeam201811 deleted, TorBrowserTeam201811R added

Replying to boklm:

Replying to gk:

Looks mostly good, thanks. One thing we should fix: {{{ exite_error("Error reading $file", 2) unless defined $json_text; }}} There is no exite_error function.

This typo is fixed in bug_26843_v4: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26843_v4&id=43c9452946313a5ab3dd064501daa05096db86eb

Looks good, thanks so much for this. I merged the patch to master (commit 43c9452946313a5ab3dd064501daa05096db86eb).

I created #28573 (moved) for the testbuild issue I had.

Setting to new, while double-checking my patch for this bug.

Trac:
Keywords: TorBrowserTeam201811R deleted, TorBrowserTeam201811 added
Status: needs_review to new

bug_26843_v5 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_26843_v5&id=a87de530dc5f5bde37079107bfa55b1033fba358) has the changes for review. I get a multi-locale Tor Browser that way. I am not sure why the extensions strings are not shown in the selected locale, though, they are always only showing up in en-US. I opened #28575 (moved) and I guess we have #27656 (moved) for the Torbutton case.

Trac:
Status: new to needs_review
Keywords: TorBrowserTeam201811 deleted, TorBrowserTeam201811R added

Okay, I have an updated branch for review, bug_26843_v6 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_26843_v6&id=1313336490abb4773e20d40347db79e46c26727e).

Compared to bug_26843_v5 it is rebased on the latest master and contains an indication in the bundle name that the .apk is actually not a single-locale one, using "multi" as Mozilla does.

Move review tickets to Decemeber.

Trac:
Keywords: TorBrowserTeam201811R deleted, TorBrowserTeam201812R added

Replying to gk:

Okay, I have an updated branch for review, bug_26843_v6 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_26843_v6&id=1313336490abb4773e20d40347db79e46c26727e).

Compared to bug_26843_v5 it is rebased on the latest master and contains an indication in the bundle name that the .apk is actually not a single-locale one, using "multi" as Mozilla does.

It seems that when we are running AB_CD=multi ./mach package, the variable MOZ_CHROME_MULTILOCALE contains all the locales we want. However, when we run ./mach build chrome-[% lang %], only the current and previous locales are included in MOZ_CHROME_MULTILOCALE. Is ./mach build chrome-[% lang %] ignoring the variable MOZ_CHROME_MULTILOCALE (and in this case it doesn't matter if it doesn't contain all the locales), or should we add all the locales before running the first ./mach build chrome-[% lang %]?

We could use this line to set all the locales in MOZ_CHROME_MULTILOCALE:

export MOZ_CHROME_MULTILOCALE='[% tmpl(c('var/locales').join(' ')) %]'

An other minor thing: I think it would be better to extract the locales in a sub-directory (such as /var/tmp/dist/locales) instead of directly in /var/tmp/dist.

Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201812R deleted, TorBrowserTeam201812 added