Special characters are not escaped in translations and break the build

As said in comment:14:ticket:27125 there are special characters (like the ') that need to get escaped in strings, otherwise the build breaks. While we did that in en-US that does not happen in translations (yet).

Trac:
Child Ticket(s): #31350 (moved)

added GeorgKoppen201904 TorBrowserTeam201904 component::community/translations owner::emmapeel priority::medium severity::normal status::needs-information type::defect labels

I agree we should not require translators to know which entities to escape and which not. However, importing the strings as they currently are breaks the build. Could we have a script running over the translated strings adding necessary escaping if that is missed before the final strings are available in the translations repo?

I wonder how other projects are dealing with that problem as I guess we are hardly the only ones getting hit by it...

I will talk with erinm and find out if that can be a feature request for Transifex

formatting rules: ​https://developer.android.com/guide/topics/resources/string-resource#FormattingAndStyling

I think i cleaned up a bit, let me know if it is still broken.

i also added a note for the translators, but it does not force the escaping.

Replying to emmapeel:

I think i cleaned up a bit, let me know if it is still broken.

i also added a note for the translators, but it does not force the escaping.

Thanks, this works for me. I'll unparent the ticket. Not sure if you want to keep it open for having a less ad hoc way of dealing with this problem implemented on the server-/repository-side. (Assuming that's the place where we want to make sure the escaping is properly done).

Trac:
Parent: #30016 (moved) to N/A

Yes, I'd consider this ticket done when transifex can escape the chars themselves and present translators with a normal string, doing the escaping on the background.

This will prevent future problems, because right now we have to review this 'by hand'.

Replying to emmapeel:

Yes, I'd consider this ticket done when transifex can escape the chars themselves and present translators with a normal string, doing the escaping on the background.

This will prevent future problems, because right now we have to review this 'by hand'.

Or maybe there is some other automated step we can do, keeping Transifex out of the loop. I'll ask around what Mozilla is doing.

Trac:
Keywords: N/A deleted, GeorgKoppen201904, TorBrowserTeam201904 added

Okay, I talked to Pike who helped me a lot (thanks for that). Mozilla does not have an automatic way to escape strings properly. However, they have a tool called compare-locales that is used to make sure to check for issues before a build breaks. See: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/compare-locales for usage info. Moreover, I got told that we can run their tool by creating an l10n.toml to run against the l10n repo, with the android-dtd tests as defined in https://searchfox.org/mozilla-central/rev/d143f8ce30d1bcfee7a1227c27bf876a85f8cede/mobile/android/locales/l10n.toml#131-136.

Thus, I guess instead of using their l10n repo we could point to a location of our choice to check for Android issues.

emmapeel: Is that enough to get you started?

well there are other types of documents that escape characters, like PO files I believe. It would be a matter of creating a different type of document. I need to check with erinm on that.

regarding how other projectss do it, they do checks by themselves, see for example in Tails:

https://git-tails.immerda.ch/jenkins-tools/tree/slaves/check_po

transifex does not provide any kind of checks (Weblate does check some stuff tho, tsck tsck).

I think it is important to have a CI test that ensures that the translations aren't breaking anything. For F-Droid, I wrote an emulator test that loads all strings in all locales. It has found many crasher bugs. Feel free to take it under the license you need, it will need some tweaking to work in a different app (it is GPLv3+ in F-Droid): https://gitlab.com/fdroid/fdroidclient/blob/master/app/src/androidTest/java/org/fdroid/fdroid/LocalizationTest.java

IMHO, the translation tool should properly escape the translator's input whenever possible. I believe Weblate does a pretty good job of this. Weblate checks are also quite useful for this kind of thing

emmapeel: what do we need to get this bug resolved/unblocked? Luckily, I ran a test build before starting the build for 8.5.4 in order to figure whether we could use the latest translations for mobile. But it turns out we can't as 21677307d87d3216eede51a6ad36bf26e7937667 is the last good commit before ''s get added unescaped to ca. That alone is unfortunate but it's even more so that this blocks other unaffected locales from getting the latest translations.

Trac:
Cc: emmapeel, igt0, gk, hans@guardianproject.info, ggus to emmapeel, igt0, gk, hans@guardianproject.info, ggus, pili
Status: new to needs_information

The people of transifex says they cannot make a different escaping than the one on the dtd specs, so we should create a githook for the files at https://gitweb.torproject.org/translation.git/tree/en-US/torbrowser_strings.dtd?h=tba-torbrowserstringsdtd to be tested with the compare-locales mentioned by geko.

This was discussed during the ux team meeting a couple of weeks ago (http://meetbot.debian.net/tor-meeting/2019/tor-meeting.2019-10-01-13.59.log.html):

14:32:23 we have a bot that pulls new translations from transifex without any verification. so at some point we should do a verification. maybe as part of the pulling from transifex. 14:32:33 but anyway we can discuss on the ticket if you have any ideas 14:32:44 it should be the bot that does the checks 14:32:49 not git with a hook 14:33:45 something called from https://gitweb.torproject.org/translation-tools.git/tree/update_translations 14:34:12 sounds like this is a discussion that we can maybe take to #tor-l10n after? :) 14:34:29 I just wanted to make the right people aware of it 14:34:33 sounds like this is not a gitadmin ticket 14:35:04 let's move it to the right owner then and discuss it with them, does anyone know who that should be? 14:35:55 whoever maintains that bot 14:36:09 it is a bit big for me to code a l10n verification script in shellscript tbh 14:36:26 maybe bolkm can help us here? 14:36:33 i maintain the process of running the bot and make little changes, but a verification thing.. uff... 14:37:27 which bot is this and where is the code? 14:38:31 the code is at https://gitweb.torproject.org/translation-tools.git/tree/update_translations and is the bot that pushes the transifex updates to https://gitweb.torproject.org/translation.git 14:39:16 lives at https://db.torproject.org/machines.cgi?host=majus 14:40:30 and we think that the bot should check that special characters are escaped? 14:40:45 I think python escapes the characters automatically nowadays 14:40:56 there should be something called here: https://gitweb.torproject.org/translation-tools.git/tree/update_translations#n39 14:41:10 before the commits to check if the diff is legit 14:41:52 so basically we need a confident bash coder :) 14:41:59 the escaping of certain characters on the android files is not according to standard, and when adding ' in the translations, the build will break. transifex didnt wanted to create a new parsing because it is against the escaping rules of dtd files

We need someone to help emmapeel with this verification script.

mentioned in issue #32650 (moved)

mentioned in issue #31350 (moved)

moved to tpo/community/l10n#30054 (moved)

mentioned in issue tpo/community/l10n#32650 (moved)

mentioned in issue tpo/applications/tor-browser#40353