Opened 2 weeks ago

Closed 9 days ago

Last modified 9 days ago

#32342 closed defect (fixed)

Tor Browser for Android crashes when opening the locale pane in the settings

Reported by: gk Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Major Keywords: tbb-mobile, tbb-crash, tbb-9.0-issues, tbb-regression, tbb-9.0.1-can, TorBrowserTeam201911R
Cc: sysrqb, sisbell Actual Points: .2
Parent ID: Points:
Reviewer: Sponsor:

Description

Try changing the locale via the settings. The result is the app crashing:

10-30 11:06:17.176 18942 18942 D GeckoLocaleList: Building locales list. Current locale: sv_SE
10-30 11:06:17.191 18942 18942 E GeckoJarReader: !!! BUG 849589 !!! origUrl=jar:jar:file:/data/app/org.torproject.torbrowser-1/base.apk!/assets/omni.ja!/res/multilocale.txt
10-30 11:06:17.191 18942 18942 E GeckoJarReader: java.lang.IllegalArgumentException: Got class java.util.zip.InflaterInputStream, but expected ByteBufferInputStream!
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at org.mozilla.gecko.mozglue.NativeZip.<init>(NativeZip.java:30)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at org.mozilla.gecko.util.GeckoJarReader.getStream(GeckoJarReader.java:208)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at org.mozilla.gecko.util.GeckoJarReader.getText(GeckoJarReader.java:86)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at org.mozilla.gecko.BrowserLocaleManager.getPackagedLocaleTags(BrowserLocaleManager.java:425)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at org.mozilla.gecko.preferences.LocaleListPreference.getUsableLocales(LocaleListPreference.java:222)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at org.mozilla.gecko.preferences.LocaleListPreference.buildList(LocaleListPreference.java:293)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at org.mozilla.gecko.preferences.LocaleListPreference.<init>(LocaleListPreference.java:110)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at java.lang.reflect.Constructor.newInstance(Native Method)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.preference.GenericInflater.createItem(GenericInflater.java:385)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.preference.GenericInflater.createItemFromTag(GenericInflater.java:432)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.preference.GenericInflater.rInflate(GenericInflater.java:483)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.preference.GenericInflater.rInflate(GenericInflater.java:495)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.preference.GenericInflater.inflate(GenericInflater.java:327)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.preference.GenericInflater.inflate(GenericInflater.java:264)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.preference.PreferenceManager.inflateFromResource(PreferenceManager.java:274)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.preference.PreferenceFragment.addPreferencesFromResource(PreferenceFragment.java:302)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at org.mozilla.gecko.preferences.GeckoPreferenceFragment.loadPreferences(GeckoPreferenceFragment.java:201)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at org.mozilla.gecko.preferences.GeckoPreferenceFragment.onCreate(GeckoPreferenceFragment.java:73)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.Fragment.performCreate(Fragment.java:2242)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:948)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:1164)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.BackStackRecord.run(BackStackRecord.java:793)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.FragmentManagerImpl.execPendingActions(FragmentManager.java:1557)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.FragmentController.execPendingActions(FragmentController.java:326)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.Activity.performStart(Activity.java:6942)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3276)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3415)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.ActivityThread.access$1100(ActivityThread.java:229)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1821)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.os.Handler.dispatchMessage(Handler.java:102)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.os.Looper.loop(Looper.java:148)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at android.app.ActivityThread.main(ActivityThread.java:7331)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at java.lang.reflect.Method.invoke(Native Method)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1230)
10-30 11:06:17.191 18942 18942 E GeckoJarReader: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1120)
10-30 11:06:17.191 18942 18942 D AndroidRuntime: Shutting down VM
10-30 11:06:17.191 18942 18942 E AndroidRuntime: FATAL EXCEPTION: main
10-30 11:06:17.191 18942 18942 E AndroidRuntime: Process: org.torproject.torbrowser, PID: 18942
10-30 11:06:17.191 18942 18942 E AndroidRuntime: java.lang.RuntimeException: Unable to start activity ComponentInfo{org.torproject.torbrowser/org.mozilla.gecko.preferences.GeckoPreferences}: android.view.InflateException: Binary XML file line #5: Error inflating class java.lang.reflect.Constructor
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3319)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3415)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.ActivityThread.access$1100(ActivityThread.java:229)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1821)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.os.Handler.dispatchMessage(Handler.java:102)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.os.Looper.loop(Looper.java:148)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.ActivityThread.main(ActivityThread.java:7331)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at java.lang.reflect.Method.invoke(Native Method)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1230)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1120)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: Caused by: android.view.InflateException: Binary XML file line #5: Error inflating class java.lang.reflect.Constructor
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.preference.GenericInflater.createItem(GenericInflater.java:399)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.preference.GenericInflater.createItemFromTag(GenericInflater.java:432)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.preference.GenericInflater.rInflate(GenericInflater.java:483)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.preference.GenericInflater.rInflate(GenericInflater.java:495)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.preference.GenericInflater.inflate(GenericInflater.java:327)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.preference.GenericInflater.inflate(GenericInflater.java:264)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.preference.PreferenceManager.inflateFromResource(PreferenceManager.java:274)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.preference.PreferenceFragment.addPreferencesFromResource(PreferenceFragment.java:302)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at org.mozilla.gecko.preferences.GeckoPreferenceFragment.loadPreferences(GeckoPreferenceFragment.java:201)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at org.mozilla.gecko.preferences.GeckoPreferenceFragment.onCreate(GeckoPreferenceFragment.java:73)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.Fragment.performCreate(Fragment.java:2242)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:948)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:1164)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.BackStackRecord.run(BackStackRecord.java:793)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.FragmentManagerImpl.execPendingActions(FragmentManager.java:1557)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.FragmentController.execPendingActions(FragmentController.java:326)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.Activity.performStart(Activity.java:6942)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3276)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	... 9 more
10-30 11:06:17.191 18942 18942 E AndroidRuntime: Caused by: java.lang.reflect.InvocationTargetException
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at java.lang.reflect.Constructor.newInstance(Native Method)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at android.preference.GenericInflater.createItem(GenericInflater.java:385)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	... 26 more
10-30 11:06:17.191 18942 18942 E AndroidRuntime: Caused by: java.lang.IllegalArgumentException: !!! BUG 849589 !!! origUrl=jar:jar:file:/data/app/org.torproject.torbrowser-1/base.apk!/assets/omni.ja!/res/multilocale.txt
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at org.mozilla.gecko.util.GeckoJarReader.getStream(GeckoJarReader.java:212)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at org.mozilla.gecko.util.GeckoJarReader.getText(GeckoJarReader.java:86)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at org.mozilla.gecko.BrowserLocaleManager.getPackagedLocaleTags(BrowserLocaleManager.java:425)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at org.mozilla.gecko.preferences.LocaleListPreference.getUsableLocales(LocaleListPreference.java:222)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at org.mozilla.gecko.preferences.LocaleListPreference.buildList(LocaleListPreference.java:293)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	at org.mozilla.gecko.preferences.LocaleListPreference.<init>(LocaleListPreference.java:110)
10-30 11:06:17.191 18942 18942 E AndroidRuntime: 	... 28 more

Child Tickets

Change History (22)

comment:1 in reply to:  description Changed 2 weeks ago by sysrqb

Replying to gk:

Try changing the locale via the settings. The result is the app crashing:

10-30 11:06:17.191 18942 18942 E GeckoJarReader: !!! BUG 849589 !!! origUrl=jar:jar:file:/data/app/org.torproject.torbrowser-1/base.apk!/assets/omni.ja!/res/multilocale.txt
10-30 11:06:17.176 18942 18942 D GeckoLocaleList: Building locales list. Current locale: sv_SE

Opened 7 years ago Closed 6 years ago
https://bugzilla.mozilla.org/show_bug.cgi?id=849589

Hrm. I'll need to look at this more tomorrow.

[edit: added correct quote-line]

Last edited 2 weeks ago by sysrqb (previous) (diff)

comment:2 Changed 2 weeks ago by sysrqb

Lucky for us, Mozilla saw this in their 65.0b4 release, which points us at exactly the same problem.
https://bugzilla.mozilla.org/show_bug.cgi?id=1513870

$ unzip -lv ~/tor-browser-9.0-android-x86-multi.apk | grep omni.ja
10875447  Defl:N 10295450   5% 2000-01-01 00:00 c9c4b19e  assets/omni.ja

GeKo thinks this could be fallout from our solution in #31564

comment:3 Changed 13 days ago by sysrqb

9.0a7:

 1174440  Defl:N   417613  64% 2000-01-01 00:00 3169aef4  lib/x86_64/libmozglue.so
 1781088  Defl:N   593355  67% 2000-01-01 00:00 dcacddb5  lib/x86_64/libmozavcodec.so
 1858081  Defl:N  1787604   4% 2000-01-01 00:00 6e9c0248  assets/distribution/extensions/https-everywhere-eff@eff.org.xpi
 1884528  Defl:N   912276  52% 2000-01-01 00:00 025746a3  lib/x86_64/libnss3.so
 2339663  Defl:X   269250  89% 2010-01-01 00:00 bb2bb5ce  assets/common/geoip6
 4073678  Defl:X  1190281  71% 2010-01-01 00:00 2c54d862  assets/common/geoip
 5720092  Defl:X  2443472  57% 2010-01-01 00:00 401195d9  classes.dex
 7105248  Defl:N  2391049  66% 2000-01-01 00:00 2bb41c2d  lib/x86_64/libObfs4proxy.so
 7478968  Defl:N  2698159  64% 2000-01-01 00:00 f391dcb9  lib/x86_64/libTor.so
10836161  Stored 10836161   0% 2010-01-01 00:00 71404561  assets/omni.ja
86656560  Defl:N 32635129  62% 2000-01-01 00:00 355d0169  lib/x86_64/libxul.so
137422584         61169376  56%                            1561 files

9.0a8:

 1174440  Defl:N   417612  64% 2000-01-01 00:00 fa4ea2be  lib/x86_64/libmozglue.so
 1781088  Defl:N   593362  67% 2000-01-01 00:00 e0620690  lib/x86_64/libmozavcodec.so
 1858081  Defl:N  1787604   4% 2000-01-01 00:00 6e9c0248  assets/distribution/extensions/https-everywhere-eff@eff.org.xpi
 1884528  Defl:N   912275  52% 2000-01-01 00:00 9f197d32  lib/x86_64/libnss3.so
 2339663  Defl:N   295431  87% 2000-01-01 00:00 bb2bb5ce  assets/common/geoip6
 4073678  Defl:N  1191956  71% 2000-01-01 00:00 2c54d862  assets/common/geoip
 5720064  Defl:N  2453573  57% 2000-01-01 00:00 be25d81c  classes.dex
 7105248  Defl:N  2391049  66% 2000-01-01 00:00 2bb41c2d  lib/x86_64/libObfs4proxy.so
 7478968  Defl:N  2698159  64% 2000-01-01 00:00 f391dcb9  lib/x86_64/libTor.so
10866315  Defl:N 10307038   5% 2000-01-01 00:00 152db749  assets/omni.ja
86656560  Defl:N 32634495  62% 2000-01-01 00:00 0dec3529  lib/x86_64/libxul.so
137614012         59830866  57%                            1561 files

So, comparing 9.0a7 and 9.0a8 show omni.ja was simply stored in 9.0a7 (indicating it was already compressed), while it was compressed using deflate in 9.0a8. We can see the size of the apk decreased, as well (presumably because an inflated omni.ja allowed higher overall compression of the apk):

-rw-r--r-- 1 sysrqb sysrqb 59M Oct 31 15:53 tor-browser-9.0a7-android-x86_64-multi.apk
-rw-r--r-- 1 sysrqb sysrqb 58M Oct 31 15:53 tor-browser-9.0a8-android-x86_64-multi.apk

comment:4 Changed 13 days ago by Horizon78

Also affected is the function to change the language! ... The error is reproducible! ... Crashes every time you go to settings to change the language! ... Please fix !!! thanks

Version: 9.0

68.2.0
Firefox for Android Release

October 22, 2019
Version 68.2.0, first offered to Release channel users on October 22, 2019

Last edited 13 days ago by Horizon78 (previous) (diff)

comment:5 Changed 13 days ago by sisbell

We've seen the omni.ja problem before. The original approach was to just delete and add what we didn't need or needed respectively to the final apk. This meant, we weren't unzipping and zipping the apk. With the ApkTool approach for reproducibility, it's repackaging without the correct compression. So I suspect its a problem there.

comment:6 in reply to:  4 Changed 13 days ago by sysrqb

Replying to Horizon78:

Also affected is the function to change the language! ... The error is reproducible! ... Crashes every time you go to settings to change the language! ... Please fix !!! thanks

Yes, sorry if that wasn't clear, that is the purpose of this bug.

comment:7 in reply to:  5 ; Changed 13 days ago by sysrqb

Replying to sisbell:

We've seen the omni.ja problem before. The original approach was to just delete and add what we didn't need or needed respectively to the final apk. This meant, we weren't unzipping and zipping the apk. With the ApkTool approach for reproducibility, it's repackaging without the correct compression. So I suspect its a problem there.

It seems this is due to unzipping and then zipping the entire archive. zip decides if a file should be stored or deflated automatically. The man page says

zip automatically chooses the better of the two (deflation or store or, if
       bzip2 is selected, bzip2 or store) for each file to be compressed.

We can ignore the bzip2 part of that.

One terribly hacky solution is using the current patch, but then deleting assets/omni.ja from the apk and then re-adding it using zip -Z store assets/omni.ja, effectively forcing zip not deflating omni.ja. I don't remember the details of #31564 well enough right now to remember if there is a better alternative right now.

I worry there are other bugs lurking as a result of our unzip/zip, but I don't have a better solution right now.

comment:8 in reply to:  7 Changed 13 days ago by sysrqb

Replying to sysrqb:

I worry there are other bugs lurking as a result of our unzip/zip, but I don't have a better solution right now.

In fact:

$ unzip -vl tor-browser-9.0a7-android-x86_64-multi.apk | grep -c Stored
981
$ unzip -vl tor-browser-9.0a8-android-x86_64-multi.apk | grep -c Stored
45
$ unzip -vl tor-browser-9.0a7-android-x86_64-multi.apk | grep -c Defl:N
17
$ unzip -vl tor-browser-9.0a8-android-x86_64-multi.apk | grep -c Defl:N
1516

These numbers aren't very helpful, except showing the majority of files in 9.0a7 are Stored while the majority of files in 9.0a8 are Defl:N.

A little more informative:

$ unzip -vl tor-browser-9.0a7-android-x86_64-multi.apk | grep Defl:N
 1858081  Defl:N  1787604   4% 2000-01-01 00:00 6e9c0248  assets/distribution/extensions/https-everywhere-eff@eff.org.xpi
  548510  Defl:N   537879   2% 2000-01-01 00:00 f556e20f  assets/distribution/extensions/{73a6fe31-595d-460b-a920-fcc0f8843232}.xpi
 7105248  Defl:N  2391049  66% 2000-01-01 00:00 2bb41c2d  lib/x86_64/libObfs4proxy.so
 7478968  Defl:N  2698159  64% 2000-01-01 00:00 f391dcb9  lib/x86_64/libTor.so
  510160  Defl:N   210494  59% 2000-01-01 00:00 d10747be  lib/x86_64/libfreebl3.so
   30728  Defl:N    12928  58% 2000-01-01 00:00 07c892e9  lib/x86_64/liblgpllibs.so
 1781088  Defl:N   593355  67% 2000-01-01 00:00 dcacddb5  lib/x86_64/libmozavcodec.so
  186608  Defl:N    71807  62% 2000-01-01 00:00 fe062a40  lib/x86_64/libmozavutil.so
 1174440  Defl:N   417613  64% 2000-01-01 00:00 3169aef4  lib/x86_64/libmozglue.so
 1884528  Defl:N   912276  52% 2000-01-01 00:00 025746a3  lib/x86_64/libnss3.so
  450840  Defl:N   181388  60% 2000-01-01 00:00 ed1e782b  lib/x86_64/libnssckbi.so
    6144  Defl:N     1701  72% 2000-01-01 00:00 4fb02e64  lib/x86_64/libplugin-container.so
  248776  Defl:N   103114  59% 2000-01-01 00:00 36153725  lib/x86_64/libsoftokn3.so
86656560  Defl:N 32635129  62% 2000-01-01 00:00 355d0169  lib/x86_64/libxul.so
  179977  Defl:N    64862  64% 2010-01-01 00:00 0ae40655  META-INF/TBA_ALPH.SF
    1791  Defl:N     1522  15% 2010-01-01 00:00 9789b962  META-INF/TBA_ALPH.RSA
  179850  Defl:N    64218  64% 2010-01-01 00:00 171cbda6  META-INF/MANIFEST.MF

I don't know what change was introduced between a7 and a8, but there are a lot of files that changed between them.

comment:9 Changed 13 days ago by sisbell

The last time we encountered the issue was #28144

comment:10 Changed 12 days ago by sysrqb

More information. These two apks are built using the same code (same tor browser tags), except the first apk was not decompiled/recompiled with apktools, and the second one was decompiled/recompiled.

$ unzip -vl tor-browser-tbb-nightly-android-x86_64-multi-qa.apk | grep -c Stored
992
$ unzip -vl tor-browser-tbb-nightly-android-x86_64-multi-qa.apk | grep -c Defl:N
17
$ unzip -vl tor-browser-tbb-nightly-android-x86_64-multi-qa.apk | grep -c Defl:X
601
$ unzip -vl tor-browser-tbb-nightly-android-x86_64-multi-qa-apktools.apk | grep -c Stored
50
$ unzip -vl tor-browser-tbb-nightly-android-x86_64-multi-qa-apktools.apk | grep -c Defl:N
1523
$ unzip -vl tor-browser-tbb-nightly-android-x86_64-multi-qa-apktools.apk | grep -c Defl:X
0

Interestingly, the apk after undergoing the apktools operations has nearly 40 fewer files than the original:

$ unzip -vl tor-browser-tbb-nightly-android-x86_64-multi-qa.apk | tail
 1884528  Defl:N   912273  52% 2000-01-01 00:00 ce735645  lib/x86_64/libnss3.so
  450840  Defl:N   181385  60% 2000-01-01 00:00 aa82c9ac  lib/x86_64/libnssckbi.so
    6144  Defl:N     1699  72% 2000-01-01 00:00 3fcc6519  lib/x86_64/libplugin-container.so
  248776  Defl:N   103213  59% 2000-01-01 00:00 7c9b3218  lib/x86_64/libsoftokn3.so
86677040  Defl:N 32647854  62% 2000-01-01 00:00 72313002  lib/x86_64/libxul.so
  185868  Defl:N    67119  64% 2010-01-01 00:00 e03d0b4b  META-INF/ANDROIDQ.SF
    1202  Defl:N     1061  12% 2010-01-01 00:00 c37c148a  META-INF/ANDROIDQ.RSA
  185806  Defl:N    65400  65% 2010-01-01 00:00 1f308f3c  META-INF/MANIFEST.MF
--------          -------  ---                            -------
140875793         62633420  56%                            1610 files

$ unzip -vl tor-browser-tbb-nightly-android-x86_64-multi-qa-apktools.apk | tail
     548  Defl:N      306  44% 2000-01-01 00:00 f4fe1304  res/xml/search_widget_info.xml
     528  Defl:N      309  42% 2000-01-01 00:00 61084447  res/xml/searchable.xml
     480  Defl:N      273  43% 2000-01-01 00:00 a65db3b7  res/xml/separator.xml
  989460  Defl:N   182780  82% 2000-01-01 00:00 0a521646  resources.arsc
      10  Defl:N        9  10% 2000-01-01 00:00 74f0468e  ua-update.json
  178021  Defl:N    65101  63% 2000-01-01 00:00 4544c219  META-INF/ANDROIDQ.SF
    1202  Defl:N     1060  12% 2000-01-01 00:00 d3bd34b2  META-INF/ANDROIDQ.RSA
  177897  Defl:N    64294  64% 2000-01-01 00:00 1ba5281c  META-INF/MANIFEST.MF
--------          -------  ---                            -------
141003330         61214222  57%                            1573 files

I don't know the tradeoffs of Defl:N vs. Defl:X. At this point I'll write a patch for the "terrible hack" I mentioned earlier (explicitly force zip "storing" omni.ja), unless someone else has a better idea.

comment:11 Changed 12 days ago by gk

We already have

unzip $apk -d tmp
rm $apk
cd tmp

so, rezipping omni.ja there while we are at it would work, no? (not sure whether that's what you meant).

That said is omni.ja the only file affected here or do we need to redo more?

comment:12 in reply to:  11 Changed 12 days ago by sysrqb

Replying to gk:

We already have

unzip $apk -d tmp
rm $apk
cd tmp

so, rezipping omni.ja there while we are at it would work, no? (not sure whether that's what you meant).

omni.ja is still zipped at this point. It seems the problem is the next step where the files are zipped again. It is here where zip compresses omni.ja a second time, instead of simply storing the file in the archive, unchanged.

We can correct this after we rezip the apk files by deleting assets/omni.ja in the archive and then re-add it, except this time we explicitly say it should only be stored. I'm thinking about something like this:

$ unzip tor-browser-9.0a8-android-x86_64-multi.apk assets/omni.ja
Archive:  tor-browser-9.0a8-android-x86_64-multi_omnija_store.apk
  inflating: assets/omni.ja          
$ zip -d tor-browser-9.0a8-android-x86_64-multi.apk assets/omni.ja
deleting: assets/omni.ja
$ zip -Z store tor-browser-9.0a8-android-x86_64-multi.apk assets/omni.ja
  adding: assets/omni.ja (stored 0%)

That said is omni.ja the only file affected here or do we need to redo more?

Okay, I think I understand the difference between Defl:N and Defl:X now. From the zipinfo man page (with some snipping for clarity):

There are six methods known at present:  storing (no compression), [...], and deflating.
In addition, there are [...]  four levels of deflating (superfast, fast, normal,
maximum compression).  zipinfo represents these methods and their sub-methods as follows:
[...], defS, defF, defN, and defX.

It seems simply zipping the apk files really messes with the how the files are added into the archive. I don't have a good understanding of the impact besides the effect it has on omni.ja.

I wonder if we should use aapt from the Android build-tools in the future instead of zip:

 aapt a[dd] [-v] file.{zip,jar,apk} file1 [file2 ...]
   Add specified files to Zip-compatible archive.

comment:13 Changed 12 days ago by sysrqb

Keywords: TorBrowserTeam201911R added; TorBrowserTeam201910 removed
Status: newneeds_review

Okay, it looks like this patch works (at least as far as it stores omni.ja instead of deflating it).

$ unzip -vl testbuild/tor-browser-tbb-nightly-android-x86_64-multi-qa.apk | grep omni.ja
11117897  Stored 11117897   0% 2000-01-01 00:00 6516d0c2  assets/omni.ja

I pushed branch bug32342_00. I tested it and changing locales does not result in the app crashing anymore.

comment:14 in reply to:  11 Changed 12 days ago by sysrqb

Replying to gk:

That said is omni.ja the only file affected here or do we need to redo more?

I opened #32360 as a follow up for this.

comment:15 in reply to:  13 ; Changed 12 days ago by gk

Resolution: fixed
Status: needs_reviewclosed

Replying to sysrqb:

Okay, it looks like this patch works (at least as far as it stores omni.ja instead of deflating it).

$ unzip -vl testbuild/tor-browser-tbb-nightly-android-x86_64-multi-qa.apk | grep omni.ja
11117897  Stored 11117897   0% 2000-01-01 00:00 6516d0c2  assets/omni.ja

I pushed branch bug32342_00. I tested it and changing locales does not result in the app crashing anymore.

Looks good to me. I wonder whether the fix has any new reproducibility implications. Like, are the new zip incantations causing issues? I guess, we'll find that out soon... ;) Anyway, I applied the patch to master (commit 3c7fb0e610a4f85d7b157221ac9049d99d958c24) and maint-9.0 (commit 3d5f4308312bdece464dba6b55ef0be33f6bb257) so that both the upcoming 9.5a1 and 9.0.1 have the fix.

comment:16 in reply to:  15 Changed 9 days ago by gk

Resolution: fixed
Status: closedreopened

Replying to gk:

Replying to sysrqb:

Okay, it looks like this patch works (at least as far as it stores omni.ja instead of deflating it).

$ unzip -vl testbuild/tor-browser-tbb-nightly-android-x86_64-multi-qa.apk | grep omni.ja
11117897  Stored 11117897   0% 2000-01-01 00:00 6516d0c2  assets/omni.ja

I pushed branch bug32342_00. I tested it and changing locales does not result in the app crashing anymore.

Looks good to me. I wonder whether the fix has any new reproducibility implications. Like, are the new zip incantations causing issues? I guess, we'll find that out soon... ;)

Yeah, and it seems my fear was not unsubstantiated here...

comment:17 Changed 9 days ago by boklm

Maybe we could use the zip option -n ja to avoid compressing omni.ja, without having to remove it and add it again.

comment:18 in reply to:  17 ; Changed 9 days ago by gk

Replying to boklm:

Maybe we could use the zip option -n ja to avoid compressing omni.ja, without having to remove it and add it again.

That sounds like a smart idea. Do you want to try that?

comment:19 in reply to:  18 Changed 9 days ago by boklm

Replying to gk:

Replying to boklm:

Maybe we could use the zip option -n ja to avoid compressing omni.ja, without having to remove it and add it again.

That sounds like a smart idea. Do you want to try that?

Ok, I'm now testing a 9.0.1 build with this patch:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_32342-9.0.1&id=c13d170d9182d9f8809734995ae8cfdd721b5faf

comment:20 Changed 9 days ago by boklm

Status: reopenedneeds_review

The branch bug_32342-9.0.1 includes the revert of the previous commit, and the commit using the zip -n ja option:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_32342-9.0.1&id=c13d170d9182d9f8809734995ae8cfdd721b5faf

I did two builds with this patch on 9.0.1 (only rebuilding the projects/tor-browser part), and got matching apks. unzip -vl also shows that omni.ja is Stored, so this seems to be working.

The branch bug_32342-master has the same patches for the master branch:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_32342-master&id=631ff5a05eda26da61f75f175c056142c171640a

comment:21 in reply to:  20 Changed 9 days ago by gk

Resolution: fixed
Status: needs_reviewclosed

Replying to boklm:

The branch bug_32342-9.0.1 includes the revert of the previous commit, and the commit using the zip -n ja option:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_32342-9.0.1&id=c13d170d9182d9f8809734995ae8cfdd721b5faf

I did two builds with this patch on 9.0.1 (only rebuilding the projects/tor-browser part), and got matching apks. unzip -vl also shows that omni.ja is Stored, so this seems to be working.

The branch bug_32342-master has the same patches for the master branch:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_32342-master&id=631ff5a05eda26da61f75f175c056142c171640a

Thanks. I've picked this up to redo the .apks for 9.0.1 and merged your stable branch to maint-9.0 (commits c80c3516ce9e4eada876fdcbb8361749097cd984 and c13d170d9182d9f8809734995ae8cfdd721b5faf). I did the same for master with your master version (commits d8f62baa1d4436e90db842cc8ab42b4454b87684 and 631ff5a05eda26da61f75f175c056142c171640a).

Please, everyone, add your points to this ticket.

comment:22 Changed 9 days ago by boklm

Actual Points: .2

Adding my points.

Note: See TracTickets for help on using tickets.