Remove apktool workaround in #31564
We found a reproducibility issue on Android with the switch to Firefox 68 ESR and the respective toolchain and fixed it by using an apktool downloaded from the Internet. We should remove that workaronud and replace it with a better one (e.g. by switching our compile environment to Debian Buster and using the means the distro provides us with).
Activity
Trac:
Parent Ticket: #33659 (moved)
Note: the switch to Debian 10 (aka Buster) is done in #31130 (moved).
One potential problem with using
apktoolto assemble actual releases: it is considered a suspicious mark since these reassembly techniques are mostly used by malware: https://rednaga.io/2016/07/31/detecting_pirated_and_malicious_android_apps_with_apkid/I tried installing apktool when building the container. I download it
https://deb.debian.org/debian/pool/main/a/apktool/apktool_2.4.0-1_all.deb
and use
dpkg -i ./apktool_2.4.0-1_all.debI get a bunch of missing dependencies so I'm not sure this is the simplest approach
apt version: 1.8.2 Selecting previously unselected package apktool. (Reading database ... 4868 files and directories currently installed.) Preparing to unpack ./apktool_2.4.0-1_all.deb ... Unpacking apktool (2.4.0-1) ... dpkg: dependency problems prevent configuration of apktool: apktool depends on aapt; however: Package aapt is not installed. apktool depends on android-framework-res; however: Package android-framework-res is not installed. apktool depends on default-jre-headless | java8-runtime-headless; however: Package default-jre-headless is not installed. Package java8-runtime-headless is not installed. apktool depends on libantlr3-runtime-java; however: Package libantlr3-runtime-java is not installed. apktool depends on libcommons-cli-java; however: Package libcommons-cli-java is not installed. apktool depends on libcommons-io-java; however: Package libcommons-io-java is not installed. apktool depends on libcommons-lang3-java; however: Package libcommons-lang3-java is not installed. apktool depends on libguava-java; however: Package libguava-java is not installed. apktool depends on libsmali-java (>= 2.2.1); however: Package libsmali-java is not installed. apktool depends on libstringtemplate-java; however: Package libstringtemplate-java is not installed. apktool depends on libxmlunit-java; however: Package libxmlunit-java is not installed. apktool depends on libxpp3-java; however: Package libxpp3-java is not installed. apktool depends on libyaml-snake-java; however: Package libyaml-snake-java is not installed. dpkg: error processing package apktool (--install): dependency problems - leaving unconfigured Errors were encountered while processing: apktool-
Okay, so two questions here:
-
I've been skimming the comments in #31564 (moved) again but did not find an answer to my first question in comment:29:ticket:31564: Do we know whether apktool 2.3.4 that comes natively with Buster works or not? https://github.com/iBotPeaches/Apktool/issues/1399 gives conflicting information. If it does work there is no need to do some
apktoolrelated gymnastics here. -
I am a bit confused about the approach of picking a newer
apktool.deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?
-
Replying to gk:
Okay, so two questions here:
- I've been skimming the comments in #31564 (moved) again but did not find an answer to my first question in comment:29:ticket:31564: Do we know whether apktool 2.3.4 that comes natively with Buster works or not? The one in buster wasn't working when I tried. It listed as 2.3.4 but all of the dependencies were much older 2.2, so I suspect the packaging was wrong, it wasn't really updated to 2.3.x.
In the downloaded deb we see
apktool depends on libsmali-java (>= 2.2.1); however:If we can download 2.3 for these dependencies, it should work
https://github.com/iBotPeaches/Apktool/issues/1399 gives conflicting information. If it does work there is no need to do some
aptktoolrelated gymnastics here.- I am a bit confused about the approach of picking a newer
apktool.deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?
Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.
Replying to sisbell:
Replying to gk: In the downloaded deb we see
{{{ apktool depends on libsmali-java (>= 2.2.1); however: }}}
If we can download 2.3 for these dependencies, it should work
Can we use apt pinning like eighhave suggested? There's no reason why we should resolve the dependencies ourselves.
- I am a bit confused about the approach of picking a newer
apktool.deb in the face of Debian's bug 942019. How would that .deb work around the problem mentioned there?https://packages.debian.org/sid/apktool
Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.
It seems like there was already an upstream bugreport which was closed as not-a-bug. https://github.com/iBotPeaches/Apktool/issues/2149
But I don't know enough to say where the bug is.
- I am a bit confused about the approach of picking a newer
Replying to sysrqb:
Replying to sisbell:
Replying to gk: In the downloaded deb we see
{{{ apktool depends on libsmali-java (>= 2.2.1); however: }}}
If we can download 2.3 for these dependencies, it should work
Can we use apt pinning like eighhave suggested? There's no reason why we should resolve the dependencies ourselves.
Initially, this was the approach I was taking but with openjdk, I ran into various issues with pinning (see #31130 (moved)). bolkm suggestion was to just download and install deb and packages directly. So far this is looking cleaner and I'm implementing now for openjdk. I'd like to keep the same approach with apktool and openjdk, if possible.
- I am a bit confused about the approach of picking a newer
apktool.deb in the face of Debian's bug 942019. How would that .deb work around the problem mentioned there?https://packages.debian.org/sid/apktool
Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.
It seems like there was already an upstream bugreport which was closed as not-a-bug. https://github.com/iBotPeaches/Apktool/issues/2149
But I don't know enough to say where the bug is.
- I am a bit confused about the approach of picking a newer
-
Replying to sisbell:
Replying to gk:
Okay, so two questions here:
- I've been skimming the comments in #31564 (moved) again but did not find an answer to my first question in comment:29:ticket:31564: Do we know whether apktool 2.3.4 that comes natively with Buster works or not? The one in buster wasn't working when I tried. It listed as 2.3.4 but all of the dependencies were much older 2.2, so I suspect the packaging was wrong, it wasn't really updated to 2.3.x.
What error did you get when trying with 2.3.4?
In the downloaded deb we see
{{{ apktool depends on libsmali-java (>= 2.2.1); however: }}}
If we can download 2.3 for these dependencies, it should work
What makes you believe the packaging was wrong?
aptktooldepends for instance onaapt(1:8.1.0+r23-3), so why can't it depend onlibsmali-java(>= 2.2.1)? -
Replying to sisbell:
Replying to gk:
[snip]
- I am a bit confused about the approach of picking a newer
apktool.deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?
Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.
Could you expand on that comment as I don't understand how it addresses my question? How are we avoiding the problem if we download the .deb with all dependencies from Debian if using the that .deb is problematic?
- I am a bit confused about the approach of picking a newer
Replying to gk:
Replying to sisbell:
Replying to gk:
[snip]
- I am a bit confused about the approach of picking a newer
apktool.deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?
Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.
Could you expand on that comment as I don't understand how it addresses my question? How are we avoiding the problem if we download the .deb with all dependencies from Debian if using the that .deb is problematic?
ApkTool is really just a wrapper around a bunch of other libraries. It's the dependent libraries we need to update. For example, we know that the deb will use smali-lib 2.2.1 (which is part of the set of older libraries with the problem) but is compatible with 2.3. So I was thinking if we can install something like libsmali-java 2.3, ApkTool should work.
- I am a bit confused about the approach of picking a newer
Replying to sisbell:
Replying to gk:
Replying to sisbell:
Replying to gk:
[snip]
- I am a bit confused about the approach of picking a newer
apktool.deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?
Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.
Could you expand on that comment as I don't understand how it addresses my question? How are we avoiding the problem if we download the .deb with all dependencies from Debian if using the that .deb is problematic?
ApkTool is really just a wrapper around a bunch of other libraries. It's the dependent libraries we need to update. For example, we know that the deb will use smali-lib 2.2.1 (which is part of the set of older libraries with the problem) but is compatible with 2.3. So I was thinking if we can install something like libsmali-java 2.3, ApkTool should work.
What makes you say that the issue is with the smali version?
When I unzip the
apktool_2.4.0.jarwe currently use, I can see that the filesmali.propertiescontains:application.version=2.2.6So it seems that
apktool_2.4.0.jarincludes smali version2.2.6. And thelibsmali-javapackage in buster is version2.2.6too, so it looks like we are using the same smali version in both cases.- I am a bit confused about the approach of picking a newer
-
Replying to sisbell:
Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.
Is aapt really the same in both cases?
I see that
apktool_2.4.0.jarincludes some files likeprebuilt/linux/aapt_64, which looks like a build of aapt. However I don't know which version it is. I tried the buster version (2.3.4-dirty) and then a custom deb install (2.4.0-dirty) and neither one works. At this point I think the report that this is an aapt issue (and not an apktool issue) is probably accurate.
The downloaded version of apktool which we are currently using has a packaged version of its own build of aapt. This one works
Android Asset Packaging Tool, v0.2-eng.ibotpe.20190216.092407While buster uses the following which doesn't work
Android Asset Packaging Tool, v0.2-27.0.1It difficult to know what the difference is due to the custom build. Both look to be some variation of 0.2 versions
As outlined in the original issue, the following resource(s) cause aapt to fail
W: res/drawable-v21/$avd_hide_password__0.xml: Invalid file name: must contain only [a-z0-9_.] W: res/drawable-v21/$avd_hide_password__1.xml: Invalid file name: must contain only [a-z0-9_.] W: res/drawable-v21/$avd_hide_password__2.xml: Invalid file name: must contain only [a-z0-9_.] W: res/drawable-v21/$avd_show_password__0.xml: Invalid file name: must contain only [a-z0-9_.] W: res/drawable-v21/$avd_show_password__1.xml: Invalid file name: must contain only [a-z0-9_.] W: res/drawable-v21/$avd_show_password__2.xml: Invalid file name: must contain only [a-z0-9_.]So after running the debugger, I could see everything was fine until the process execution of aapt. After digging through source code in various repos, I found the following commit for the custom aapt that iBotPeaches is including in the apktool. This patch ignores illegal characters in resource file names.
So there is no way the aapt packaged in Debian (regardless of version) is going to work without this patch. This is not a bug in the Debian version as aapt should reject resources.
Unless we are planning on removing apktool completely and trying another approach, we can close this issue.
For reference these resources are coming from a google support library
https://developer.android.com/reference/android/support/design/R.drawable.html#avd_show_password
-
apktool is first and foremost an inspection and reverse engineering tool. Using it in production releases will never be well supported. The marks it leaves on the APK make malware scanners consider the APK malware, since that's the vast majority of APKs assembled with apktool are malware, at least according to the sources I've discusses this with.
If you're going to make custom build tools, I think it'll be a lot more effective to customize the official Android build tools as needed, as painful as building them is. For example, the Debian Android Tools team would love to include reproducible patches in the packages, even though the Google binaries do not, as long as they don't break compatibility. Same with this project: https://android-rebuilds.beuc.net/
-
Trac:
Cc: sisbell, tbb-team to tbb-team
Description: We found a reproducibility issue on Android with the switch to Firefox 68 ESR and the respective toolchain and fixed it by using anapktooldownloaded from the Internet. We should remove that workaronud and replace it with a better one (e.g. by switching our compile environment to Debian Buster and using the means the distro provides us with.to
We found a reproducibility issue on Android with the switch to Firefox 68 ESR and the respective toolchain and fixed it by using an
apktooldownloaded from the Internet. We should remove that workaronud and replace it with a better one (e.g. by switching our compile environment to Debian Buster and using the means the distro provides us with). -
I talked with @boklm about this at the Reproducible Builds summit. I've now forgotten what the original reproducible issue was that this ticket is dealing with but I do remember that I think the best way forward is to work with https://android-rebuilds.beuc.net/ to rebuild the things TBB needs with reproducibility fixes included. This would mostly mean backporting them, since I think the current versions from Google have no known reproducibility issues.
-
The original issue is: https://issuetracker.google.com/issues/110237303?pli=1
Apparently it's fixed in more recent versions of Android Gradle Plugin: I added sorting the input files in AGP in change list with Change-Id: I371304c8a6d244f8bb7833609f464eba000e608f - it should be available in android gradle plugin version 3.5 Canary 7 We cherry picked it into 3.4 and it should be available in the next 3.4 canary. We'll see what we can do about 3.3.
Trac:
Parent: N/A to #33659 (moved)mentioned in issue #33659 (moved)