Opened 13 months ago

Last modified 5 months ago

#31992 assigned defect

Remove apktool workaround in #31564

Reported by: gk Owned by: sisbell
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, ReleaseTrainMigration, TorBrowserTeam202006, gitlab-tb-tor-browser-build
Cc: tbb-team Actual Points:
Parent ID: #33659 Points: 0.25
Reviewer: sysrqb Sponsor: Sponsor58-can

Description (last modified by gk)

We found a reproducibility issue on Android with the switch to Firefox 68 ESR and the respective toolchain and fixed it by using an apktool downloaded from the Internet. We should remove that workaronud and replace it with a better one (e.g. by switching our compile environment to Debian Buster and using the means the distro provides us with).

Child Tickets

Change History (34)

comment:1 Changed 13 months ago by gk

Note: the switch to Debian 10 (aka Buster) is done in #31130.

comment:2 Changed 13 months ago by boklm

See ticket:31564#comment:60 for the apt pinning configuration to install the apktool package from testing.

comment:3 Changed 13 months ago by sysrqb

Points: 0.25

comment:4 Changed 12 months ago by eighthave

One potential problem with using apktool to assemble actual releases: it is considered a suspicious mark since these reassembly techniques are mostly used by malware: https://rednaga.io/2016/07/31/detecting_pirated_and_malicious_android_apps_with_apkid/

comment:5 Changed 12 months ago by sisbell

I tried installing apktool when building the container. I download it

https://deb.debian.org/debian/pool/main/a/apktool/apktool_2.4.0-1_all.deb

and use

dpkg -i ./apktool_2.4.0-1_all.deb

I get a bunch of missing dependencies so I'm not sure this is the simplest approach

apt version: 1.8.2
Selecting previously unselected package apktool.
(Reading database ... 4868 files and directories currently installed.)
Preparing to unpack ./apktool_2.4.0-1_all.deb ...
Unpacking apktool (2.4.0-1) ...
dpkg: dependency problems prevent configuration of apktool:
 apktool depends on aapt; however:
  Package aapt is not installed.
 apktool depends on android-framework-res; however:
  Package android-framework-res is not installed.
 apktool depends on default-jre-headless | java8-runtime-headless; however:
  Package default-jre-headless is not installed.
  Package java8-runtime-headless is not installed.
 apktool depends on libantlr3-runtime-java; however:
  Package libantlr3-runtime-java is not installed.
 apktool depends on libcommons-cli-java; however:
  Package libcommons-cli-java is not installed.
 apktool depends on libcommons-io-java; however:
  Package libcommons-io-java is not installed.
 apktool depends on libcommons-lang3-java; however:
  Package libcommons-lang3-java is not installed.
 apktool depends on libguava-java; however:
  Package libguava-java is not installed.
 apktool depends on libsmali-java (>= 2.2.1); however:
  Package libsmali-java is not installed.
 apktool depends on libstringtemplate-java; however:
  Package libstringtemplate-java is not installed.
 apktool depends on libxmlunit-java; however:
  Package libxmlunit-java is not installed.
 apktool depends on libxpp3-java; however:
  Package libxpp3-java is not installed.
 apktool depends on libyaml-snake-java; however:
  Package libyaml-snake-java is not installed.

dpkg: error processing package apktool (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 apktool

comment:6 Changed 12 months ago by gk

Okay, so two questions here:

1) I've been skimming the comments in #31564 again but did not find an answer to my first question in comment:29:ticket:31564: Do we know whether apktool 2.3.4 that comes natively with Buster works or not? https://github.com/iBotPeaches/Apktool/issues/1399 gives conflicting information. If it does work there is no need to do some apktool related gymnastics here.

2) I am a bit confused about the approach of picking a newer apktool .deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?

Last edited 11 months ago by gk (previous) (diff)

comment:7 in reply to:  6 ; Changed 12 months ago by sisbell

Replying to gk:

Okay, so two questions here:

1) I've been skimming the comments in #31564 again but did not find an answer to my first question in comment:29:ticket:31564: Do we know whether apktool 2.3.4 that comes natively with Buster works or not?

The one in buster wasn't working when I tried. It listed as 2.3.4 but all of the dependencies were much older 2.2, so I suspect the packaging was wrong, it wasn't really updated to 2.3.x.

In the downloaded deb we see

 apktool depends on libsmali-java (>= 2.2.1); however:

If we can download 2.3 for these dependencies, it should work

https://github.com/iBotPeaches/Apktool/issues/1399 gives conflicting information. If it does work there is no need to do some aptktool related gymnastics here.

2) I am a bit confused about the approach of picking a newer apktool .deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?

Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.

comment:8 in reply to:  7 ; Changed 12 months ago by sysrqb

Replying to sisbell:

Replying to gk:
In the downloaded deb we see

 apktool depends on libsmali-java (>= 2.2.1); however:

If we can download 2.3 for these dependencies, it should work

Can we use apt pinning like eighhave suggested? There's no reason why we should resolve the dependencies ourselves.

2) I am a bit confused about the approach of picking a newer apktool .deb in the face of Debian's bug 942019. How would that .deb work around the problem mentioned there?https://packages.debian.org/sid/apktool

Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.

It seems like there was already an upstream bugreport which was closed as not-a-bug.
https://github.com/iBotPeaches/Apktool/issues/2149

But I don't know enough to say where the bug is.

comment:9 in reply to:  8 Changed 12 months ago by sisbell

Replying to sysrqb:

Replying to sisbell:

Replying to gk:
In the downloaded deb we see

 apktool depends on libsmali-java (>= 2.2.1); however:

If we can download 2.3 for these dependencies, it should work

Can we use apt pinning like eighhave suggested? There's no reason why we should resolve the dependencies ourselves.

Initially, this was the approach I was taking but with openjdk, I ran into various issues with pinning (see #31130). bolkm suggestion was to just download and install deb and packages directly. So far this is looking cleaner and I'm implementing now for openjdk. I'd like to keep the same approach with apktool and openjdk, if possible.

2) I am a bit confused about the approach of picking a newer apktool .deb in the face of Debian's bug 942019. How would that .deb work around the problem mentioned there?https://packages.debian.org/sid/apktool

Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.

It seems like there was already an upstream bugreport which was closed as not-a-bug.
https://github.com/iBotPeaches/Apktool/issues/2149

But I don't know enough to say where the bug is.

comment:10 in reply to:  7 Changed 12 months ago by gk

Replying to sisbell:

Replying to gk:

Okay, so two questions here:

1) I've been skimming the comments in #31564 again but did not find an answer to my first question in comment:29:ticket:31564: Do we know whether apktool 2.3.4 that comes natively with Buster works or not?

The one in buster wasn't working when I tried. It listed as 2.3.4 but all of the dependencies were much older 2.2, so I suspect the packaging was wrong, it wasn't really updated to 2.3.x.

What error did you get when trying with 2.3.4?

In the downloaded deb we see

 apktool depends on libsmali-java (>= 2.2.1); however:

If we can download 2.3 for these dependencies, it should work

What makes you believe the packaging was wrong? aptktool depends for instance on aapt (1:8.1.0+r23-3), so why can't it depend on libsmali-java (>= 2.2.1)?

comment:11 in reply to:  7 ; Changed 12 months ago by gk

Replying to sisbell:

Replying to gk:

[snip]

2) I am a bit confused about the approach of picking a newer apktool .deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?

Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.

Could you expand on that comment as I don't understand how it addresses my question? How are we avoiding the problem if we download the .deb with all dependencies from Debian if using the that .deb is problematic?

comment:12 in reply to:  11 ; Changed 12 months ago by sisbell

Replying to gk:

Replying to sisbell:

Replying to gk:

[snip]

2) I am a bit confused about the approach of picking a newer apktool .deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?

Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.

Could you expand on that comment as I don't understand how it addresses my question? How are we avoiding the problem if we download the .deb with all dependencies from Debian if using the that .deb is problematic?

ApkTool is really just a wrapper around a bunch of other libraries. It's the dependent libraries we need to update. For example, we know that the deb will use smali-lib 2.2.1 (which is part of the set of older libraries with the problem) but is compatible with 2.3. So I was thinking if we can install something like libsmali-java 2.3, ApkTool should work.

comment:13 in reply to:  12 Changed 12 months ago by boklm

Replying to sisbell:

Replying to gk:

Replying to sisbell:

Replying to gk:

[snip]

2) I am a bit confused about the approach of picking a newer apktool .deb in the face of Debian's [bug 942019 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942019]. How would that .deb work around the problem mentioned there?

Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.

Could you expand on that comment as I don't understand how it addresses my question? How are we avoiding the problem if we download the .deb with all dependencies from Debian if using the that .deb is problematic?

ApkTool is really just a wrapper around a bunch of other libraries. It's the dependent libraries we need to update. For example, we know that the deb will use smali-lib 2.2.1 (which is part of the set of older libraries with the problem) but is compatible with 2.3. So I was thinking if we can install something like libsmali-java 2.3, ApkTool should work.

What makes you say that the issue is with the smali version?

When I unzip the apktool_2.4.0.jar we currently use, I can see that the file smali.properties contains:

application.version=2.2.6

So it seems that apktool_2.4.0.jar includes smali version 2.2.6. And the libsmali-java package in buster is version 2.2.6 too, so it looks like we are using the same smali version in both cases.

comment:14 in reply to:  7 Changed 12 months ago by boklm

Replying to sisbell:

Given the one from the internet works and the one in buster doesn't, it is unlikely to have anything to do with aapt itself (which is the same in both cases), as suggested in the bugreport.

Is aapt really the same in both cases?

I see that apktool_2.4.0.jar includes some files like prebuilt/linux/aapt_64, which looks like a build of aapt. However I don't know which version it is.

comment:15 Changed 12 months ago by sysrqb

Keywords: TorBrowserTeam201911 added

comment:16 Changed 11 months ago by sysrqb

Cc: tbb-team added
Owner: changed from tbb-team to sisbell
Status: newassigned

comment:17 Changed 11 months ago by sisbell

I tried the buster version (2.3.4-dirty) and then a custom deb install (2.4.0-dirty) and neither one works. At this point I think the report that this is an aapt issue (and not an apktool issue) is probably accurate.

The downloaded version of apktool which we are currently using has a packaged version of its own build of aapt. This one works

Android Asset Packaging Tool, v0.2-eng.ibotpe.20190216.092407

While buster uses the following which doesn't work

Android Asset Packaging Tool, v0.2-27.0.1

It difficult to know what the difference is due to the custom build. Both look to be some variation of 0.2 versions

comment:18 Changed 11 months ago by sisbell

As outlined in the original issue, the following resource(s) cause aapt to fail

W: res/drawable-v21/$avd_hide_password__0.xml: Invalid file name: must contain only [a-z0-9_.]
W: res/drawable-v21/$avd_hide_password__1.xml: Invalid file name: must contain only [a-z0-9_.]
W: res/drawable-v21/$avd_hide_password__2.xml: Invalid file name: must contain only [a-z0-9_.]
W: res/drawable-v21/$avd_show_password__0.xml: Invalid file name: must contain only [a-z0-9_.]
W: res/drawable-v21/$avd_show_password__1.xml: Invalid file name: must contain only [a-z0-9_.]
W: res/drawable-v21/$avd_show_password__2.xml: Invalid file name: must contain only [a-z0-9_.]

So after running the debugger, I could see everything was fine until the process execution of aapt. After digging through source code in various repos, I found the following commit for the custom aapt that iBotPeaches is including in the apktool. This patch ignores illegal characters in resource file names.

https://github.com/iBotPeaches/platform_frameworks_base/commit/672206c5229c259e174b0a8d6d2bfdeeb753149c

So there is no way the aapt packaged in Debian (regardless of version) is going to work without this patch. This is not a bug in the Debian version as aapt should reject resources.

Unless we are planning on removing apktool completely and trying another approach, we can close this issue.


For reference these resources are coming from a google support library

https://developer.android.com/reference/android/support/design/R.drawable.html#avd_show_password

https://android.googlesource.com/platform/frameworks/support/+/support-library-27.0.2/design/res/drawable-v21/

comment:19 Changed 11 months ago by eighthave

apktool is first and foremost an inspection and reverse engineering tool. Using it in production releases will never be well supported. The marks it leaves on the APK make malware scanners consider the APK malware, since that's the vast majority of APKs assembled with apktool are malware, at least according to the sources I've discusses this with.

If you're going to make custom build tools, I think it'll be a lot more effective to customize the official Android build tools as needed, as painful as building them is. For example, the Debian Android Tools team would love to include reproducible patches in the packages, even though the Google binaries do not, as long as they don't break compatibility. Same with this project: https://android-rebuilds.beuc.net/

comment:20 Changed 11 months ago by gk

Cc: sisbell removed
Description: modified (diff)

comment:21 Changed 11 months ago by pili

Keywords: TorBrowserTeam201912 added; TorBrowserTeam201911 removed

Moving tickets to December

comment:22 Changed 10 months ago by eighthave

I talked with @boklm about this at the Reproducible Builds summit. I've now forgotten what the original reproducible issue was that this ticket is dealing with but I do remember that I think the best way forward is to work with https://android-rebuilds.beuc.net/ to rebuild the things TBB needs with reproducibility fixes included. This would mostly mean backporting them, since I think the current versions from Google have no known reproducibility issues.

comment:23 Changed 10 months ago by boklm

The original issue is:
https://issuetracker.google.com/issues/110237303?pli=1

Apparently it's fixed in more recent versions of Android Gradle Plugin:

I added sorting the input files in AGP in change list with Change-Id: I371304c8a6d244f8bb7833609f464eba000e608f - it should be available in android gradle plugin version 3.5 Canary 7
We cherry picked it into 3.4 and it should be available in the next 3.4 canary. We'll see what we can do about 3.3.

comment:24 Changed 10 months ago by sysrqb

Keywords: TorBrowserTeam202001 added; TorBrowserTeam201912 removed

comment:25 Changed 9 months ago by pili

Keywords: TorBrowserTeam202002 added; TorBrowserTeam202001 removed

Moving tickets to February

comment:26 Changed 9 months ago by sysrqb

Keywords: ReleaseTrainMigration added

Release Train Migration.

comment:27 Changed 8 months ago by sisbell

Do we still want to do this? It is a high level of effort and with fenix coming up it may be worth closing this issue.

comment:28 Changed 8 months ago by pili

Keywords: TorBrowserTeam202003 added; TorBrowserTeam202002 removed

We are no longer in February, moving tickets

comment:29 Changed 7 months ago by pili

Sponsor: Sponsor58-can

comment:30 Changed 7 months ago by pili

Keywords: TorBrowserTeam202004 added; TorBrowserTeam202003 removed

We are no longer in March

comment:31 Changed 7 months ago by pili

Reviewer: sysrqb

comment:32 Changed 7 months ago by pili

Parent ID: #33659

comment:33 Changed 5 months ago by gk

Keywords: TorBrowserTeam202006 added; TorBrowserTeam202004 removed

comment:34 Changed 5 months ago by gk

Keywords: gitlab-tb-tor-browser-build added

Add magic gitlab keyword.

Note: See TracTickets for help on using tickets.