Opened 7 weeks ago

Closed 6 days ago

Last modified 5 days ago

#31564 closed defect (fixed)

Android bundles based on ESR 68 are not built reproducibly anymore

Reported by: gk Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-9.0-must-alpha, GeorgKoppen201909, TorBrowserTeam201910R
Cc: sisbell, boklm Actual Points:
Parent ID: #30324 Points: 10
Reviewer: Sponsor:

Description

I compared two builds on two different machines with what is very likely our toolchain to come (armv7) and it turns out the .apk files are nor built reproducibly anymore.

Child Tickets

Change History (62)

comment:1 Changed 7 weeks ago by gk

Okay, here are my findings so far: the sole differences (apart from the signature and different sha1 sums in the MANIFEST files) are in resources.arsc. The good news here is that disassembling the whole .apk with something like apktools shows not differences anymore apart from the signature and sha1 ones mentioned above which are due to resources.arsc being different. Thus, the problem lies in how that file gets assembled.

Dump the contents with aapt dump resources and diffing the results gets us differences like:

<         resource 0x7f01000c org.torproject.torbrowser_nightly:anim/design_bottom_sheet_slide_in: t=0x03 d=0x0000025e (s=0x0008 r=0x00)
<         resource 0x7f01000d org.torproject.torbrowser_nightly:anim/design_bottom_sheet_slide_out: t=0x03 d=0x0000025f (s=0x0008 r=0x00)
---
>         resource 0x7f01000c org.torproject.torbrowser_nightly:anim/design_bottom_sheet_slide_in: t=0x03 d=0x0000025d (s=0x0008 r=0x00)
>         resource 0x7f01000d org.torproject.torbrowser_nightly:anim/design_bottom_sheet_slide_out: t=0x03 d=0x0000025e (s=0x0008 r=0x00)

comment:2 Changed 6 weeks ago by boklm

This looks like this issue:
https://issuetracker.google.com/issues/110237303?pli=1
(which annoyingly requires a google account just to see it)

Apparently it has been fixed with:

I added sorting the input files in AGP in change list with Change-Id: I371304c8a6d244f8bb7833609f464eba000e608f - it should be available in android gradle plugin version 3.5 Canary 7
We cherry picked it into 3.4 and it should be available in the next 3.4 canary. We'll see what we can do about 3.3.

comment:3 Changed 6 weeks ago by gk

Hm. Did they say when it got introduced? Is our reproducible builds setup broken that we did not detect that problem during the esr60 cycle? Or is that really a thing that got introduced with the new toolchain we use.

That said, sisbell, what are our options here? Can we just use a newer gradle plugin which has this fixed? Or do we need to build that one ourselves and use it then or...?

comment:4 Changed 6 weeks ago by boklm

Someone on the ticket is mentioning that:

The latest version that worked was using com.android.tools.build:gradle:3.0.1, buildToolsVersion '26.0.2' and aaptOptions { cruncherEnabled = false }

We were using 3.0.1 in esr60, so this is probably why we didn't get the issue previously.

comment:5 in reply to:  3 Changed 6 weeks ago by sisbell

Replying to gk:

Hm. Did they say when it got introduced? Is our reproducible builds setup broken that we did not detect that problem during the esr60 cycle? Or is that really a thing that got introduced with the new toolchain we use.

That said, sisbell, what are our options here? Can we just use a newer gradle plugin which has this fixed? Or do we need to build that one ourselves and use it then or...?

In the google issuetracker. one person mentioned that they got around the issue by using aapt2 directly with the correct order in the linker. I think this is worth investigating since it could potentially be scripted without requiring a patch. My only concern is that Firefox has a customized build so it may not be extendible to this approach. It requires looking into.

https://developer.android.com/studio/command-line/aapt2

 aapt2 link -o output.apk
 -I android_sdk/platforms/android_version/android.jar
    compiled/res/values_values.arsc.flat
    compiled/res/drawable_Image.flat --manifest /path/to/AndroidManifest.xml -v

comment:6 Changed 6 weeks ago by sisbell

I've tried doing the linking for resources and then merging the apks but haven't had much luck when starting the app. I get various crashes. I'll investigate more.

08-31 02:02:43.923 31309 31309 W PackageManager: Failure retrieving xml 0x7f130014 in package org.torproject.torbrowser
08-31 02:02:43.923 31309 31309 W PackageManager: android.content.res.Resources$NotFoundException: Resource ID #0x7f130014
08-31 02:02:43.923 31309 31309 W PackageManager: 	at android.content.res.ResourcesImpl.getValue(ResourcesImpl.java:237)

comment:7 Changed 6 weeks ago by cypherpunks

Is our reproducible builds setup broken that we did not detect that problem during the esr60 cycle? Or is that really a thing that got introduced with the new toolchain we use.

https://hg.mozilla.org/mozilla-central/rev/787dce710dce

comment:8 in reply to:  7 Changed 6 weeks ago by sisbell

Replying to cypherpunks:

Is our reproducible builds setup broken that we did not detect that problem during the esr60 cycle? Or is that really a thing that got introduced with the new toolchain we use.

https://hg.mozilla.org/mozilla-central/rev/787dce710dce

Thanks, I'll try android.enableAapt2=false and see if we get a different result for the resource file.

comment:9 Changed 6 weeks ago by sisbell

The enableAapt2=false option results in a proguard error. So the current firefox build doesn't work out of the box with aapt2 disabled.

 0:45.03 Warning: there were 13 unresolved references to classes or interfaces.
 0:45.03          You may need to add missing library jars or update their versions.
 0:45.03          If your code works fine without the missing classes, you can suppress
 0:45.03          the warnings with '-dontwarn' options.
 0:45.03          (http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedclass)
 0:45.04 Warning: Exception while processing task java.io.IOException: Please correct the above warnings first.
 0:45.04 Thread(Tasks limiter_1): destruction
 0:45.04 > Task :app:transformClassesAndResourcesWithProguardForWithoutGeckoBinariesDebug FAILED
 0:45.04 FAILURE: Build failed with an exception.
 0:45.04 * What went wrong:
 0:45.04 Execution failed for task ':app:transformClassesAndResourcesWithProguardForWithoutGeckoBinariesDebug'.

comment:10 in reply to:  9 Changed 6 weeks ago by gk

Replying to sisbell:

The enableAapt2=false option results in a proguard error. So the current firefox build doesn't work out of the box with aapt2 disabled.

I am not convinced yet we should go that route. It seems the move to aapt2 might have helped in "fixing" unexplainable crashes. We should not risk getting those back if possible. We should maybe have hit those by now (and maybe users have but did not report) but, still, I am wary of switching back to the older tool.

Could we downgrade the gradle plugin "easily" to an earlier version that was still good?

comment:11 Changed 6 weeks ago by sisbell

I tried just adding the 3.0.1 plugin and it did not build

 0:37.44 21:25:31.172 [DEBUG] [org.gradle.launcher.daemon.server.exec.ExecuteBuild] The daemon has finished executing the build.
 0:37.44 21:25:31.228 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClient] Received result Failure[value=org.gradle.initialization.ReportedException: org.gradle.internal.exceptions.LocationAwareException: A problem occurred configuring project ':app'.] from daemon DaemonInfo{pid=4198, address=[5ae804d2-ef75-4ba7-9d8c-51a300cbcdd7 port:46335, addresses:[/0:0:0:0:0:0:0:1, /127.0.0.1]], state=Busy, lastBusy=1567286702135, context=DefaultDaemonContext[uid=62e70e30-6e9a-4c42-91a7-9887ca646824,javaHome=/usr/lib/jvm/java-8-openjdk-amd64,daemonRegistryDir=/var/tmp/dist/android-toolchain/gradle/daemon,pid=4198,idleTimeout=120000,daemonOpts=-Xmx4608M,-Dfile.encoding=utf-8,-Duser.country=US,-Duser.language=en,-Duser.variant]} (build should be done).
 0:37.44 21:25:31.228 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: dispatching class org.gradle.launcher.daemon.protocol.Finished
 0:37.82 Traceback (most recent call last):
 0:37.82   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
 0:37.82     "__main__", fname, loader, pkg_name)
 0:37.82   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
 0:37.82     exec code in run_globals
 0:37.82   File "/var/tmp/build/firefox-2dbb7aefeaeb/python/mozbuild/mozbuild/action/file_generate.py", line 120, in <module>
 0:37.82     sys.exit(main(sys.argv[1:]))
 0:37.82   File "/var/tmp/build/firefox-2dbb7aefeaeb/python/mozbuild/mozbuild/action/file_generate.py", line 71, in main
 0:37.82     ret = module.__dict__[method](output, *args.additional_arguments, **kwargs)
 0:37.82   File "/var/tmp/build/firefox-2dbb7aefeaeb/mobile/android/gradle.py", line 46, in assemble_app
 0:37.82     return android('assemble-app')
 0:37.82   File "/var/tmp/build/firefox-2dbb7aefeaeb/mobile/android/gradle.py", line 38, in android
 0:37.82     subprocess.check_call(cmd, env=env)
 0:37.82   File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
 0:37.82     raise CalledProcessError(retcode, cmd)
 0:37.82 subprocess.CalledProcessError: Command '['/var/tmp/build/firefox-2dbb7aefeaeb/obj-arm-linux-androideabi/_virtualenvs/init/bin/python', u'/var/tmp/build/firefox-2dbb7aefeaeb/mach', 'android', 'assemble-app']' returned non-zero exit status 1
 0:37.84 backend.mk:64: recipe for target '.deps/android_apks.stub' failed
 0:37.84 make[4]: *** [.deps/android_apks.stub] Error 1

comment:12 Changed 6 weeks ago by sisbell

So the 3.0.1 fails due to needing to accept license agreements for SDK. Notice that it is requiring build tools 26.0.2. So the downgrading the plugin also requires downgrading the toolchain as well.

 0:37.28 21:25:31.055 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Build operation 'Configure build' completed
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] A problem occurred configuring project ':app'.
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > You have not accepted the license agreements of the following SDK components:
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   [Android SDK Build-Tools 26.0.2].
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   Before building your project, you need to accept the license agreements and complete the installation of the missing components using the Android Studio SDK Manager.
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   Alternatively, to learn how to transfer the license agreements from one workstation to another, go to http://d.android.com/r/studio-ui/export-licenses.html
 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

comment:13 Changed 6 weeks ago by sisbell

Looks like Signal created an apkdiff script that ignores resources.arsc

https://github.com/signalapp/Signal-Android/blob/master/apkdiff/apkdiff.py

Briar went the route of a deterministic file system

https://code.briarproject.org/briar/briar-reproducer/commit/22d04ff8bba956ec9647fd583ec655df691e15e5

Are either of these approaches workable for us?

A third route is to build and patch the gradle plugin ourselves but I have been unable to find the elusive change-id that google mentions.

comment:14 Changed 6 weeks ago by cypherpunks

The latest version that worked was using com.android.tools.build:gradle:3.0.1, buildToolsVersion '26.0.2' and aaptOptions { cruncherEnabled = false }

So, did you test with just adding aaptOptions { cruncherEnabled = false }? Any change in diff?
https://bugzilla.mozilla.org/show_bug.cgi?id=1266156

Last edited 6 weeks ago by cypherpunks (previous) (diff)

comment:16 in reply to:  13 Changed 6 weeks ago by gk

Replying to sisbell:

Looks like Signal created an apkdiff script that ignores resources.arsc

https://github.com/signalapp/Signal-Android/blob/master/apkdiff/apkdiff.py

Briar went the route of a deterministic file system

https://code.briarproject.org/briar/briar-reproducer/commit/22d04ff8bba956ec9647fd583ec655df691e15e5

Are either of these approaches workable for us?

Sounds to me like wallpapering over the underlying problem by taking extra steps during the verification process to work around the still existing differences, no? If so, I am not a big fan of those approaches.

A third route is to build and patch the gradle plugin ourselves but I have been unable to find the elusive change-id that google mentions.

Yes, that's the preferred way I think.

comment:17 in reply to:  13 Changed 6 weeks ago by boklm

Replying to sisbell:

Looks like Signal created an apkdiff script that ignores resources.arsc

https://github.com/signalapp/Signal-Android/blob/master/apkdiff/apkdiff.py

This one doesn't seem to "fix" the problem, but only ignore it.

Briar went the route of a deterministic file system

https://code.briarproject.org/briar/briar-reproducer/commit/22d04ff8bba956ec9647fd583ec655df691e15e5

This one could maybe work as a temporary workaround. However this requires fuse to be installed on the build machine, and I'm not sure if different distributions have different fuse versions/setup that could make things more complicate.

A third route is to build and patch the gradle plugin ourselves but I have been unable to find the elusive change-id that google mentions.

This sounds like the best way to fix it. However I have not even been able to find their source code repository so far. Do you know if they have one?

comment:18 Changed 6 weeks ago by gk

Keywords: tbb-9.0-must-alpha added; tbb-9.0-must-nightly removed

Move must-nightly items to must-alpha ones.

comment:19 Changed 6 weeks ago by gk

Keywords: TorBrowserTeam201909 added; TorBrowserTeam201908 removed

Moving must-alpha tickets to September.

comment:20 Changed 6 weeks ago by gk

Keywords: GeorgKoppen201909 added; GeorgKoppen201908 removed

First ticket move of mine.

comment:21 Changed 6 weeks ago by pili

Points: 5

comment:22 Changed 5 weeks ago by sisbell

I downloaded the android gradle plugin repo. Its 8 GB. Most of the documentation and README files for the project are incorrect and/or not up-to-date. With the exception of the instructions for downloading the repo, not a single command worked as given, without some sort of modification.

The blocker for me came with being unable to find or download the google vendor code for the build. Its similar to the following reported issue:

https://stackoverflow.com/questions/50946201/aosp-does-not-have-tools-vendor-google3-project

It does not appear that the full build is open-source.

After some more research and investigation, I do have an alternative solution. We can use apktool. This will decompile the resources and then we can recompile with aapt2, which does not have the bug, as we would be bypassing the gradle plugin during a re-packaging phase. I've gone through apktool code and their is no issue on ordering of resources.

https://github.com/iBotPeaches/Apktool

I'll bring this up for discussion but the approach would be to create a project apktool project and to build the tool. Then use the tool to decompile and recompile resources in consistent order prior to packaging everything in the firefox project build (or in tor-browser).

comment:23 in reply to:  22 Changed 5 weeks ago by boklm

Replying to sisbell:

I downloaded the android gradle plugin repo.

Out of curiosity, do you have a link to the repo?

comment:24 Changed 5 weeks ago by cypherpunks

Doesn't look better than Briar's approach.

comment:25 Changed 5 weeks ago by sisbell

Here's the link to how to download the repo

https://android.googlesource.com/platform/tools/base/+/studio-master-dev/source.md

I checkout out branch 3.4.0.

comment:26 Changed 4 weeks ago by sisbell

I did the apktool decompile/decompile approach. The apktool that can be added in config/arch_deps is an old version of apktool with a bug

I: Using Apktool 2.2.1-dirty on tor-browser-unsigned-unaligned.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /home/rbm/.local/share/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values */* XMLs...
Exception in thread "main" java.lang.NullPointerException
	at brut.androlib.res.data.value.ResStyleValue.serializeToResValuesXml(ResStyleValue.java:58)
	at brut.androlib.res.AndrolibResources.generateValuesFile(AndrolibResources.java:516)
	at brut.androlib.res.AndrolibResources.decode(AndrolibResources.java:267)
	at brut.androlib.Androlib.decodeResourcesFull(Androlib.java:132)
	at brut.androlib.ApkDecoder.decode(ApkDecoder.java:108)
	at brut.apktool.Main.cmdDecode(Main.java:166)
	at brut.apktool.Main.main(Main.java:81)

https://github.com/iBotPeaches/Apktool/issues/1399

So I'm using apktool 2.4.0, downloading it as part of the build.

When re-zipping the file, I was getting some zip entry extra field flags that would change each build. I'm not exactly sure what the extra field info was as its platform specific and not standard fields like timestamp.. I removed these using the -X option. After that, multiple builds of the apk result in the same checksum. I'll need someone to verify that the checksum matches across different build machine OSes.

There is some room to cleanup the build further if it is verified to fix this bug.

The code is at https://github.com/sisbell/tor-browser-build/commit/694de9f7431b022cf974aa5bd8b6150e59f0bcbf

comment:27 Changed 4 weeks ago by sisbell

Status: newneeds_review

comment:28 Changed 4 weeks ago by boklm

Keywords: TorBrowserTeam201909R added; TorBrowserTeam201909 removed

comment:29 Changed 4 weeks ago by gk

Hm. So, I am not overly happy using some random .jar file from the Internet. Do you know whether version 2.3.4 would work here as well? We could then think to upgrade to Debian buster which comes with it (and which is a thing we want to do anyway, see: #31130). If it's just a matter of replacing "stretch" with "buster" and then using apktool 2.3.4 (and nothing else breaks) then that would be the better way to go I think. Could you test whether that is a workable solution?

Last edited 4 weeks ago by gk (previous) (diff)

comment:30 Changed 4 weeks ago by gk

Status: needs_reviewneeds_information

comment:31 in reply to:  26 Changed 4 weeks ago by boklm

Replying to sisbell:

When re-zipping the file, I was getting some zip entry extra field flags that would change each build. I'm not exactly sure what the extra field info was as its platform specific and not standard fields like timestamp.. I removed these using the -X option. After that, multiple builds of the apk result in the same checksum. I'll need someone to verify that the checksum matches across different build machine OSes.

That's probably not enough to make it reproductible on multiple machines as the order in which files are included in the zip might be different. You can fix it with:

diff --git a/projects/tor-browser/build.android b/projects/tor-browser/build.android
index f93e46d..f621a83 100644
--- a/projects/tor-browser/build.android
+++ b/projects/tor-browser/build.android
@@ -44,8 +44,10 @@ java -jar $apktool b -o $resfix decompiled
 # Fix timestamps and remove extra field info from zip entries
 unzip $resfix -d tmp
 cd tmp
-find . -exec [% c("var/touch") %] {} \;
-zip -rX $resfix .
+[% c('zip', {
+  zip_src => [ '.' ],
+  zip_args => '$resfix',
+  }) %]
 
 # Sign a QA build. This apk is not a debug version and doesn't contain a debug flag in the manifest
 java -jar /usr/share/apksigner/apksigner.jar sign --verbose --min-sdk-version [% c("var/android_min_api") %] --ks $rootdir/android-qa.keystore --out $qa_apk --in $resfix --ks-key-alias androidqakey --key-pass pass:android --ks-pass pass:android

Also do we really need a separate $resfix file, or could we just overwrite the $apk?

comment:32 in reply to:  29 ; Changed 3 weeks ago by sisbell

I wouldn't exactly call it a random file, as its maintained by the apktool project but if we are updating to buster, then we can remove this step.

I did move forward with the buster approach. The issue I've run into is that openjdk-8 is not supported out of the box for buster. I'll need to download openjdk-8 and unpack it. I'm thinking of doing this within the android-toolchain project so we only have to download once.

Replying to gk:

Hm. So, I am not overly happy using some random .jar file from the Internet. Do you know whether version 2.3.4 would work here as well? We could then think to upgrade to Debian buster which comes with it (and which is a thing we want to do anyway, see: #31130). If it's just a matter of replacing "stretch" with "buster" and then using apktool 2.3.4 (and nothing else breaks) then that would be the better way to go I think. Could you test whether that is a workable solution?

comment:33 in reply to:  32 Changed 3 weeks ago by gk

Replying to sisbell:

I wouldn't exactly call it a random file, as its maintained by the apktool project but if we are updating to buster, then we can remove this step.

I did move forward with the buster approach. The issue I've run into is that openjdk-8 is not supported out of the box for buster.

Why do we need openjdk-8 in particular? Can't we use any newer version that is available, like openjdk-11?

comment:34 Changed 3 weeks ago by gk

Keywords: TorBrowserTeam201909 added; TorBrowserTeam201909R removed
Status: needs_informationneeds_revision

comment:35 Changed 3 weeks ago by eighthave

The Android tools e.g. Gradle Android Plugin need opendk-8, they don't run with a newer JDK unless you are using quite new versions, like gradle 5.x. and a Gradle Android PLugin version that matches.

I really highly recommend never building anything from the Android source tree from scratch. It is an utter nightmare of tangled dependencies, out of date documentation, custom build systems, and "prebuilt" binaries. We the Debian Android Tools Team have put a ton of work into real free software builds, and sadly, its far from the whole suite.

Last edited 3 weeks ago by eighthave (previous) (diff)

comment:36 Changed 3 weeks ago by eighthave

Also, we've used the briar sorted-fs hack in a number of apps in fdroid, it works well.

comment:37 in reply to:  35 Changed 3 weeks ago by gk

Replying to eighthave:

The Android tools e.g. Gradle Android Plugin need opendk-8, they don't run with a newer JDK unless you are using quite new versions, like gradle 5.x. and a Gradle Android PLugin version that matches.

That's very unfortunate. :(

I really highly recommend never building anything from the Android source tree from scratch.

Yeah, I want to have this, too. So far the main problem is lack of time to set this properly up, in particular while we are transitioning to the new esr.

comment:38 Changed 3 weeks ago by sisbell

I was able to successfully use openjdk-11 with gradle 4.10.2 for tor-onion-proxy-library and tor-android-service. I do encounter an NPE with this configuration while building the firefox project. I'm tracking that one down.

 0:11.99 make[4]: Leaving directory '/var/tmp/build/firefox-80f3dafdd420/obj-x86_64-linux-android/toolkit/components/telemetry'
 0:19.08 FAILURE: Build failed with an exception.
 0:19.08 * What went wrong:
 0:19.08 A problem occurred configuring project ':app'.
 0:19.08 > java.lang.NullPointerException (no error message)
 0:19.08 * Try:
 0:19.08 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

comment:39 Changed 3 weeks ago by sisbell

I managed to solve the NPE problem using the gradle property mentioned here:

https://issuetracker.google.com/u/0/issues/72872257

But it fails later on in the build. It looks like it's hit or miss depending on the gradle task/plugin. I'm going to close this line of testing and move on to another approach.

comment:40 Changed 3 weeks ago by eighthave

Building gradle from source might not be so bad, if you don't mind using gradle binaries to build it. Maybe the Debian patch for 4.4.1 will work for 4.10.2. Gradle Android Plugin newer than 2.2.2 requires Java and Kotlin, so its a beast. Kotlin is close to being packaged in Debian, but not there yet.

I should also say, if you decide you want to build a piece from source, please check in with the Debian Android Tools Team, since we've been working on that for years. We can give you a quick assessment of how hard it would be. For example, building Gradle Android Plugin v2.2.2 from source is not bad, anything newer is much harder, and what makes it hard changes every other minor release (new libs, different build tools, etc). You can reach us on #debian-mobile on OFTC and android-tools-devel@… (no subscrition necessary to post)

comment:41 Changed 3 weeks ago by eighthave

Also, openjdk-8 from sid/unstable is meant to run on buster, since it is there so we can bootstrap Kotlin: https://packages.debian.org/source/sid/openjdk-8 That might not be there forever, but at least a year.

You could probably build gradle 4.10.2 using Debian/buster's gradle 4.4.1, if you want to avoid gradle binaries.

comment:42 Changed 2 weeks ago by sisbell

So I tried a few different approaches and what seems to be working is

Adding the following to projects/debootstrap-image/config

echo 'deb http://ftp.us.debian.org/debian sid main' >> ./base-image/etc/apt/sources.list

This allows us to install openjdk-8-jdk for buster.

I still need to verify apktools with a complete build, which I'm doing now.

comment:43 in reply to:  42 Changed 2 weeks ago by boklm

Replying to sisbell:

So I tried a few different approaches and what seems to be working is

Adding the following to projects/debootstrap-image/config

echo 'deb http://ftp.us.debian.org/debian sid main' >> ./base-image/etc/apt/sources.list

As it is going to be removed at some point, we should probably use https://snapshot.debian.org/ to install it.

comment:44 Changed 2 weeks ago by sisbell

So the buster/openjdk-8 part of the build works perfectly now. But when I get to tor-browser project, I get a failure with apktool 2.4 (the one packaged for buster).

W: aapt: brut.common.BrutException: brut.common.BrutException: Could not extract resource: /prebuilt/linux/aapt_64 (defaulting to $PATH binary)
W: res/drawable-v21/$avd_hide_password__0.xml: Invalid file name: must contain only [a-z0-9_.]
W: res/drawable-v21/$avd_hide_password__1.xml: Invalid file name: must contain only [a-z0-9_.]

So the question becomes why does the 2.4 apktool downloaded from repo work, but the one packaged in buster doesn't?

The downloaded one which works:

Apktool v2.4.0 - a tool for reengineering Android apk files
with smali v2.2.6 and baksmali v2.2.6

The one included in buster (which is busted)

Apktool v2.4.0-dirty - a tool for reengineering Android apk files
with smali v2.2.7-debian and baksmali v2.2.7-debian

However when I check /usr/share/java directory, I see smali version 2.2.3. This doesn't match 2.2.7 version the tool says its using. 2.2.3. is the same version of smali included in stretch. So the smali dependencies are the same in stretch/buster

-rw-r--r-- 1 root root 127604 Jul 22 00:53 baksmali-2.2.3.jar
-rw-r--r-- 1 root root 292694 Jul 22 00:53 smali-2.2.3.jar
-rw-r--r-- 1 root root 26148 Jul 22 00:53 smali-util-2.2.3.jar
-rw-r--r-- 1 root root 1037269 Jul 22 00:53 dexlib2-2.2.3.jar

The resource causing the problem is a resource in a google material-ui library that firefox doesn't use. So I considered shrinking the resources during the build, but due to the implementation of Firefox on Android, which uses a dynamic lookup of resource ids, the shrink resource option is not enabled.
https://bugzilla.mozilla.org/show_bug.cgi?id=1229269

So this effectively blocks the use of apktool packaged with buster.

comment:45 Changed 2 weeks ago by gk

Okay, I guess we are back at the step where we take the apktool from the internet? Do we have something else we could try? If not, then let's stick to that plan (+ see boklm's feedback in comment:31).

comment:46 in reply to:  45 Changed 2 weeks ago by sisbell

Replying to gk:

Okay, I guess we are back at the step where we take the apktool from the internet? Do we have something else we could try? If not, then let's stick to that plan (+ see boklm's feedback in comment:31).

The original path I took was to use aapt/aapt2 directly, as the last step of the firefox build. I had some missing resources that crashed the apk, so rather than track those down, I thought decompiling/recompiling the apk would prove simple, since the apk has everything it needs already. I can certainly track these resources down with the direct aapt approach, its tedious but should work.

On the plus side, we at least have a verified solution for upgrading Android builds to buster so all the work is not lost.

comment:47 Changed 2 weeks ago by eighthave

For apktool, you could probably just include the testing package directly in buster
if you're not already using buster-backports.

comment:48 Changed 2 weeks ago by eighthave

If you have a reproducible test case for this BrutException failure, please file a bug report (e.g. send an email to submit@bugs.debian.org with Package: apktool as the first line in the body).

The version number in the smali lib filenames is only cosmetic, I guess the buildsystem failed there.

comment:49 Changed 13 days ago by pili

Keywords: TorBrowserTeam201910 added

comment:50 Changed 13 days ago by pili

Keywords: TorBrowserTeam201909 removed

comment:51 in reply to:  44 ; Changed 12 days ago by boklm

Replying to sisbell:

So the buster/openjdk-8 part of the build works perfectly now. But when I get to tor-browser project, I get a failure with apktool 2.4 (the one packaged for buster).

W: aapt: brut.common.BrutException: brut.common.BrutException: Could not extract resource: /prebuilt/linux/aapt_64 (defaulting to $PATH binary)
W: res/drawable-v21/$avd_hide_password__0.xml: Invalid file name: must contain only [a-z0-9_.]
W: res/drawable-v21/$avd_hide_password__1.xml: Invalid file name: must contain only [a-z0-9_.]

So the question becomes why does the 2.4 apktool downloaded from repo work, but the one packaged in buster doesn't?

Could you share the changes you used to test this? From the logs it looks like you are using apktool 2.4.0, but buster is supposed to only have 2.3.4, so I'm wondering if you used a mix of packages from buster and sid or testing, which might be the cause of the error.

comment:52 in reply to:  51 ; Changed 12 days ago by sisbell

Yes I just added the sid repo to the sources.list, so it also includes the stable and unstable repos. Should I just try this with only the sid repo?

Replying to boklm:

Replying to sisbell:

So the buster/openjdk-8 part of the build works perfectly now. But when I get to tor-browser project, I get a failure with apktool 2.4 (the one packaged for buster).

W: aapt: brut.common.BrutException: brut.common.BrutException: Could not extract resource: /prebuilt/linux/aapt_64 (defaulting to $PATH binary)
W: res/drawable-v21/$avd_hide_password__0.xml: Invalid file name: must contain only [a-z0-9_.]
W: res/drawable-v21/$avd_hide_password__1.xml: Invalid file name: must contain only [a-z0-9_.]

So the question becomes why does the 2.4 apktool downloaded from repo work, but the one packaged in buster doesn't?

Could you share the changes you used to test this? From the logs it looks like you are using apktool 2.4.0, but buster is supposed to only have 2.3.4, so I'm wondering if you used a mix of packages from buster and sid or testing, which might be the cause of the error.

comment:53 in reply to:  52 Changed 12 days ago by boklm

Replying to sisbell:

Yes I just added the sid repo to the sources.list, so it also includes the stable and unstable repos. Should I just try this with only the sid repo?

No, it should have both. But you should configure pinning so that the unstable repository has lower priority, and only the openjdk-8 package is installed from unstable.

This page has some examples:
https://serverfault.com/questions/22414/how-can-i-run-debian-stable-but-install-some-packages-from-testing

But as said during the meeting, we can start with the solution using the downloaded apktool_2.4.0.jar if that's faster.

comment:54 Changed 9 days ago by boklm

Keywords: TorBrowserTeam201910R added; TorBrowserTeam201910 removed
Status: needs_revisionneeds_review

In branch bug_31564_v2 from my git repository I took the patch from comment 26, with the changes I suggested in comment 31:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_31564_v2&id=57732f7eee36ae7a3951bb937cf3d591b887cadb

I tested this patch on top of 9.0a7, and built all the Android bundles on two machines, and got matching builds. So it seems to be fixing the issue. However I didn't check that the bundles are still running correctly.

comment:55 in reply to:  54 Changed 9 days ago by boklm

However I didn't check that the bundles are still running correctly.

I uploaded them there:
https://people.torproject.org/~boklm/builds/bug_31564/

comment:56 in reply to:  52 Changed 7 days ago by boklm

Replying to sisbell:

Yes I just added the sid repo to the sources.list, so it also includes the stable and unstable repos. Should I just try this with only the sid repo?

By the way with "share the changes" I was thinking about sharing a patch, as a vague description of the changes is not very useful if I want to have a quick look at the issue.

comment:57 Changed 6 days ago by gk

Resolution: fixed
Status: needs_reviewclosed

Looks good to me. I gave the aarch64 .apk a cursory test on one of my devices and it is still running. Merged to master (commit 708ce94dba87d70033d61c930c542cc924dee9ad).

Last edited 6 days ago by gk (previous) (diff)

comment:58 Changed 6 days ago by gk

I created #31992 for a better workaround.

comment:59 Changed 6 days ago by eighthave

If you just need a single package from unstable, I think it would be easiest to wget it, then verify with sha256, e.g. the info listed on https://packages.debian.org/sid/all/apktool/download:

$ wget https://deb.debian.org/debian/pool/main/a/apktool/apktool_2.4.0-1_all.deb
$ echo '15a2b72d1892cfd5e4a3e878d6286d6f3f3e4ff6dcf9819d726db10626bd8c01  apktool_2.4.0-1_all.deb' > apktool_2.4.0-1_all.deb.sha256
$ sha256sum -c apktool_2.4.0-1_all.deb.sha256
$ dpkg -i apktool_2.4.0-1_all.deb

Otherwise, I'd use "Apt Pinning". That's a bit tricky to setup, ping me if you want me to write that, I've done it a lot.

Last edited 6 days ago by eighthave (previous) (diff)

comment:60 Changed 6 days ago by eighthave

I had to write the apt pin for something else, so here it is:

/etc/apt/preferences.d/debian-unstable.pref:

Package: *
Pin: release a=unstable
Pin-Priority: 100

Package: apktool
Pin: release a=unstable
Pin-Priority: 500

Then in my apt sources.list, I have:

deb https://deb.debian.org/debian unstable main

comment:61 in reply to:  48 Changed 5 days ago by gk

Replying to eighthave:

If you have a reproducible test case for this BrutException failure, please file a bug report (e.g. send an email to submit@bugs.debian.org with Package: apktool as the first line in the body).

The version number in the smali lib filenames is only cosmetic, I guess the buildsystem failed there.

Done. It's bug 942019.

comment:62 Changed 5 days ago by sisbell

Points: 510
Note: See TracTickets for help on using tickets.