Android bundles based on ESR 68 are not built reproducibly anymore
I compared two builds on two different machines with what is very likely our toolchain to come (armv7) and it turns out the .apk files are nor built reproducibly anymore.
Activity
Trac:
Parent Ticket: #30324 (moved)
Okay, here are my findings so far: the sole differences (apart from the signature and different sha1 sums in the MANIFEST files) are in
resources.arsc. The good news here is that disassembling the whole .apk with something likeapktoolsshows not differences anymore apart from the signature and sha1 ones mentioned above which are due toresources.arscbeing different. Thus, the problem lies in how that file gets assembled.Dump the contents with
aapt dump resourcesand diffing the results gets us differences like:< resource 0x7f01000c org.torproject.torbrowser_nightly:anim/design_bottom_sheet_slide_in: t=0x03 d=0x0000025e (s=0x0008 r=0x00) < resource 0x7f01000d org.torproject.torbrowser_nightly:anim/design_bottom_sheet_slide_out: t=0x03 d=0x0000025f (s=0x0008 r=0x00) --- > resource 0x7f01000c org.torproject.torbrowser_nightly:anim/design_bottom_sheet_slide_in: t=0x03 d=0x0000025d (s=0x0008 r=0x00) > resource 0x7f01000d org.torproject.torbrowser_nightly:anim/design_bottom_sheet_slide_out: t=0x03 d=0x0000025e (s=0x0008 r=0x00)
This looks like this issue: https://issuetracker.google.com/issues/110237303?pli=1 (which annoyingly requires a google account just to see it)
Apparently it has been fixed with:
I added sorting the input files in AGP in change list with Change-Id: I371304c8a6d244f8bb7833609f464eba000e608f - it should be available in android gradle plugin version 3.5 Canary 7 We cherry picked it into 3.4 and it should be available in the next 3.4 canary. We'll see what we can do about 3.3.
-
Hm. Did they say when it got introduced? Is our reproducible builds setup broken that we did not detect that problem during the esr60 cycle? Or is that really a thing that got introduced with the new toolchain we use.
That said, sisbell, what are our options here? Can we just use a newer gradle plugin which has this fixed? Or do we need to build that one ourselves and use it then or...?
Replying to gk:
Hm. Did they say when it got introduced? Is our reproducible builds setup broken that we did not detect that problem during the esr60 cycle? Or is that really a thing that got introduced with the new toolchain we use.
That said, sisbell, what are our options here? Can we just use a newer gradle plugin which has this fixed? Or do we need to build that one ourselves and use it then or...?
In the google issuetracker. one person mentioned that they got around the issue by using aapt2 directly with the correct order in the linker. I think this is worth investigating since it could potentially be scripted without requiring a patch. My only concern is that Firefox has a customized build so it may not be extendible to this approach. It requires looking into.
https://developer.android.com/studio/command-line/aapt2
aapt2 link -o output.apk -I android_sdk/platforms/android_version/android.jar compiled/res/values_values.arsc.flat compiled/res/drawable_Image.flat --manifest /path/to/AndroidManifest.xml -vI've tried doing the linking for resources and then merging the apks but haven't had much luck when starting the app. I get various crashes. I'll investigate more.
08-31 02:02:43.923 31309 31309 W PackageManager: Failure retrieving xml 0x7f130014 in package org.torproject.torbrowser 08-31 02:02:43.923 31309 31309 W PackageManager: android.content.res.Resources$NotFoundException: Resource ID #0x7f130014 08-31 02:02:43.923 31309 31309 W PackageManager: at android.content.res.ResourcesImpl.getValue(ResourcesImpl.java:237)Is our reproducible builds setup broken that we did not detect that problem during the esr60 cycle? Or is that really a thing that got introduced with the new toolchain we use. https://hg.mozilla.org/mozilla-central/rev/787dce710dce
Replying to cypherpunks:
Is our reproducible builds setup broken that we did not detect that problem during the esr60 cycle? Or is that really a thing that got introduced with the new toolchain we use. https://hg.mozilla.org/mozilla-central/rev/787dce710dce
Thanks, I'll try android.enableAapt2=false and see if we get a different result for the resource file.
The enableAapt2=false option results in a proguard error. So the current firefox build doesn't work out of the box with aapt2 disabled.
0:45.03 Warning: there were 13 unresolved references to classes or interfaces. 0:45.03 You may need to add missing library jars or update their versions. 0:45.03 If your code works fine without the missing classes, you can suppress 0:45.03 the warnings with '-dontwarn' options. 0:45.03 (http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedclass) 0:45.04 Warning: Exception while processing task java.io.IOException: Please correct the above warnings first. 0:45.04 Thread(Tasks limiter_1): destruction 0:45.04 > Task :app:transformClassesAndResourcesWithProguardForWithoutGeckoBinariesDebug FAILED 0:45.04 FAILURE: Build failed with an exception. 0:45.04 * What went wrong: 0:45.04 Execution failed for task ':app:transformClassesAndResourcesWithProguardForWithoutGeckoBinariesDebug'.-
Replying to sisbell:
The enableAapt2=false option results in a proguard error. So the current firefox build doesn't work out of the box with aapt2 disabled.
I am not convinced yet we should go that route. It seems the move to
aapt2might have helped in "fixing" unexplainable crashes. We should not risk getting those back if possible. We should maybe have hit those by now (and maybe users have but did not report) but, still, I am wary of switching back to the older tool.Could we downgrade the gradle plugin "easily" to an earlier version that was still good?
I tried just adding the 3.0.1 plugin and it did not build
0:37.44 21:25:31.172 [DEBUG] [org.gradle.launcher.daemon.server.exec.ExecuteBuild] The daemon has finished executing the build. 0:37.44 21:25:31.228 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClient] Received result Failure[value=org.gradle.initialization.ReportedException: org.gradle.internal.exceptions.LocationAwareException: A problem occurred configuring project ':app'.] from daemon DaemonInfo{pid=4198, address=[5ae804d2-ef75-4ba7-9d8c-51a300cbcdd7 port:46335, addresses:[/0:0:0:0:0:0:0:1, /127.0.0.1]], state=Busy, lastBusy=1567286702135, context=DefaultDaemonContext[uid=62e70e30-6e9a-4c42-91a7-9887ca646824,javaHome=/usr/lib/jvm/java-8-openjdk-amd64,daemonRegistryDir=/var/tmp/dist/android-toolchain/gradle/daemon,pid=4198,idleTimeout=120000,daemonOpts=-Xmx4608M,-Dfile.encoding=utf-8,-Duser.country=US,-Duser.language=en,-Duser.variant]} (build should be done). 0:37.44 21:25:31.228 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: dispatching class org.gradle.launcher.daemon.protocol.Finished 0:37.82 Traceback (most recent call last): 0:37.82 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main 0:37.82 "__main__", fname, loader, pkg_name) 0:37.82 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code 0:37.82 exec code in run_globals 0:37.82 File "/var/tmp/build/firefox-2dbb7aefeaeb/python/mozbuild/mozbuild/action/file_generate.py", line 120, in <module> 0:37.82 sys.exit(main(sys.argv[1:])) 0:37.82 File "/var/tmp/build/firefox-2dbb7aefeaeb/python/mozbuild/mozbuild/action/file_generate.py", line 71, in main 0:37.82 ret = module.__dict__[method](output, *args.additional_arguments, **kwargs) 0:37.82 File "/var/tmp/build/firefox-2dbb7aefeaeb/mobile/android/gradle.py", line 46, in assemble_app 0:37.82 return android('assemble-app') 0:37.82 File "/var/tmp/build/firefox-2dbb7aefeaeb/mobile/android/gradle.py", line 38, in android 0:37.82 subprocess.check_call(cmd, env=env) 0:37.82 File "/usr/lib/python2.7/subprocess.py", line 186, in check_call 0:37.82 raise CalledProcessError(retcode, cmd) 0:37.82 subprocess.CalledProcessError: Command '['/var/tmp/build/firefox-2dbb7aefeaeb/obj-arm-linux-androideabi/_virtualenvs/init/bin/python', u'/var/tmp/build/firefox-2dbb7aefeaeb/mach', 'android', 'assemble-app']' returned non-zero exit status 1 0:37.84 backend.mk:64: recipe for target '.deps/android_apks.stub' failed 0:37.84 make[4]: *** [.deps/android_apks.stub] Error 1So the 3.0.1 fails due to needing to accept license agreements for SDK. Notice that it is requiring build tools 26.0.2. So the downgrading the plugin also requires downgrading the toolchain as well.
0:37.28 21:25:31.055 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Build operation 'Configure build' completed 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception. 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong: 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] A problem occurred configuring project ':app'. 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > You have not accepted the license agreements of the following SDK components: 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] [Android SDK Build-Tools 26.0.2]. 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Before building your project, you need to accept the license agreements and complete the installation of the missing components using the Android Studio SDK Manager. 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Alternatively, to learn how to transfer the license agreements from one workstation to another, go to http://d.android.com/r/studio-ui/export-licenses.html 0:37.37 21:25:31.062 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]Looks like Signal created an apkdiff script that ignores resources.arsc
https://github.com/signalapp/Signal-Android/blob/master/apkdiff/apkdiff.py
Briar went the route of a deterministic file system
https://code.briarproject.org/briar/briar-reproducer/commit/22d04ff8bba956ec9647fd583ec655df691e15e5
Are either of these approaches workable for us?
A third route is to build and patch the gradle plugin ourselves but I have been unable to find the elusive change-id that google mentions.
The latest version that worked was using com.android.tools.build:gradle:3.0.1, buildToolsVersion '26.0.2' and aaptOptions { cruncherEnabled = false } So, did you test with just adding
aaptOptions { cruncherEnabled = false }? Any change in diff? https://bugzilla.mozilla.org/show_bug.cgi?id=1266156-
Replying to sisbell:
Looks like Signal created an apkdiff script that ignores resources.arsc
https://github.com/signalapp/Signal-Android/blob/master/apkdiff/apkdiff.py
Briar went the route of a deterministic file system
https://code.briarproject.org/briar/briar-reproducer/commit/22d04ff8bba956ec9647fd583ec655df691e15e5
Are either of these approaches workable for us?
Sounds to me like wallpapering over the underlying problem by taking extra steps during the verification process to work around the still existing differences, no? If so, I am not a big fan of those approaches.
A third route is to build and patch the gradle plugin ourselves but I have been unable to find the elusive change-id that google mentions.
Yes, that's the preferred way I think.
-
Replying to sisbell:
Looks like Signal created an apkdiff script that ignores resources.arsc
https://github.com/signalapp/Signal-Android/blob/master/apkdiff/apkdiff.py
This one doesn't seem to "fix" the problem, but only ignore it.
Briar went the route of a deterministic file system
https://code.briarproject.org/briar/briar-reproducer/commit/22d04ff8bba956ec9647fd583ec655df691e15e5
This one could maybe work as a temporary workaround. However this requires fuse to be installed on the build machine, and I'm not sure if different distributions have different fuse versions/setup that could make things more complicate.
A third route is to build and patch the gradle plugin ourselves but I have been unable to find the elusive change-id that google mentions.
This sounds like the best way to fix it. However I have not even been able to find their source code repository so far. Do you know if they have one?