Add option to build without using containers

Trac:
Child Ticket(s): #30089 (moved)

added TorBrowserTeam201904R component::applications/tor browser owner::tbb-team priority::medium resolution::fixed severity::normal status::closed tbb-8.5 tbb-rbm type::task labels

Replying to boklm:

By default the Tor Browser build is done inside containers, which we run using runc, which require root access.

In some cases, such as F-Droid builds (#27539 (moved)), this can be a problem. I think we should be able to add an option to do the builds without using containers.

How would that help the F-Droid case given that F-Droid is taking just the signature for an app and applying that one to the built they made themselves? I mean the non-container build would then need to match our build done in a containerized setup so that F-Droid would get a bit-for-bit identical output. Maybe I am too pessimistic here but I'd be surprised if that worked out-of-the-box...

Replying to gk:

Replying to boklm:

By default the Tor Browser build is done inside containers, which we run using runc, which require root access.

In some cases, such as F-Droid builds (#27539 (moved)), this can be a problem. I think we should be able to add an option to do the builds without using containers.

How would that help the F-Droid case given that F-Droid is taking just the signature for an app and applying that one to the built they made themselves? I mean the non-container build would then need to match our build done in a containerized setup so that F-Droid would get a bit-for-bit identical output. Maybe I am too pessimistic here but I'd be surprised if that worked out-of-the-box...

If the F-Droid build VM is using Debian Strech too, then I think the main differences between our build environment and the F-Droid build environment would be:

the username of the build user. In the rbm build we use an rbm user, but I think we can add an option to change it if that is an issue.
the list of packages installed. In our case we build each component with only the minimum set of dependencies required to build this component, while in the F-Droid case they would always have the dependencies required to build all of the components. I think in most cases the additional packages should not change the output of the build. Maybe in some cases it will requires some changes to avoid the effect of some additional package.

Unless I'm forgetting an other important difference, I think it should not be very difficult to get matching builds. Although I did not try it, so maybe I'm too optimistic.

I started working on a patch for that, and after comparing a build done without containers, and a normal build with containers, I get an almost matching apk. The only difference is:

Binary files 1/META-INF/ANDROIDQ.RSA and 2/META-INF/ANDROIDQ.RSA differ
diff -r 1/META-INF/ANDROIDQ.SF 2/META-INF/ANDROIDQ.SF
2,4c2,4
< SHA1-Digest-Manifest-Main-Attributes: nJdsdwNyiHipRN3sUz498qUG+L0=
< SHA1-Digest-Manifest: vF7lnm0ro+G4diQqON1e3X8+snc=
< Created-By: 1.8.0_171 (Oracle Corporation)
---
> SHA1-Digest-Manifest-Main-Attributes: E8IbsJM7v8R6d8dy6OqA+g4Q9EE=
> SHA1-Digest-Manifest: cBYG++7ArsIP93o3h2mGE0OM4WI=
> Created-By: 1.8.0_212 (Oracle Corporation)
diff -r 1/META-INF/MANIFEST.MF 2/META-INF/MANIFEST.MF
2c2
< Created-By: 1.8.0_171 (Oracle Corporation)
---
> Created-By: 1.8.0_212 (Oracle Corporation)

It seems to be because the tor-browser step is done in a buster container. The debug signature is also causing the build without containers to stall.

So I think if we find an other workaround for the debug signature that doesn't require using buster, we should be able to get matching build without and without containers.

Wow, you have achieved reproducible builds then! The oracle version number diff can be solved by using apksigner rather than jarsigner to make the APK Signatures. apksigner is what the Android SDK has been using by default for about 2 years now, and is reproducibly built in Debian stretch and buster:

https://tests.reproducible-builds.org/debian/rb-pkg/stretch/amd64/android-platform-tools-apksig.html

If you need the newer apksigner in Debian/stretch, I can backport it. As far as I remember, the changes are only to support APK v2 and v3 signatures better, and were not security fixes. But it might be worth double checking my memory.

Trac:
Cc: N/A to hans@guardianproject.info

Technically, you don't need to update to apksigner although there is nothing wrong with it.

You can override the default manifest Oracle. First create the a manifest.txt to use to override entries

Created-By: Tor Build

Then use the 'm' flag

jar -cvfm tor-browser.jar manifest.txt

The generated manifest will be

Manifest-Version: 1.0
Created-By: Tor Build

Replying to sisbell:

Technically, you don't need to update to apksigner although there is nothing wrong with it.

You can override the default manifest Oracle. First create the a manifest.txt to use to override entries

{{{ Created-By: Tor Build }}}

Then use the 'm' flag

{{{ jar -cvfm tor-browser.jar manifest.txt }}}

The generated manifest will be

{{{ Manifest-Version: 1.0 Created-By: Tor Build }}}

The main issue is not that some entries in the manifest are different, but that the signing is stalling when we do it with faketime on stretch, which is the reason we currently have to do it in a buster container.

If apksigner does not stall when used with faketime on stretch (or doesn't need faketime to produce reproducible output), then that would solve the issue.

While you are at it, you should also drop faketime for signing. The Android tools have been zeroing out the dates in the ZIP header for a while now, like 2+ years. If TBA doesn't use a version of the Android tools that does that, it is not hard to do it with a tool or with Python or something as a post-build, pre-signing process.

Using the patch from branch bug_29981_v3 I have been able to do a build without containers: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_29981_v3&id=ab3c5d96be66d09d4dc98be385e495244bef48af

The apk I got does not match the one built using container, but the only difference is:

--- tor-browser-8.5a10-android-armv7-multi-qa.apk
+++ ../../testbuild/tor-browser-8.5a10-android-armv7-multi-qa.apk
├── zipinfo {}
│ @@ -1502,13 +1502,13 @@
│  -rwxr-xr-x  2.0 unx     5596 b- defX 10-Jan-01 00:00 lib/armeabi-v7a/libplugin-container.so
│  -rw-r--r--  2.0 unx      382 b- defX 10-Jan-01 00:00 application.ini
│  -rw-r--r--  2.0 unx       32 b- defX 10-Jan-01 00:00 package-name.txt
│  -rw-r--r--  2.0 unx      795 b- defX 10-Jan-01 00:00 ua-update.json
│  -rw-r--r--  2.0 unx       48 b- defX 10-Jan-01 00:00 platform.ini
│  -rw-r--r--  2.0 unx       97 b- defX 10-Jan-01 00:00 removed-files
│  -rw-r--r--  2.0 unx 10641927 b- stor 10-Jan-01 00:00 assets/omni.ja
│ --rw-r--r--  3.0 unx   502590 b- defN 00-Jan-01 00:00 assets/distribution/extensions/{73a6fe31-595d-460b-a920-fcc0f8843232}.xpi
│  -rw-r--r--  3.0 unx  1765652 b- defN 00-Jan-01 00:00 assets/distribution/extensions/https-everywhere-eff@eff.org.xpi
│ +-rw-r--r--  3.0 unx   502590 b- defN 00-Jan-01 00:00 assets/distribution/extensions/{73a6fe31-595d-460b-a920-fcc0f8843232}.xpi
│  -rw----     2.0 fat   144465 b- defN 10-Jan-01 00:00 META-INF/ANDROIDQ.SF
│  -rw----     2.0 fat     1194 b- defN 10-Jan-01 00:00 META-INF/ANDROIDQ.RSA
│  -rw----     2.0 fat   144360 b- defN 10-Jan-01 00:00 META-INF/MANIFEST.MF
│  1512 files, 58292874 bytes uncompressed, 44899152 bytes compressed:  23.0%

I think the cause of this is this line in projects/tor-browser/build.android:

zip -r -X $apk $ext_dir

Which is not adding the extensions to the .apk in a sorted order.

Wow, great progress! That sort order issue is frustrating. My experience is that doing post-processing of APKs with zip or other tools usually leads to random repro issues like that. The Android tools have been getting better, but they still have repro quirks. It seems that the zip commands are used for adding and removing static assets. I can help move that to the pre-packaging phase of the Android build, if gradle is in use.

Trac:
Keywords: N/A deleted, tbb-8.5-must added

There are two patches for review in branch bug_29981_v5: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_29981_v5&id=069c880961e93fe2f5052ae6d428b2f9efdd1d4e

I have been able to get matching builds from this branch for the android-x86 and android-armv7 builds.

The packages that I installed on a stretch machine to be able to do the build are:

build-essential python bison automake libtool zip unzip autoconf2.13 yasm openjdk-8-jdk gettext-base autotools-dev automake autoconf libtool autopoint libssl-dev pkg-config zlib1g-dev libparallel-forkmanager-perl libfile-slurp-perl bzip2 xz-utils apksigner

Trac:
Keywords: TorBrowserTeam201904 deleted, TorBrowserTeam201904R added
Status: new to needs_review

While still testing the changes here come some preliminary comments:

commit e38b5d94a19caff488382d92968dbd8cd85b18eb

The change to the README says "Adding the no_containers target will disable the use of containers", but the first thing that happens is actually building a container (called "empty"). So, we might want to be a bit more verbose explaining what is happening here to avoid confusion.

"and install build dependencies for all the components that are built" We are listing build dependencies in the README file. I think we should list all the additional build dependencies needed for building without containers as well while we are talking about them. There should be no need for builders that want to build without containers to try figuring that out themselves.

commit 069c880961e93fe2f5052ae6d428b2f9efdd1d4e

"Add the extensions in sorted order" it's not only extensions it seems? So, please adjust the commit message accordingly.

I wonder why we need container/use_container and container/disable as two separate options. If use_container is false then cleary I build without containers, no? In other words: why can't we just use/amend the already existing use_container option to avoid creating another one that seems to do the same thing but just (slightly) differently?

Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201904R deleted, TorBrowserTeam201904 added

Alright, I started a build on two different machines, both on commit 069c880961e93fe2f5052ae6d428b2f9efdd1d4e (boklm/bug_29981_v5). One machine is building with containers and one without. While this setup builds reproducibly on each machine the *.apks on one machine differ from those on the second one.

The differing parts are essentially all the Firefox .so files in assets/ or lib/. Both machines built from the same tor-browser commit (c722d57604db58695140d95565a78433989fe9ca).

I attached two .so files showing the issue.

FWIW: I built on the bare stretch machine without installing bison, yasm, and openjdk-8-jdk. (However, openjdk-8-jre is installed it seems)

Trac:
liblgpllibs_1.so

Trac:
liblgpllibs_2.so

Trac:
pkg_list.txt

Add option to build without using containers

Child items 0

Activity