Opened 5 months ago

Last modified 2 months ago

#33557 merge_ready task

Update Android Toolchain for Fenix

Reported by: sisbell Owned by: gk
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-mobile, tbb-rbm, GeorgKoppen202005, TorBrowserTeam202006R, gitlab-tb-tor-browser-build
Cc: tbb-team Actual Points:
Parent ID: #33184 Points:
Reviewer: sysrqb, ahf Sponsor: Sponsor58-must

Description

Fenix uses an updated Android SDK and NDK.

Child Tickets

Change History (35)

comment:1 Changed 5 months ago by sisbell

Require support for

  • API: Platform 29
  • NDK 21
  • Gradle 5.6.4
  • Build Tools and Platform Tools: 28.0.3

comment:2 Changed 5 months ago by sisbell

I've updated to handle toolchain used by fenix (and android-components). The standalone toolchain script is no longer supported. The following link covers the changes we need to make. On the plus side, the tar file will be smaller.

https://developer.android.com/ndk/guides/other_build_systems

In the config, I made more param versions configurable, as I expect we will need to upgrade frequently as fenix is in rapid development. I also added gradle to the path which means we don't need to specify full-path in projects.

The Commit:

https://github.com/sisbell/tor-browser-build/commits/bug-33557

comment:3 Changed 5 months ago by sisbell

Status: newneeds_review

comment:4 Changed 5 months ago by eighthave

Cc: eighthave removed

It should be possible to build the NDK from source without too much pain. Building Gradle and Android SDK without using mystery "prebuilt" binaries is very difficult. The Debian packages are the only project I'm aware of that is doing this.

FYI, F-Droid is running binary transparency logs for Gradle and Android SDK binaries, with an available static JSON URL:

comment:5 Changed 5 months ago by boklm

Keywords: TorBrowserTeam202003R added
Type: defecttask

comment:6 Changed 5 months ago by sysrqb

Reviewer: sysrqb

comment:7 in reply to:  2 ; Changed 5 months ago by gk

Replying to sisbell:

I've updated to handle toolchain used by fenix (and android-components). The standalone toolchain script is no longer supported. The following link covers the changes we need to make. On the plus side, the tar file will be smaller.

https://developer.android.com/ndk/guides/other_build_systems

In the config, I made more param versions configurable, as I expect we will need to upgrade frequently as fenix is in rapid development. I also added gradle to the path which means we don't need to specify full-path in projects.

The Commit:

https://github.com/sisbell/tor-browser-build/commits/bug-33557

I've not looked closely at how Fenix is built but isn't Mozilla using their own clang here? As we are doing for Fennec-based builds currently as well? If so, then we should bump the clang version to what Mozilla uses. And the Rust version, too. I don't see any child bugs for those in the list for the parent one and those two components are kind of toolchain components, so I figured you might want to deal with them in this ticket. :) But child bugs for #33184 would be fine, too, I think...

comment:8 in reply to:  7 Changed 5 months ago by sisbell

Replying to gk:

Replying to sisbell:

I've updated to handle toolchain used by fenix (and android-components). The standalone toolchain script is no longer supported. The following link covers the changes we need to make. On the plus side, the tar file will be smaller.

https://developer.android.com/ndk/guides/other_build_systems

In the config, I made more param versions configurable, as I expect we will need to upgrade frequently as fenix is in rapid development. I also added gradle to the path which means we don't need to specify full-path in projects.

The Commit:

https://github.com/sisbell/tor-browser-build/commits/bug-33557

I've not looked closely at how Fenix is built but isn't Mozilla using their own clang here? As we are doing for Fennec-based builds currently as well? If so, then we should bump the clang version to what Mozilla uses. And the Rust version, too. I don't see any child bugs for those in the list for the parent one and those two components are kind of toolchain components, so I figured you might want to deal with them in this ticket. :) But child bugs for #33184 would be fine, too, I think...

From what I can tell working through the build, fenix project itself is not doing any rust or native builds. This is handled by Mozillas application-services project and geckoview project

In regards to application-services, these deal with cloud services which we don't want to support.

https://github.com/mozilla/application-services

This is project is built by Mozilla and is downloaded by fenix as an aar, which native libraries.

https://maven.mozilla.org/maven2/org/mozilla/appservices/full-megazord/0.53.0/full-megazord-0.53.0.aar

Looking through the components, they are largely related to sync and firefox accounts, which I don't believe we want to support. I'm in the process of creating patches to remove these projects. I want to see what breaks without the application-service components.

The second part I see for native is is geckoview

https://searchfox.org/mozilla-central/source/mobile/android

This downloads as
https://maven.mozilla.org/maven2/org/mozilla/geckoview/geckoview-beta/74.0.20200227210932/geckoview-beta-74.0.20200227210932.aar

But yes, we do need an issue for this project (I'll open that). And this support will likely require some changes to the toolchain.

comment:9 Changed 5 months ago by sisbell

Another issue I encountered is that android-components and fenix use different version of the Android SDK, Fenix is on 28 and android-components is on 29. I'll need to bring these into alignment (up to 29 for fenix)

comment:10 Changed 5 months ago by pili

Sponsor: Sponsor58-must

Adding sponsor to tickets

comment:11 Changed 5 months ago by sisbell

This update sets the correct path to the toolchain

https://github.com/sisbell/tor-browser-build/commits/bug-33557a

comment:12 Changed 4 months ago by pili

Keywords: TorBrowserTeam202004R added; TorBrowserTeam202003R removed

We are no longer in March

comment:13 in reply to:  9 Changed 4 months ago by gk

Replying to sisbell:

Another issue I encountered is that android-components and fenix use different version of the Android SDK, Fenix is on 28 and android-components is on 29. I'll need to bring these into alignment (up to 29 for fenix)

That does not seem so easy. Right now Fenix is broken with our toolchain. It breaks during compilation like:

> Task :app:compileGeckoBetaDebugKotlin
e: /var/tmp/build/fenix-943d3837b243/app/src/main/java/org/mozilla/fenix/library/bookmarks/BookmarkController.kt: (81, 9): Val cannot be reassigned
e: /var/tmp/build/fenix-943d3837b243/app/src/main/java/org/mozilla/fenix/library/history/HistoryController.kt: (76, 9): Val cannot be reassigned
e: /var/tmp/build/fenix-943d3837b243/app/src/main/java/org/mozilla/fenix/utils/ClipboardHandler.kt: (32, 13): Val cannot be reassigned

That does not happen if I use our current toolchain. I can compile Fenix successfully with it.

I wonder if we should try harder to match what Mozilla is currently using toolchain-wise. That is whether we should bite the bullet and use different ones for Fenix and android-components...

comment:14 Changed 4 months ago by gk

Or maybe we can just ship both SDKs in our android-toolchain project and the respective code picks the one it needs somehow...

comment:16 Changed 4 months ago by gk

Another hard requirement is ndk r20 for geckoview. So, let's stick to that one instead of r21 assuming other projects don't require r21? (In that case we could think about shipping both as well, and then pointing the NDK path to the one actually needed for a project).

comment:17 Changed 4 months ago by gk

Okay, I started to look over the requirements for the different projects we need to build. Here comes what I have so far (as of June 11) and where to find the information:

m-c requirements

build-tools 29.0.3 (android-sdk.configure)
platform-29 (android.sdk.configure) (compileSdkVersion, tragetSdkVersion)
platform-tools (nothing pinned; sdkmanager fetches the latest)
sdk-tools (mozbootstrap downloads still sdk-tools-$OS-4333796.zip) (python/mozboot/mozboot/android.py)
android ndk r20 (1577220) (python/mozboot/mozboot/android.py)
min android 16 (64bit 21) (android-ndk.configure)
min sdk 16 (mobile/android/confvars.sh) (minSdkVersion)
gradle 5.1.1 (gradle/wrapper/gradle-wrapper.properties)
rust 1.41.1 (mozboot/mozboot/base.py MODERN_RUST_VERSION)
cbindgen 0.14.1 (build/moz.configure/bindgen.configure)
nasm >=2.14 (mozboot/mozboot/base.py MODERN_NASM_VERSION)
clang 9.0.1
node v10.21.0 (taskcluster/scripts/misc/repack-node.sh)
python3 >= 3.6 (build/moz.configure/init.configure)

Fenix requiremenets

platform-28 (app/build.gradle;buildSrc/src/main/java/Config.kt) (compileSdkVersion, targetSdkVersion)
min sdk 21 (buildSrc/src/main/java/Config.kt) (minSdkVersion)
gradle 5.6.4 (gradle/wrapper/gradle-wrapper.properties)

android-components requirements

platform-29 (buildSrc/src/main/java/Config.kt) (compileSdkVersion, targetSdkVersion = 29)
min sdk 21 (buildSrc/src/main/java/Config.kt) (minSdkVersion)
gradle 5.6.4 (gradle/wrapper/gradle-wrapper.properties)

Last edited 2 months ago by gk (previous) (diff)

comment:18 Changed 4 months ago by gk

Keywords: TorBrowserTeam202004 added; TorBrowserTeam202004R removed
Status: needs_reviewneeds_revision

comment:19 Changed 4 months ago by cypherpunks

build-tools 29.0.3 (python/mozboot/mozboot/android-packages.txt)
platforms android-29 (python/mozboot/mozboot/android-packages.txt)
platform-tools (python/mozboot/mozboot/android-packages.txt): >= 29.0.6 (no need to use anything different from the latest version for the current platform) https://bugzilla.mozilla.org/show_bug.cgi?id=1344244
sdk-tools (should download commandlinetools-linux-6200805_latest.zip): "SDK Tools package is deprecated and no longer receiving updates. Instead, please use the new command-line tools package." (2017) https://bugzilla.mozilla.org/show_bug.cgi?id=1519180

Another hard requirement is ndk r20 for geckoview.

android ndk >= r20 (see https://bugzilla.mozilla.org/show_bug.cgi?id=1266263#c6)

NDK r21b is LTS (security updates): https://android-developers.googleblog.com/2019/10/introducing-ndk-r21-our-first-long-term.html

rust >= 1.41.1 (python/mozboot/mozboot/base.py)
clang >= 10.0.0 (anyways) https://bugzilla.mozilla.org/show_bug.cgi?id=1616692
+ cbindgen >= 0.14.2 (cbindgen_min_version) https://bugzilla.mozilla.org/show_bug.cgi?id=1631929
node >= v12.16.3 (Latest Active LTS Version, security updates): v10.x lacks some security updates https://github.com/nodejs/node/pull/32583#issuecomment-607420108

Everything will be enforced up to 29 by Google (August 1) ;)

Fenix: https://github.com/mozilla-mobile/fenix/issues/7047

Last edited 3 months ago by cypherpunks (previous) (diff)

comment:20 Changed 4 months ago by sysrqb

https://firefox-source-docs.mozilla.org/build/buildsystem/toolchains.html#firefox-for-android-with-gradle

To build Firefox for Android with Gradle in automation, archives containing both
the Gradle executable and a Maven repository comprising the exact build
dependencies are produced and uploaded to an internal Mozilla server. The build
automation will download, verify, and extract these archive before building. These
archives provide a self-contained Gradle and Maven repository so that machines
don’t need to fetch additional Maven dependencies at build time. 

I wonder if (and how) we can re-use this. ./mach android gradle-dependencies is the starting point.

comment:21 in reply to:  14 Changed 4 months ago by gk

Replying to gk:

Or maybe we can just ship both SDKs in our android-toolchain project and the respective code picks the one it needs somehow...

The more I think the more do I feel we could and should do the same for the different Gradle versions used. The overhead is worth it if we can be sure to use the proper tools for each Mozilla projects to build, given that we are now a tiny team and weird bugs due to toolchain issues are hitting us even harder now.

comment:22 Changed 4 months ago by cypherpunks

The more I think the more do I feel we could and should do the same for the different Gradle versions used.

Just file a follow-up ticket to https://bugzilla.mozilla.org/show_bug.cgi?id=1522795 about "
Upgrade Android Gradle plugin to 3.6.*" (requires Gradle 5.6.4), and they will bump it for you.

comment:23 in reply to:  20 Changed 3 months ago by gk

Replying to sysrqb:

https://firefox-source-docs.mozilla.org/build/buildsystem/toolchains.html#firefox-for-android-with-gradle

To build Firefox for Android with Gradle in automation, archives containing both
the Gradle executable and a Maven repository comprising the exact build
dependencies are produced and uploaded to an internal Mozilla server. The build
automation will download, verify, and extract these archive before building. These
archives provide a self-contained Gradle and Maven repository so that machines
don’t need to fetch additional Maven dependencies at build time. 

I wonder if (and how) we can re-use this. ./mach android gradle-dependencies is the starting point.

Yes, seems like an interesting approach. I've opened #34110 so we can consider it at some point.

comment:24 Changed 3 months ago by gk

Keywords: tbb-rbm TorBrowserTeam202005 GeorgKoppen202005 added; Android TorBrowserTeam202004 removed

comment:25 Changed 3 months ago by gk

Cc: tbb-team added; sysrqb boklm removed
Owner: changed from tbb-team to gk
Status: needs_revisionassigned

comment:26 Changed 3 months ago by gk

Priority: MediumHigh

Moving tickets to high prio.

comment:27 Changed 3 months ago by gk

Keywords: TorBrowserTeam202005R added; TorBrowserTeam202005 removed
Status: assignedneeds_review

comment:28 Changed 3 months ago by gk

Keywords: TorBrowserTeam202005 added; TorBrowserTeam202005R removed
Status: needs_reviewneeds_revision

One annoying part I just realize is that we have $CROSS_PREFIX-ranlib, $CROSS_PREFIX-ar, $CROSS_PREFIX-ld etc. for aarch64, x86, and x86_64 which is why we can easily use var/configure_host for them, even for the clang part which has an $API included additionally.

Now, this does unfortunately not hold for armv7. There we have arm-linux-androideabi as CROSS_PREFIX but armv7a-linux-androideabi$ARCH-clang being used for the compiler.

Here is what we can do:

1) Work around that for the armv7 case (as done e.g. in the patch for #33833)
2) We could define yet another variable in rbm.conf making sure we have arm-linux-androideabi and armv7a-linux-androideabi available.
3) We could just symlink the CROSS_PREFIX for armv7 (currently arm-linux-androideabi) to armv7-linux-androideabi

I think I'll do 3) and use cross_prefix instead of configure_host as the former seems to be clearer than the latter.

Last edited 3 months ago by gk (previous) (diff)

comment:29 Changed 3 months ago by gk

Keywords: TorBrowserTeam202005R added; TorBrowserTeam202005 removed
Status: needs_revisionneeds_review

comment:31 Changed 3 months ago by gk

Keywords: TorBrowserTeam202005 added; TorBrowserTeam202005R removed
Status: needs_reviewneeds_revision

comment:32 Changed 3 months ago by gk

Keywords: TorBrowserTeam202005R added; TorBrowserTeam202005 removed
Reviewer: sysrqbsysrqb, ahf
Status: needs_revisionneeds_review

This should be in needs_review. The commit has not changed. The latest version is on bug_33559_v2 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_33559_v2&id=541ebffff200e06c1c55ab3c8facdfd02a9ae70e) (among other branches)

comment:33 Changed 3 months ago by ahf

Status: needs_reviewmerge_ready

541ebffff200e06c1c55ab3c8facdfd02a9ae70e was reviewed as part of the review for #33973.

comment:34 Changed 2 months ago by gk

Keywords: TorBrowserTeam202006R added; TorBrowserTeam202005R removed

Moving review tickets.

comment:35 Changed 2 months ago by gk

Keywords: gitlab-tb-tor-browser-build added

Add magic gitlab keyword.

Note: See TracTickets for help on using tickets.