Use Debian 10 for our Android container images
We should switch to Debian 10 for our Android builds.
Activity
Trac:
Parent Ticket: #31127 (moved)
#31924 (moved) is a duplicate.
Trac:
Cc: boklm to boklm, sisbell
If we need to install the
openjdk-8package from testing, then we should have some pinning configuration (see ticket:31564#comment:60) to only install this package from testing. We should also be using https://snapshot.debian.org/ for the testing repository to avoid issues when packages are removed or updated.And as it seems we'll need this pinning configuration for building most components, we should probably do it in
var/pre_pkginstinrbm.conf.I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: openjdk-8-jdk : Depends: openjdk-8-jre (= 8u232-b09-1) but it is not going to be installed Depends: openjdk-8-jdk-headless (= 8u232-b09-1) but it is not going to be installed E: Unable to correct problems, you have held broken packages.Also trying to use snapshot with pinning resulted in
Starting build: Mon Oct 21 23:03:49 2019 Hit:1 http://deb.debian.org/debian buster InRelease Get:2 http://deb.debian.org/debian buster/main Translation-en [5967 kB] Ign:3 https://snapshot.debian.org sid InRelease Err:4 https://snapshot.debian.org sid Release Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 185.17.185.185 443] Reading package lists... W: https://snapshot.debian.org/dists/sid/InRelease: No system certificates available. Try installing ca-certificates. W: https://snapshot.debian.org/dists/sid/Release: No system certificates available. Try installing ca-certificates. E: The repository 'https://snapshot.debian.org sid Release' does not have a Release file.Trac:
Status: new to needs_review-
Replying to sisbell:
I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
I don't think we need to have two different Ubuntu versions to generate the images. We could just use Ubuntu 19.10 to generate all images.
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I also think we should not do this configuration in
projects/debootstrap-image/config, but only in the containers that we use for android builds, for example by doing it invar/pre_pkginstinrbm.conf.Also trying to use snapshot with pinning resulted in
{{{ Starting build: Mon Oct 21 23:03:49 2019 Hit:1 http://deb.debian.org/debian buster InRelease Get:2 http://deb.debian.org/debian buster/main Translation-en [5967 kB] Ign:3 https://snapshot.debian.org sid InRelease Err:4 https://snapshot.debian.org sid Release Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 185.17.185.185 443] Reading package lists... W: https://snapshot.debian.org/dists/sid/InRelease: No system certificates available. Try installing ca-certificates. W: https://snapshot.debian.org/dists/sid/Release: No system certificates available. Try installing ca-certificates. E: The repository 'https://snapshot.debian.org sid Release' does not have a Release file. }}}
Did you try installing
ca-certificatesas suggested in the error message?Trac:
Status: needs_review to needs_revision Replying to boklm:
Replying to sisbell:
I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
I don't think we need to have two different Ubuntu versions to generate the images. We could just use Ubuntu 19.10 to generate all images. Ok I can change this.
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I had tried pinning the openjdk-8-jre and openjdk-8-headless but got the same result, they were missing.
I also think we should not do this configuration in
projects/debootstrap-image/config, but only in the containers that we use for android builds, for example by doing it invar/pre_pkginstinrbm.conf.I can move the apt sources change to rbm.conf. Assuming I don't modify the pre script in the debootstrap config for the android specific build, how is the # Bug 29158 going to be handled? Is this related to stretch or to the ubuntu version? Can it be removed now?
Also trying to use snapshot with pinning resulted in
{{{ Starting build: Mon Oct 21 23:03:49 2019 Hit:1 http://deb.debian.org/debian buster InRelease Get:2 http://deb.debian.org/debian buster/main Translation-en [5967 kB] Ign:3 https://snapshot.debian.org sid InRelease Err:4 https://snapshot.debian.org sid Release Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 185.17.185.185 443] Reading package lists... W: https://snapshot.debian.org/dists/sid/InRelease: No system certificates available. Try installing ca-certificates. W: https://snapshot.debian.org/dists/sid/Release: No system certificates available. Try installing ca-certificates. E: The repository 'https://snapshot.debian.org sid Release' does not have a Release file. }}}
Did you try installing
ca-certificatesas suggested in the error message?Replying to boklm:
Replying to sisbell:
I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
I don't think we need to have two different Ubuntu versions to generate the images. We could just use Ubuntu 19.10 to generate all images.
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I do see that without pinning we still use debian stable buster for a bunch of libraries (the first 54). I haven't been able to find a specification of how it does its resolving in this case (so I guess pinning is designed to force certain rules).
Get:50 http://deb.debian.org/debian buster/main amd64 libxdmcp-dev amd64 1:1.1.2-3 [42.2 kB] Get:51 http://deb.debian.org/debian buster/main amd64 xtrans-dev all 1.3.5-1 [100 kB] Get:52 http://deb.debian.org/debian buster/main amd64 libxcb1-dev amd64 1.13.1-2 [174 kB] Get:53 http://deb.debian.org/debian buster/main amd64 libxt-dev amd64 1:1.1.5-1+b3 [426 kB] Get:54 http://deb.debian.org/debian buster/main amd64 xdg-user-dirs amd64 0.17-2 [53.8 kB] Get:55 http://ftp.us.debian.org/debian unstable/main amd64 libglib2.0-0 amd64 2.62.1-1 [1317 kB] Get:56 http://ftp.us.debian.org/debian unstable/main amd64 libwebp6 amd64 0.6.1-2+b1 [261 kB] .... Get:114 http://ftp.us.debian.org/debian unstable/main amd64 libx11-dev amd64 2:1.6.8-1 [822 kB] Get:115 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jre-headless amd64 8u232-b09-1 [27.2 MB] Get:116 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jre amd64 8u232-b09-1 [69.7 kB] Get:117 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jdk-headless amd64 8u232-b09-1 [8183 kB] Get:118 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jdk amd64 8u232-b09-1 [1603 kB]-
Replying to sisbell:
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I had tried pinning the openjdk-8-jre and openjdk-8-headless but got the same result, they were missing.
Maybe the pinning configuration was wrong. But hard to say without seeing the actual configuration that was used, and how the openjdk package was installed.
If pinning is too complicate, an other option is to download and install the openjdk packages without using apt. In any case, adding the testing repository without pinning is not an option, as that just means we're using Debian testing and not buster.
I also think we should not do this configuration in
projects/debootstrap-image/config, but only in the containers that we use for android builds, for example by doing it invar/pre_pkginstinrbm.conf.I can move the apt sources change to rbm.conf. Assuming I don't modify the pre script in the debootstrap config for the android specific build, how is the # Bug 29158 going to be handled? Is this related to stretch or to the ubuntu version? Can it be removed now?
buster was released after
DSA 4371-1, so it is not affected by this issue. -
Replying to sisbell:
Replying to boklm:
Replying to sisbell:
I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
I don't think we need to have two different Ubuntu versions to generate the images. We could just use Ubuntu 19.10 to generate all images.
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I do see that without pinning we still use debian stable buster for a bunch of libraries (the first 54). I haven't been able to find a specification of how it does its resolving in this case (so I guess pinning is designed to force certain rules).
{{{ Get:50 http://deb.debian.org/debian buster/main amd64 libxdmcp-dev amd64 1:1.1.2-3 [42.2 kB] Get:51 http://deb.debian.org/debian buster/main amd64 xtrans-dev all 1.3.5-1 [100 kB] Get:52 http://deb.debian.org/debian buster/main amd64 libxcb1-dev amd64 1.13.1-2 [174 kB] Get:53 http://deb.debian.org/debian buster/main amd64 libxt-dev amd64 1:1.1.5-1+b3 [426 kB] Get:54 http://deb.debian.org/debian buster/main amd64 xdg-user-dirs amd64 0.17-2 [53.8 kB] Get:55 http://ftp.us.debian.org/debian unstable/main amd64 libglib2.0-0 amd64 2.62.1-1 [1317 kB] Get:56 http://ftp.us.debian.org/debian unstable/main amd64 libwebp6 amd64 0.6.1-2+b1 [261 kB] .... Get:114 http://ftp.us.debian.org/debian unstable/main amd64 libx11-dev amd64 2:1.6.8-1 [822 kB] Get:115 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jre-headless amd64 8u232-b09-1 [27.2 MB] Get:116 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jre amd64 8u232-b09-1 [69.7 kB] Get:117 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jdk-headless amd64 8u232-b09-1 [8183 kB] Get:118 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jdk amd64 8u232-b09-1 [1603 kB]
}}}
Those packages are exactly the same version in buster and in unstable.
-
Replying to sisbell:
I try to install ca-certificates but still get the same error regarding Certificate failed. I'll need to investigate more. Its installing from the following location
{{{ Get:3 http://archive.ubuntu.com/ubuntu eoan/main amd64 ca-certificates all 20190110 [146 kB] }}}
It needs to be installed in the debian buster container as this is where the apt using snapshot.debian.org is run, not the ubuntu one which we only use to run debootstrap.
Replying to boklm:
Replying to sisbell:
I try to install ca-certificates but still get the same error regarding Certificate failed. I'll need to investigate more. Its installing from the following location
Get:3 http://archive.ubuntu.com/ubuntu eoan/main amd64 ca-certificates all 20190110 [146 kB] }}}It needs to be installed in the debian buster container as this is where the apt using snapshot.debian.org is run, not the ubuntu one which we only use to run debootstrap.
Ok so the installation of ca-certificates works from container image. I'm using a specific version of the snapshot, which I think we would want to do to keep the packages pegged to a single snapshot.
{{{ https://snapshot.debian.org/archive/debian/20191022T222828Z/
Apt pinning also is working. I still get missing dependencies like below so I'm assuming I need to specify the entire dependency chain (and versions) here to get openjdk-8 installed? Or is there a simpler way?The following packages have unmet dependencies: openjdk-8-jre-headless : Depends: libc6 (>= 2.29) but 2.28-10 is to be installed E: Unable to correct problems, you have held broken packages.
-
Replying to sisbell:
I still get missing dependencies like below so I'm assuming I need to specify the entire dependency chain (and versions) here to get openjdk-8 installed? Or is there a simpler way?
{{{ The following packages have unmet dependencies: openjdk-8-jre-headless : Depends: libc6 (>= 2.29) but 2.28-10 is to be installed E: Unable to correct problems, you have held broken packages. }}}
It looks like openjdk-8-jre-headless version
8u222-b10-1from September still depends on libc6 2.28, so we could use that version instead to avoid updating the libc. When I try to apply an older snapshot, it says expired. So I'm wondering if specifying a specific snapshot release is really the the way to go (and if it isn't ,the snapshots and dependencies will change).
Running hooks in /etc/ca-certificates/update.d... done. Hit:1 http://deb.debian.org/debian buster InRelease Get:2 http://deb.debian.org/debian buster/main Translation-en [5967 kB] Get:3 https://snapshot.debian.org/archive/debian/20190930T204642Z unstable InRelease [139 kB] Reading package lists... E: Release file for https://snapshot.debian.org/archive/debian/20190930T204642Z/dists/unstable/InRelease is expired (invalid since 15d 22h 9min 17s). Updates for this repository will not be applied.My pinning specifies release. I'll look into if this is an issue with snapshots /InRelease expiration
Package: openjdk-8-jdk Pin: origin snapshot.debian.org Pin: release a=unstable Pin-Priority: 600I can get the following installed correctly in debootstrap-image config. I can tell this from the logs. But java doesn't show up in the container image when building gradle projects.
apt-get install -y -q ca-certificates-java java-common libcups2 liblcms2-2 libjpeg8 libfontconfig1 libnss3 libfreetype6 libpcsclite1 libx11-6 libxext6 libxi6 libxrender1 libxtst6 libglib2.0-0 libxrandr2 libxinerama1 libgl1-mesa-glx libgtk2.0-0 libatk-wrapper-java-jni libgif7 libpulse0 isc-dhcp-common dpkg -i ./openjdk-8-jre-headless.deb dpkg -i ./openjdk-8-jdk-headless.deb dpkg -i ./openjdk-8-jre.deb dpkg -i ./openjdk-8-jdk.deb- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jre-headless_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jre-headless.deb - URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jre_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jre.deb - URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jdk_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jdk.deb - URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jdk-headless_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jdk-headless.debI tried moving everything above to container-image config but the build fails. I'm not sure why there would be a difference.
E: Unable to locate package libjpeg8-
Replying to sisbell:
I can get the following installed correctly in debootstrap-image config. I can tell this from the logs. But java doesn't show up in the container image when building gradle projects.
{{{ apt-get install -y -q ca-certificates-java java-common libcups2 liblcms2-2 libjpeg8 libfontconfig1 libnss3 libfreetype6 libpcsclite1 libx11-6 libxext6 libxi6 libxrender1 libxtst6 libglib2.0-0 libxrandr2 libxinerama1 libgl1-mesa-glx libgtk2.0-0 libatk-wrapper-java-jni libgif7 libpulse0 isc-dhcp-common dpkg -i ./openjdk-8-jre-headless.deb dpkg -i ./openjdk-8-jdk-headless.deb dpkg -i ./openjdk-8-jre.deb dpkg -i ./openjdk-8-jdk.deb }}}
{{{
- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jre-headless_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jre-headless.deb
- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jre_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jre.deb
- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jdk_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jdk.deb
- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jdk-headless_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jdk-headless.deb }}}
Why are you using Ubuntu packages? We want to use Debian buster, so Ubuntu package is not what we should use.
And the Ubuntu container is only used to run
debootstrapto create the Debian tarballs, so we don't need openjdk there.I tried moving everything above to container-image config but the build fails. I'm not sure why there would be a difference.
{{{ E: Unable to locate package libjpeg8 }}}
Difficult to say without seeing actual patches (I have no idea where you added those lines in the
debootstrap-imageandcontainer-imageconfigs), but maybe you installed the packages in the Ubuntu container in the first case, and in the buster container in the second.But anyway, as I said in comment:2 and comment:5 we should not do this in
debootstrap-imageorcontainer-image, but inrbm.confinvar/pre_pkginst(or maybevar/post_pkginst). And as we don't want to copy the URLs in theinput_filesof all projects, we should download them invar/pre_pkginst(orvar/post_pkginst) using wget (followed by a sha256sum verification). I'm still cleaning up the full build. I also need to add the checksums but the following works perfectly within the rbm.conf
post_pkginst: | OPENJDK_URL=https://snapshot.debian.org/archive/debian/20191028T162808Z/pool/main/o/openjdk-8 JDK_VERSION=8u222-b10-1~deb9u1_amd64 apt-get install -y -q wget ca-certificates-java java-common libcups2 liblcms2-2 libjpeg62-turbo libfontconfig1 libnss3 libfreetype6 libpcsclite1 libx11-6 libglib2.0-0 libxrandr2 libxinerama1 libgl1 libgl1-mesa-glx libgtk2.0-0 libatk-wrapper-java-jni libgif7 libpulse0 wget $OPENJDK_URL/openjdk-8-jdk_$JDK_VERSION.deb wget $OPENJDK_URL/openjdk-8-jdk-headless_$JDK_VERSION.deb wget $OPENJDK_URL/openjdk-8-jre-headless_$JDK_VERSION.deb wget $OPENJDK_URL/openjdk-8-jre_$JDK_VERSION.deb dpkg -i ./openjdk-8-jre-headless_$JDK_VERSION.deb dpkg -i ./openjdk-8-jdk-headless_$JDK_VERSION.deb dpkg -i ./openjdk-8-jre_$JDK_VERSION.deb dpkg -i ./openjdk-8-jdk_$JDK_VERSION.deb-
8u222-b10-1~deb9u1is the version that was built for stretch. Is there any reason to use that one rather than8u222-b10-1from Unstable?Or if we decide to use the package from stretch, then we might as well use the latest version of the package (as it shouldn't require a new glibc version in this case), which is
8u232-b09-1~deb9u1. Replying to boklm:
8u222-b10-1~deb9u1is the version that was built for stretch. Is there any reason to use that one rather than8u222-b10-1from Unstable? I'm going from your comment:2 which said I should be using snapshot, rather than unstable. This 222 version is the one in snapshot. Should I switch back to unstable?Or if we decide to use the package from stretch, then we might as well use the latest version of the package (as it shouldn't require a new glibc version in this case), which is
8u232-b09-1~deb9u1.So should we go with stretch from snapshot or the unstable version?
-
Replying to sisbell:
Replying to boklm:
8u222-b10-1~deb9u1is the version that was built for stretch. Is there any reason to use that one rather than8u222-b10-1from Unstable? I'm going from your comment:2 which said I should be using snapshot, rather than unstable. This 222 version is the one in snapshot. Should I switch back to unstable?I did not say snapshot rather than unstable, which does not make any sense since snapshot contains packages from all suites, including unstable. The packages in unstable are changing frequently, so using snapshot.debian.org allows us to use the packages that were available on a specific date.
If we want to use the package from unstable, the latest version that does not require a new glibc seems to be
8u222-b10-1, and we can download it from snapshot.debian.org: https://snapshot.debian.org/archive/debian/20190930T145141Z/pool/main/o/openjdk-8/openjdk-8-jdk_8u222-b10-1_amd64.debIt seems using the package from stretch is an other option. And in this case we could use the latest version of the package (which is
8u232-b09-1~deb9u1currently) as it doesn't require a new glibc: https://deb.debian.org/debian/pool/main/o/openjdk-8/openjdk-8-jdk_8u232-b09-1~deb9u1_amd64.debOr if we decide to use the package from stretch, then we might as well use the latest version of the package (as it shouldn't require a new glibc version in this case), which is
8u232-b09-1~deb9u1.So should we go with stretch from snapshot or the unstable version?
The package from stretch might be better since it allows us to use a version with the latest security fixes. However I don't know if this package works correctly on buster, so this needs to be tested.
I tried the stretch version but got the following when building the firefox project (although tor-android-service and tor-onion-proxy-library built fine). I'll collect some more logs on the NPE and then I'll try out the buster version next.
0:11.78 ../../../config/nsinstall -R -m 644 'TelemetryEventEnums.h' '../../../dist/include/mozilla' 0:11.78 ../../../config/nsinstall -R -m 644 'TelemetryHistogramEnums.h' '../../../dist/include/mozilla' 0:11.78 ../../../config/nsinstall -R -m 644 'TelemetryProcessEnums.h' '../../../dist/include/mozilla' 0:11.78 ../../../config/nsinstall -R -m 644 'TelemetryScalarEnums.h' '../../../dist/include/mozilla' 0:11.78 make[4]: Leaving directory '/var/tmp/build/firefox-e14b7b74e92c/obj-arm-linux-androideabi/toolkit/components/telemetry' 0:18.14 FAILURE: Build failed with an exception. 0:18.14 * What went wrong: 0:18.14 A problem occurred configuring project ':app'. 0:18.14 > java.lang.NullPointerException (no error message) 0:18.14 * Try: 0:18.14 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. 0:18.14 * Get more help at https://help.gradle.org 0:18.14 Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. 0:18.14 Use '--warning-mode all' to show the individual deprecation warnings. 0:18.14 See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings 0:18.14 BUILD FAILED in 13sOnce I set JAVA_HOME env, it clears up the NPE error. So everything looks good with building with stretch openjdK, up through firefox project.
Replying to sisbell:
I tried the stretch version but got the following when building the firefox project (although tor-android-service and tor-onion-proxy-library built fine). I'll collect some more logs on the NPE and then I'll try out the buster version next.
{{{ 0:11.78 ../../../config/nsinstall -R -m 644 'TelemetryEventEnums.h' '../../../dist/include/mozilla' 0:11.78 ../../../config/nsinstall -R -m 644 'TelemetryHistogramEnums.h' '../../../dist/include/mozilla' 0:11.78 ../../../config/nsinstall -R -m 644 'TelemetryProcessEnums.h' '../../../dist/include/mozilla' 0:11.78 ../../../config/nsinstall -R -m 644 'TelemetryScalarEnums.h' '../../../dist/include/mozilla' 0:11.78 make[4]: Leaving directory '/var/tmp/build/firefox-e14b7b74e92c/obj-arm-linux-androideabi/toolkit/components/telemetry' 0:18.14 FAILURE: Build failed with an exception. 0:18.14 * What went wrong: 0:18.14 A problem occurred configuring project ':app'. 0:18.14 > java.lang.NullPointerException (no error message) 0:18.14 * Try: 0:18.14 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. 0:18.14 * Get more help at https://help.gradle.org 0:18.14 Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. 0:18.14 Use '--warning-mode all' to show the individual deprecation warnings. 0:18.14 See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings 0:18.14 BUILD FAILED in 13s
}}}
I have the latest.
https://github.com/sisbell/tor-browser-build/commit/608f2c9623235f574518f9351c28ea4a8c833101
This builds through firefox but fails with https-everywhere stage. I'll fix https-everywhere project to handle buster. I also still need to get the sha checks but thought I would put it out for review of what I have so I can go ahead and make any changes and to make sure this is the approach everyone agrees on.
Using directory /home/shane/github/tor-browser-4/out/firefox/firefox-e14b7b74e92c-android-armv7-b6443a Building project https-everywhere - https-everywhere-2019.6.27-03f7dd.xpi Tag 2019.6.27 is signed with key 1073E74EB38BD6D19476CBF8EA9DBF9FB761A677 Created /home/shane/github/tor-browser-4/tmp/rbm-dEXsD/https-everywhere-2019.6.27.tar.gz Building project container-image - container-image_stretch-amd64-4c35a5822622.tar.gz Building project debootstrap-image - container-image_stretch-amd64-2.tar.gz Using file /home/shane/github/tor-browser-4/out/debootstrap-image/container-image_ubuntu-base-19.10-base-amd64.tar.gz Using file /home/shane/github/tor-browser-4/out/debootstrap-image/apt_1.6.6ubuntu0.1_amd64.deb Using file /home/shane/github/tor-browser-4/out/debootstrap-image/libapt-pkg5.0_1.6.6ubuntu0.1_amd64.deb Build log: /home/shane/github/tor-browser-4/logs/debootstrap-image.log ---- Starting build: Mon Nov 4 06:41:08 2019 dpkg: warning: downgrading apt from 1.9.4 to 1.6.6ubuntu0.1 (Reading database ... 4098 files and directories currently installed.) Preparing to unpack ./apt_1.6.6ubuntu0.1_amd64.deb ... Unpacking apt (1.6.6ubuntu0.1) over (1.9.4) ... Selecting previously unselected package libapt-pkg5.0:amd64. Preparing to unpack .../libapt-pkg5.0_1.6.6ubuntu0.1_amd64.deb ... Unpacking libapt-pkg5.0:amd64 (1.6.6ubuntu0.1) ... dpkg: dependency problems prevent configuration of libapt-pkg5.0:amd64: dpkg (1.19.7ubuntu2) breaks libapt-pkg5.0 (<< 1.7~b) and is installed. Version of libapt-pkg5.0:amd64 to be configured is 1.6.6ubuntu0.1. dpkg: error processing package libapt-pkg5.0:amd64 (--install): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of apt: apt depends on libapt-pkg5.0 (>= 1.6.6ubuntu0.1); however: Package libapt-pkg5.0:amd64 is not configured yet. dpkg: error processing package apt (--install): dependency problems - leaving unconfigured Processing triggers for libc-bin (2.30-0ubuntu2) ... Errors were encountered while processing: libapt-pkg5.0:amd64 aptTrac:
Status: needs_revision to needs_review
Keywords: N/A deleted, TorBrowserTeam201911R added-
Replying to sisbell:
This builds through firefox but fails with https-everywhere stage. I'll fix https-everywhere project to handle buster.
I think the update of http-everywhere to use buster is unrelated to this ticket. So it can be done in a separate ticket.
-
Replying to sisbell:
I have the latest.
https://github.com/sisbell/tor-browser-build/commit/608f2c9623235f574518f9351c28ea4a8c833101
This builds through firefox but fails with https-everywhere stage. I'll fix https-everywhere project to handle buster. I also still need to get the sha checks but thought I would put it out for review of what I have so I can go ahead and make any changes and to make sure this is the approach everyone agrees on.
{{{
Using directory /home/shane/github/tor-browser-4/out/firefox/firefox-e14b7b74e92c-android-armv7-b6443a Building project https-everywhere - https-everywhere-2019.6.27-03f7dd.xpi Tag 2019.6.27 is signed with key 1073E74EB38BD6D19476CBF8EA9DBF9FB761A677 Created /home/shane/github/tor-browser-4/tmp/rbm-dEXsD/https-everywhere-2019.6.27.tar.gz Building project container-image - container-image_stretch-amd64-4c35a5822622.tar.gz Building project debootstrap-image - container-image_stretch-amd64-2.tar.gz Using file /home/shane/github/tor-browser-4/out/debootstrap-image/container-image_ubuntu-base-19.10-base-amd64.tar.gz Using file /home/shane/github/tor-browser-4/out/debootstrap-image/apt_1.6.6ubuntu0.1_amd64.deb Using file /home/shane/github/tor-browser-4/out/debootstrap-image/libapt-pkg5.0_1.6.6ubuntu0.1_amd64.deb Build log: /home/shane/github/tor-browser-4/logs/debootstrap-image.log
Starting build: Mon Nov 4 06:41:08 2019 dpkg: warning: downgrading apt from 1.9.4 to 1.6.6ubuntu0.1 (Reading database ... 4098 files and directories currently installed.) Preparing to unpack ./apt_1.6.6ubuntu0.1_amd64.deb ... Unpacking apt (1.6.6ubuntu0.1) over (1.9.4) ... Selecting previously unselected package libapt-pkg5.0:amd64. Preparing to unpack .../libapt-pkg5.0_1.6.6ubuntu0.1_amd64.deb ... Unpacking libapt-pkg5.0:amd64 (1.6.6ubuntu0.1) ... dpkg: dependency problems prevent configuration of libapt-pkg5.0:amd64: dpkg (1.19.7ubuntu2) breaks libapt-pkg5.0 (<< 1.7~b) and is installed. Version of libapt-pkg5.0:amd64 to be configured is 1.6.6ubuntu0.1.
dpkg: error processing package libapt-pkg5.0:amd64 (--install): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of apt: apt depends on libapt-pkg5.0 (>= 1.6.6ubuntu0.1); however: Package libapt-pkg5.0:amd64 is not configured yet.
dpkg: error processing package apt (--install): dependency problems - leaving unconfigured Processing triggers for libc-bin (2.30-0ubuntu2) ... Errors were encountered while processing: libapt-pkg5.0:amd64 apt
}}}
I think you get this error when building any debootstrap image that is not buster. In this case https-everywhere is using stretch.
The issue is that in
projects/debootstrap-image/configyou are trying to install some downloaded apt packages, which doesn't work since those packages are for an Ubuntu release older than the one we now use. Since we're using Ubuntu 19.10 now, there is no need to install specific apt packages. -
Some comments:
-
This ticket is about using Debian 10 for our Android containers and #31992 (moved) is about removing our
apktoolworkaround. Yet, it seems you are essentially mixing both issues in one commit saying it fixes #31110 (moved) which is confusing. Could we have two commits here one for #31110 (moved) (that is upgrading tobuster) and one (on top of it) for #31992 (moved)? Ideally, the build would still work if each commit would be used for building. -
Please remove the whitespace at the end of
+ JDK_VERSION=8u232-b09-1~deb9u1_amd64Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201911R deleted, TorBrowserTeam201911 added -
I cleaned up the configs. I added an install_jdk flag to prevent openjdk install in https-everywhere project. Otherwise the build would break on installation.
https://github.com/sisbell/tor-browser-build/commit/1291e758e91784d739c9051873dc07073811df16
It looks very close, I'm still investigating why I get the following error in tor-browser project when using apksigner.
inflating: tmp/removed-files Error: Unable to access jarfile /usr/share/apksigner/apksigner.jarTrac:
Keywords: TorBrowserTeam201911 deleted, TorBrowserTeam201911R added
Status: needs_revision to needs_review-
Replying to sisbell:
I cleaned up the configs. I added an install_jdk flag to prevent openjdk install in https-everywhere project. Otherwise the build would break on installation.
https://github.com/sisbell/tor-browser-build/commit/1291e758e91784d739c9051873dc07073811df16
It looks very close, I'm still investigating why I get the following error in tor-browser project when using apksigner.
{{{ inflating: tmp/removed-files
Error: Unable to access jarfile /usr/share/apksigner/apksigner.jar}}}
Once you fixed that bit, please add the ticket back into
needs_review. And please address all comments before putting a ticket intoneeds_review(That does not necessarily mean you need to fix all concerns as they might not be valid, but they should be addressed).Trac:
Keywords: TorBrowserTeam201911R deleted, TorBrowserTeam201911 added
Status: needs_review to needs_revision -
Replying to sisbell:
I cleaned up the configs. I added an install_jdk flag to prevent openjdk install in https-everywhere project. Otherwise the build would break on installation.
I think you don't need an install_jdk flag. You can just define
var/pre_pkginstto empty in http-everywhere. -
Replying to sisbell:
It looks very close, I'm still investigating why I get the following error in tor-browser project when using apksigner.
{{{ inflating: tmp/removed-files
Error: Unable to access jarfile /usr/share/apksigner/apksigner.jar}}}
The path to apksigner in buster is
/usr/lib/android-sdk/build-tools/debian/apksigner.jar. Replying to boklm:
Replying to sisbell:
It looks very close, I'm still investigating why I get the following error in tor-browser project when using apksigner.
{{{ inflating: tmp/removed-files
Error: Unable to access jarfile /usr/share/apksigner/apksigner.jar}}}
The path to apksigner in buster is
/usr/lib/android-sdk/build-tools/debian/apksigner.jar.So I think removing the install_jdk and getting the apksigner path updated should do the trick to get everything building. I'll get this building today and test out.
Fixed
- added correct apksigner path for debian
- removed use of install_jdk flag (override pre_pkginst)
https://github.com/sisbell/tor-browser-build/commit/da928c8cc046ad71e718ddd1eb2c4690af8097dd
Trac:
Keywords: TorBrowserTeam201911 deleted, TorBrowserTeam201911R added
Status: needs_revision to needs_review-
Replying to sisbell:
Fixed
- added correct apksigner path for debian
- removed use of install_jdk flag (override pre_pkginst)
https://github.com/sisbell/tor-browser-build/commit/da928c8cc046ad71e718ddd1eb2c4690af8097dd
in
var/pre_pkginst:sha256sum 04f35746d85ea065a06a79a0ffb95825bd591a1b3a0f7cc641d06e7d263bbef5 openjdk-8-jdk_$JDK_VERSION.deb | sha256sum -c sha256sum 92b4f8fb77d793a86e0b03b3b0750592b40a26a5d75956d10dd984a7b3aad4c9 openjdk-8-jdk-headless_$JDK_VERSION.deb | sha256sum -c sha256sum 84bf52b6cce20ead08b0d5b9fd9b81b4aa3da385ca951b313fe11d5cb1aa4d17 openjdk-8-jre-headless_$JDK_VERSION.deb | sha256sum -c sha256sum 75057205b56791edbc4dd0358cb56efd7d20ce0cb730d6ecba38e37267d72b9b openjdk-8-jre_$JDK_VERSION.deb | sha256sum -cThis is wrong, we could replace the checksum in those lines by anything, and it will never fail. Running
sha245sum checksum some-file | sha256sum -cwill complain that the filechecksumdoes not exist, but it is basically equivalent to runningsha245sum some-file | sha256sum -cwhich is equivalent to doing nothing.Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201911R deleted, TorBrowserTeam201911 added I needed to echo the checksum, so made this change and verified that an incorrect checksum fails the build.
https://github.com/sisbell/tor-browser-build/commit/9107a6849ec8f7c7e3073a125e93750bf3eb529e
Trac:
Status: needs_revision to needs_review
Keywords: TorBrowserTeam201911 deleted, TorBrowserTeam201911R added-
Replying to sisbell:
I needed to echo the checksum, so made this change and verified that an incorrect checksum fails the build.
https://github.com/sisbell/tor-browser-build/commit/9107a6849ec8f7c7e3073a125e93750bf3eb529e
This is still not good:
+ sha256sum 04f35746d85ea065a06a79a0ffb95825bd591a1b3a0f7cc641d06e7d263bbef5 openjdk-8-jdk_$JDK_VERSION.deb | sha256sum -c + sha256sum 92b4f8fb77d793a86e0b03b3b0750592b40a26a5d75956d10dd984a7b3aad4c9 openjdk-8-jdk-headless_$JDK_VERSION.deb | sha256sum -c + sha256sum 84bf52b6cce20ead08b0d5b9fd9b81b4aa3da385ca951b313fe11d5cb1aa4d17 openjdk-8-jre-headless_$JDK_VERSION.deb | sha256sum -c + sha256sum 75057205b56791edbc4dd0358cb56efd7d20ce0cb730d6ecba38e37267d72b9b openjdk-8-jre_$JDK_VERSION.deb | sha256sum -cTrac:
Keywords: TorBrowserTeam201911R deleted, TorBrowserTeam201911 added
Status: needs_review to needs_revision Sorry about that, I made a mistake on the check in
Try this one
https://github.com/sisbell/tor-browser-build/commit/dac5088c383692aa22617d7d1e628e4a348abab5
a bad checksum will look like
2019-11-13 18:05:53 (3.98 MB/s) - 'openjdk-8-jre_8u232-b09-1~deb9u1_amd64.deb' saved [69506/69506] openjdk-8-jdk_8u232-b09-1~deb9u1_amd64.deb: FAILED sha256sum: WARNING: 1 computed checksum did NOT matchTrac:
Status: needs_revision to needs_review
Keywords: TorBrowserTeam201911 deleted, TorBrowserTeam201911R addedYes, we need these components. Removing them results in
dpkg: dependency problems prevent configuration of openjdk-8-jdk:amd64: openjdk-8-jdk:amd64 depends on openjdk-8-jdk-headless (= 8u232-b09-1~deb9u1); however: Package openjdk-8-jdk-headless is not installed. dpkg: error processing package openjdk-8-jdk:amd64 (--install): dependency problems - leaving unconfiguredSimilar errors occur if removing the JRE packages.
Replying to gk:
How much of OpenJDK do we really need? That is, do we really need jdk, jdk-headless, jre, and jre-headless? If not, please remove the parts that are not needed (together with dependencies not needed).
-
Replying to sisbell:
Yes, we need these components. Removing them results in
{{{ dpkg: dependency problems prevent configuration of openjdk-8-jdk:amd64: openjdk-8-jdk:amd64 depends on openjdk-8-jdk-headless (= 8u232-b09-1~deb9u1); however: Package openjdk-8-jdk-headless is not installed.
dpkg: error processing package openjdk-8-jdk:amd64 (--install): dependency problems - leaving unconfigured }}}
Similar errors occur if removing the JRE packages.
I wonder why then just using the headless versions and not the other ones are working for me. Could you please fix that up? Please make sure as well that we really need all the packages you want to install with
apt-get installin thepre_pkginststep.Replying to gk:
How much of OpenJDK do we really need? That is, do we really need jdk, jdk-headless, jre, and jre-headless? If not, please remove the parts that are not needed (together with dependencies not needed).
Trac:
Keywords: TorBrowserTeam201911R deleted, TorBrowserTeam201911 added
Status: needs_review to needs_revision I just included the headless jre-headless/jdk-headless and dependencies in this version. I verified a complete build.
https://github.com/sisbell/tor-browser-build/commit/f370600beb2b3fb5ff722f5a737874074fb60397
Trac:
Keywords: TorBrowserTeam201911 deleted, TorBrowserTeam201911R added
Status: needs_revision to needs_review-
Replying to sisbell:
I just included the headless jre-headless/jdk-headless and dependencies in this version. I verified a complete build.
https://github.com/sisbell/tor-browser-build/commit/f370600beb2b3fb5ff722f5a737874074fb60397
Thanks! If you look at the dependencies you'll see that
libxtst6already depends onlibx11-6,libxext6, andlibxi6. So, there is no need to add them explicitly (we don't add all the other implicit dependencies either).Trac:
Keywords: TorBrowserTeam201911R deleted, TorBrowserTeam201911 added
Status: needs_review to needs_revision Looks like we just need one additional dependency installed for the JDK headless installation. I verified a full build.
https://github.com/sisbell/tor-browser-build/commit/005b6651ba42737c7e1bd177f01286294355c02f
Trac:
Status: needs_revision to needs_review