Use Debian 10 for our Android container images
We should switch to Debian 10 for our Android builds.
Activity
Trac:
Parent Ticket: #31127 (moved)
#31924 (moved) is a duplicate.
Trac:
Cc: boklm to boklm, sisbell
If we need to install the
openjdk-8package from testing, then we should have some pinning configuration (see ticket:31564#comment:60) to only install this package from testing. We should also be using https://snapshot.debian.org/ for the testing repository to avoid issues when packages are removed or updated.And as it seems we'll need this pinning configuration for building most components, we should probably do it in
var/pre_pkginstinrbm.conf.I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: openjdk-8-jdk : Depends: openjdk-8-jre (= 8u232-b09-1) but it is not going to be installed Depends: openjdk-8-jdk-headless (= 8u232-b09-1) but it is not going to be installed E: Unable to correct problems, you have held broken packages.Also trying to use snapshot with pinning resulted in
Starting build: Mon Oct 21 23:03:49 2019 Hit:1 http://deb.debian.org/debian buster InRelease Get:2 http://deb.debian.org/debian buster/main Translation-en [5967 kB] Ign:3 https://snapshot.debian.org sid InRelease Err:4 https://snapshot.debian.org sid Release Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 185.17.185.185 443] Reading package lists... W: https://snapshot.debian.org/dists/sid/InRelease: No system certificates available. Try installing ca-certificates. W: https://snapshot.debian.org/dists/sid/Release: No system certificates available. Try installing ca-certificates. E: The repository 'https://snapshot.debian.org sid Release' does not have a Release file.Trac:
Status: new to needs_review-
Replying to sisbell:
I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
I don't think we need to have two different Ubuntu versions to generate the images. We could just use Ubuntu 19.10 to generate all images.
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I also think we should not do this configuration in
projects/debootstrap-image/config, but only in the containers that we use for android builds, for example by doing it invar/pre_pkginstinrbm.conf.Also trying to use snapshot with pinning resulted in
{{{ Starting build: Mon Oct 21 23:03:49 2019 Hit:1 http://deb.debian.org/debian buster InRelease Get:2 http://deb.debian.org/debian buster/main Translation-en [5967 kB] Ign:3 https://snapshot.debian.org sid InRelease Err:4 https://snapshot.debian.org sid Release Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 185.17.185.185 443] Reading package lists... W: https://snapshot.debian.org/dists/sid/InRelease: No system certificates available. Try installing ca-certificates. W: https://snapshot.debian.org/dists/sid/Release: No system certificates available. Try installing ca-certificates. E: The repository 'https://snapshot.debian.org sid Release' does not have a Release file. }}}
Did you try installing
ca-certificatesas suggested in the error message?Trac:
Status: needs_review to needs_revision Replying to boklm:
Replying to sisbell:
I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
I don't think we need to have two different Ubuntu versions to generate the images. We could just use Ubuntu 19.10 to generate all images. Ok I can change this.
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I had tried pinning the openjdk-8-jre and openjdk-8-headless but got the same result, they were missing.
I also think we should not do this configuration in
projects/debootstrap-image/config, but only in the containers that we use for android builds, for example by doing it invar/pre_pkginstinrbm.conf.I can move the apt sources change to rbm.conf. Assuming I don't modify the pre script in the debootstrap config for the android specific build, how is the # Bug 29158 going to be handled? Is this related to stretch or to the ubuntu version? Can it be removed now?
Also trying to use snapshot with pinning resulted in
{{{ Starting build: Mon Oct 21 23:03:49 2019 Hit:1 http://deb.debian.org/debian buster InRelease Get:2 http://deb.debian.org/debian buster/main Translation-en [5967 kB] Ign:3 https://snapshot.debian.org sid InRelease Err:4 https://snapshot.debian.org sid Release Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 185.17.185.185 443] Reading package lists... W: https://snapshot.debian.org/dists/sid/InRelease: No system certificates available. Try installing ca-certificates. W: https://snapshot.debian.org/dists/sid/Release: No system certificates available. Try installing ca-certificates. E: The repository 'https://snapshot.debian.org sid Release' does not have a Release file. }}}
Did you try installing
ca-certificatesas suggested in the error message?Replying to boklm:
Replying to sisbell:
I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
I don't think we need to have two different Ubuntu versions to generate the images. We could just use Ubuntu 19.10 to generate all images.
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I do see that without pinning we still use debian stable buster for a bunch of libraries (the first 54). I haven't been able to find a specification of how it does its resolving in this case (so I guess pinning is designed to force certain rules).
Get:50 http://deb.debian.org/debian buster/main amd64 libxdmcp-dev amd64 1:1.1.2-3 [42.2 kB] Get:51 http://deb.debian.org/debian buster/main amd64 xtrans-dev all 1.3.5-1 [100 kB] Get:52 http://deb.debian.org/debian buster/main amd64 libxcb1-dev amd64 1.13.1-2 [174 kB] Get:53 http://deb.debian.org/debian buster/main amd64 libxt-dev amd64 1:1.1.5-1+b3 [426 kB] Get:54 http://deb.debian.org/debian buster/main amd64 xdg-user-dirs amd64 0.17-2 [53.8 kB] Get:55 http://ftp.us.debian.org/debian unstable/main amd64 libglib2.0-0 amd64 2.62.1-1 [1317 kB] Get:56 http://ftp.us.debian.org/debian unstable/main amd64 libwebp6 amd64 0.6.1-2+b1 [261 kB] .... Get:114 http://ftp.us.debian.org/debian unstable/main amd64 libx11-dev amd64 2:1.6.8-1 [822 kB] Get:115 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jre-headless amd64 8u232-b09-1 [27.2 MB] Get:116 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jre amd64 8u232-b09-1 [69.7 kB] Get:117 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jdk-headless amd64 8u232-b09-1 [8183 kB] Get:118 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jdk amd64 8u232-b09-1 [1603 kB]-
Replying to sisbell:
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I had tried pinning the openjdk-8-jre and openjdk-8-headless but got the same result, they were missing.
Maybe the pinning configuration was wrong. But hard to say without seeing the actual configuration that was used, and how the openjdk package was installed.
If pinning is too complicate, an other option is to download and install the openjdk packages without using apt. In any case, adding the testing repository without pinning is not an option, as that just means we're using Debian testing and not buster.
I also think we should not do this configuration in
projects/debootstrap-image/config, but only in the containers that we use for android builds, for example by doing it invar/pre_pkginstinrbm.conf.I can move the apt sources change to rbm.conf. Assuming I don't modify the pre script in the debootstrap config for the android specific build, how is the # Bug 29158 going to be handled? Is this related to stretch or to the ubuntu version? Can it be removed now?
buster was released after
DSA 4371-1, so it is not affected by this issue. -
Replying to sisbell:
Replying to boklm:
Replying to sisbell:
I kept in stretch for desktop builds, while adding in buster for android.
https://github.com/sisbell/tor-browser-build/commit/8a39f6de0ea833f63ae00e5d3a6f2e2503a6572a
I don't think we need to have two different Ubuntu versions to generate the images. We could just use Ubuntu 19.10 to generate all images.
The above commit works fine for openjdk-8. When I attempted to add pinning I would get some dependency errors and I didn't see any advantage to making this more complicated.
No pinning means that we are not using Debian buster, but Debian testing, which is not what we want (and not a good idea for reproducible builds as this distribution is changing a lot all the time). From the error it looks like
openjdk-8-jreandopenjdk-8-jdk-headlessneed to be added to the pinning configuration too.I do see that without pinning we still use debian stable buster for a bunch of libraries (the first 54). I haven't been able to find a specification of how it does its resolving in this case (so I guess pinning is designed to force certain rules).
{{{ Get:50 http://deb.debian.org/debian buster/main amd64 libxdmcp-dev amd64 1:1.1.2-3 [42.2 kB] Get:51 http://deb.debian.org/debian buster/main amd64 xtrans-dev all 1.3.5-1 [100 kB] Get:52 http://deb.debian.org/debian buster/main amd64 libxcb1-dev amd64 1.13.1-2 [174 kB] Get:53 http://deb.debian.org/debian buster/main amd64 libxt-dev amd64 1:1.1.5-1+b3 [426 kB] Get:54 http://deb.debian.org/debian buster/main amd64 xdg-user-dirs amd64 0.17-2 [53.8 kB] Get:55 http://ftp.us.debian.org/debian unstable/main amd64 libglib2.0-0 amd64 2.62.1-1 [1317 kB] Get:56 http://ftp.us.debian.org/debian unstable/main amd64 libwebp6 amd64 0.6.1-2+b1 [261 kB] .... Get:114 http://ftp.us.debian.org/debian unstable/main amd64 libx11-dev amd64 2:1.6.8-1 [822 kB] Get:115 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jre-headless amd64 8u232-b09-1 [27.2 MB] Get:116 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jre amd64 8u232-b09-1 [69.7 kB] Get:117 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jdk-headless amd64 8u232-b09-1 [8183 kB] Get:118 http://ftp.us.debian.org/debian unstable/main amd64 openjdk-8-jdk amd64 8u232-b09-1 [1603 kB]
}}}
Those packages are exactly the same version in buster and in unstable.
-
Replying to sisbell:
I try to install ca-certificates but still get the same error regarding Certificate failed. I'll need to investigate more. Its installing from the following location
{{{ Get:3 http://archive.ubuntu.com/ubuntu eoan/main amd64 ca-certificates all 20190110 [146 kB] }}}
It needs to be installed in the debian buster container as this is where the apt using snapshot.debian.org is run, not the ubuntu one which we only use to run debootstrap.
Replying to boklm:
Replying to sisbell:
I try to install ca-certificates but still get the same error regarding Certificate failed. I'll need to investigate more. Its installing from the following location
Get:3 http://archive.ubuntu.com/ubuntu eoan/main amd64 ca-certificates all 20190110 [146 kB] }}}It needs to be installed in the debian buster container as this is where the apt using snapshot.debian.org is run, not the ubuntu one which we only use to run debootstrap.
Ok so the installation of ca-certificates works from container image. I'm using a specific version of the snapshot, which I think we would want to do to keep the packages pegged to a single snapshot.
{{{ https://snapshot.debian.org/archive/debian/20191022T222828Z/
Apt pinning also is working. I still get missing dependencies like below so I'm assuming I need to specify the entire dependency chain (and versions) here to get openjdk-8 installed? Or is there a simpler way?The following packages have unmet dependencies: openjdk-8-jre-headless : Depends: libc6 (>= 2.29) but 2.28-10 is to be installed E: Unable to correct problems, you have held broken packages.
-
Replying to sisbell:
I still get missing dependencies like below so I'm assuming I need to specify the entire dependency chain (and versions) here to get openjdk-8 installed? Or is there a simpler way?
{{{ The following packages have unmet dependencies: openjdk-8-jre-headless : Depends: libc6 (>= 2.29) but 2.28-10 is to be installed E: Unable to correct problems, you have held broken packages. }}}
It looks like openjdk-8-jre-headless version
8u222-b10-1from September still depends on libc6 2.28, so we could use that version instead to avoid updating the libc. When I try to apply an older snapshot, it says expired. So I'm wondering if specifying a specific snapshot release is really the the way to go (and if it isn't ,the snapshots and dependencies will change).
Running hooks in /etc/ca-certificates/update.d... done. Hit:1 http://deb.debian.org/debian buster InRelease Get:2 http://deb.debian.org/debian buster/main Translation-en [5967 kB] Get:3 https://snapshot.debian.org/archive/debian/20190930T204642Z unstable InRelease [139 kB] Reading package lists... E: Release file for https://snapshot.debian.org/archive/debian/20190930T204642Z/dists/unstable/InRelease is expired (invalid since 15d 22h 9min 17s). Updates for this repository will not be applied.My pinning specifies release. I'll look into if this is an issue with snapshots /InRelease expiration
Package: openjdk-8-jdk Pin: origin snapshot.debian.org Pin: release a=unstable Pin-Priority: 600I can get the following installed correctly in debootstrap-image config. I can tell this from the logs. But java doesn't show up in the container image when building gradle projects.
apt-get install -y -q ca-certificates-java java-common libcups2 liblcms2-2 libjpeg8 libfontconfig1 libnss3 libfreetype6 libpcsclite1 libx11-6 libxext6 libxi6 libxrender1 libxtst6 libglib2.0-0 libxrandr2 libxinerama1 libgl1-mesa-glx libgtk2.0-0 libatk-wrapper-java-jni libgif7 libpulse0 isc-dhcp-common dpkg -i ./openjdk-8-jre-headless.deb dpkg -i ./openjdk-8-jdk-headless.deb dpkg -i ./openjdk-8-jre.deb dpkg -i ./openjdk-8-jdk.deb- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jre-headless_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jre-headless.deb - URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jre_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jre.deb - URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jdk_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jdk.deb - URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jdk-headless_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jdk-headless.debI tried moving everything above to container-image config but the build fails. I'm not sure why there would be a difference.
E: Unable to locate package libjpeg8-
Replying to sisbell:
I can get the following installed correctly in debootstrap-image config. I can tell this from the logs. But java doesn't show up in the container image when building gradle projects.
{{{ apt-get install -y -q ca-certificates-java java-common libcups2 liblcms2-2 libjpeg8 libfontconfig1 libnss3 libfreetype6 libpcsclite1 libx11-6 libxext6 libxi6 libxrender1 libxtst6 libglib2.0-0 libxrandr2 libxinerama1 libgl1-mesa-glx libgtk2.0-0 libatk-wrapper-java-jni libgif7 libpulse0 isc-dhcp-common dpkg -i ./openjdk-8-jre-headless.deb dpkg -i ./openjdk-8-jdk-headless.deb dpkg -i ./openjdk-8-jre.deb dpkg -i ./openjdk-8-jdk.deb }}}
{{{
- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jre-headless_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jre-headless.deb
- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jre_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jre.deb
- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jdk_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jdk.deb
- URL: http://archive.ubuntu.com/ubuntu/pool/universe/o/openjdk-8/openjdk-8-jdk-headless_8u232-b09-0ubuntu1_amd64.deb filename: openjdk-8-jdk-headless.deb }}}
Why are you using Ubuntu packages? We want to use Debian buster, so Ubuntu package is not what we should use.
And the Ubuntu container is only used to run
debootstrapto create the Debian tarballs, so we don't need openjdk there.I tried moving everything above to container-image config but the build fails. I'm not sure why there would be a difference.
{{{ E: Unable to locate package libjpeg8 }}}
Difficult to say without seeing actual patches (I have no idea where you added those lines in the
debootstrap-imageandcontainer-imageconfigs), but maybe you installed the packages in the Ubuntu container in the first case, and in the buster container in the second.But anyway, as I said in comment:2 and comment:5 we should not do this in
debootstrap-imageorcontainer-image, but inrbm.confinvar/pre_pkginst(or maybevar/post_pkginst). And as we don't want to copy the URLs in theinput_filesof all projects, we should download them invar/pre_pkginst(orvar/post_pkginst) using wget (followed by a sha256sum verification).