Ubuntu 10.04 LTS is EOL April 2015
Ubuntu 10.04 turns 5 this month. This means that it will officially be end-of-life. We need to decide what this means for our gitian guest VMs for Linux. I think if we transition the Linux builders to 12.04, we risk incompatibility with Debian/oldstable, Debian/stable and Centos 5.x, due primarily to a slightly newer glibc in 12.04 than those systems (12.04 uses glibc 2.15, where as Debian/stable went with glibc 2.13-and-change).
I think my preferred option long-term is to switch us to Debian guest VM images, but it looks like the only way to do that is to also switch to using a sketch python-vm-builder fork completed as a Summer of Code project way back in 2009: https://wiki.debian.org/VMBuilder
The Bitcoin people must be dealing with this issue as well. We should see if we can find out what they are doing.
This is probably also something we should postpone until after 4.5 is out, due to the potential build incompatibility and build system stability issues.
Activity
So, I had been looking a bit at switching to Ubuntu 12.04 (#15551 (moved)) as the new Debian stable is supposed to come out shortly.
What the Bitcoin people are doing is basically what I did for #10125 (moved) (getting Debian as a host OS going) + they use 64 bit Ubuntu Precise: https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-descriptors/gitian-linux.yml. Here is what they did/do with respect to older glibc versions: https://git.thwg.org/wozz/bitcoin/commit/49a3352c1c05319c5fa22c4cb6819371d71de74d, https://git.thwg.org/wozz/bitcoin/commit/d5aab704908d6bea48bd7e7b36a3a49ece7d5c5f, https://git.thwg.org/wozz/bitcoin/commit/ffc6b678b9d12a8f963ab362b952fd6b7e6c4ec7.
Sounds not like anything we could easily do. Switching to Debian guest VMs does not sound trivial either. I wonder how many user are really affected if we switched to Ubuntu Precise after Debian Jessy gets out. I'd be inclined to argue Debian old-stable users (i.e. people using Wheezy then) should upgrade to stable as I can't imagine we want to start switching to old stable for building our Linux packages (not talking about the ones for OS X and Windows). Hmm...
Trac:
Cc: N/A to gk
Keywords: gitian deleted, tbb-gitian added
Replying to mikeperry:
I think my preferred option long-term is to switch us to Debian guest VM images, but it looks like the only way to do that is to also switch to using a sketch python-vm-builder fork completed as a Summer of Code project way back in 2009: https://wiki.debian.org/VMBuilder
I don't see any RFP (request for package) asking to get python-vm-builder in Debian on https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=wnpp. Of course, just asking likely won't be enough, but if we don't even ask, certainly it won't ever happen :) Shall I file a RFP bug there?
Also, it seems that vmbuilder does something very similar to live-build, so it might be worth looking into it.
Replying to gk:
I'd be inclined to argue Debian old-stable users (i.e. people using Wheezy then) should upgrade to stable
Note that Debian oldstable is supported for a full year after a new stable is out. So users should, and will upgrade, sure, but not necessarily right now: that's what the transition period is for, and I doubt we can force users to go through the upgrade exactly when we want it. If we try to force it, I'm afraid that the end-result, for many Debian users, would simply be "what?! Tor Browser is broken", which isn't very nice UX, even with awesome blog posts explaining them (in English) why they should upgrade their OS.
as I can't imagine we want to start switching to old stable for building our Linux packages
Given Wheezy is pretty solid a release, supported for a full more year (and probably longer, as it seems likely that the LTS effort will apply to Wheezy too), this wouldn't seem that crazy an option to me.
It at least would do the first migration step, away from an unsupported Ubuntu release, to using Debian for the build VMs. And in a year, once Wheezy is officially deprecated (the LTS effort is mostly focussed on server use cases in practice), then we can definitely stop supporting Wheezy users, and then we can use Jessie VMs to build Tor Browser.
Replying to mikeperry:
Unfortunately, I think the Tails people will not be able to use our builds if we switch to 12.04 before they switch to Jessy.
Indeed, that's (unfortunately) very likely. Thanks for keeping the Tails usecase in mind!
But perhaps it would be good to check before giving up, if it's cheap: Wheezy has glibc 2.13, Precise has 2.15 -- the delta perhaps isn't that big => maybe there's no new symbol introduced in 2.13..2.15 that Tor Browser builds would pick up? It might be, in the end, that it's only about one symbol or two, and then perhaps a thin compatibility layer hack, like the bitcoin folks have done, could be an acceptable workaround for a year?
(BTW, I'm pretty sure we'll release Tails based on Jessie by the end of the year, but I doubt it'll happen before November.)
-
Replying to intrigeri:
Replying to mikeperry:
I think my preferred option long-term is to switch us to Debian guest VM images, but it looks like the only way to do that is to also switch to using a sketch python-vm-builder fork completed as a Summer of Code project way back in 2009: https://wiki.debian.org/VMBuilder
I don't see any RFP (request for package) asking to get python-vm-builder in Debian on https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=wnpp. Of course, just asking likely won't be enough, but if we don't even ask, certainly it won't ever happen :) Shall I file a RFP bug there?
Also, it seems that vmbuilder does something very similar to live-build, so it might be worth looking into it.
Thanks for the pointer. I think filing an RFP sounds like a good idea if we go with VMBuilder after we patched it in a way to support up-to-date Debian systems (we'll see if we do that).
Replying to gk:
I'd be inclined to argue Debian old-stable users (i.e. people using Wheezy then) should upgrade to stable
Note that Debian oldstable is supported for a full year after a new stable is out. So users should, and will upgrade, sure, but not necessarily right now: that's what the transition period is for, and I doubt we can force users to go through the upgrade exactly when we want it. If we try to force it, I'm afraid that the end-result, for many Debian users, would simply be "what?! Tor Browser is broken", which isn't very nice UX, even with awesome blog posts explaining them (in English) why they should upgrade their OS.
Good point and I agree with you.
as I can't imagine we want to start switching to old stable for building our Linux packages
Given Wheezy is pretty solid a release, supported for a full more year (and probably longer, as it seems likely that the LTS effort will apply to Wheezy too), this wouldn't seem that crazy an option to me.
It at least would do the first migration step, away from an unsupported Ubuntu release, to using Debian for the build VMs. And in a year, once Wheezy is officially deprecated (the LTS effort is mostly focussed on server use cases in practice), then we can definitely stop supporting Wheezy users, and then we can use Jessie VMs to build Tor Browser.
Maybe, but note that Ubuntu Precise will be supported almost until end of 2017 which might cause similar glibc issues if we want to support that long-term Ubuntu distro while pondering switching to Jessie.
Replying to mikeperry:
Unfortunately, I think the Tails people will not be able to use our builds if we switch to 12.04 before they switch to Jessy.
Indeed, that's (unfortunately) very likely. Thanks for keeping the Tails usecase in mind!
But perhaps it would be good to check before giving up, if it's cheap: Wheezy has glibc 2.13, Precise has 2.15 -- the delta perhaps isn't that big => maybe there's no new symbol introduced in 2.13..2.15 that Tor Browser builds would pick up? It might be, in the end, that it's only about one symbol or two, and then perhaps a thin compatibility layer hack, like the bitcoin folks have done, could be an acceptable workaround for a year?
Having Lucid as the OS for the guest VM is exactly due to glibc issues we encountered in the past (a thing the Tor Messenger people had to learn as well). We could check if it is really just one or two symbols but I am not so happy about that idea given our numerous Tor Browser parts and the already fragile build system.
(BTW, I'm pretty sure we'll release Tails based on Jessie by the end of the year, but I doubt it'll happen before November.)
So, sticking to Ubuntu Precise a la #15551 (moved) is no option then. The deterministic build system does already have enough kludges and I am not too eager to add another one in the Bitcoin style. Seems Mike was right and we should try to get Gitian working with Debian guest VMs. Exciting.
-
Another option might be to use whatever OS you prefer (e.g. Debian Jessie) for the build VM, and have the build system download the oldest glibc we want to support somewhere (
apt-get source eglibc/oldstable), and export whatever environment variables are needed so that everything is built against that older libc instead of the system one. No idea how it's doable in practice, you'll know better :) -
Replying to gk:
So, sticking to Ubuntu Precise a la #15551 (moved) is no option then. The deterministic build system does already have enough kludges and I am not too eager to add another one in the Bitcoin style. Seems Mike was right and we should try to get Gitian working with Debian guest VMs. Exciting.
Hrm... Maybe I was a bit fast here. Having the switch to Ubuntu Precise as a stop-gap might be working and be worthwhile given that 10.04 is already EOL in two weeks and switching to Debian might not be ready in time. If I take my branch posted in #14730 (moved) and make sure I compile everything without
FORTIFY_SOURCEthen Tor Browser seems already to be working in Tails. The__fdelt_chkfunction is the only one that is problematic. Quite tempting and might be doable to get fixed properly within the next two/three weeks. -
I spent a great deal of my time today investigating this closer and think we should not go the Bitcoin way as we both want to have Debian guest VMs in the long run and I fear this contains some stability risk given that we have much more moving parts not under our control that are (possibly) affected. I think a sane thing to do is trying to get the switch to Debian implemented for 5.0. It might be ambitious due to the unknowns which are always involved when we switch to a new Firefox ESR but I am optimistic.
If that sounds like a proper plan then we have two options for the next one/two releases: We keep Lucid VMs for Linux or we basically switch to the branch I worked on for #15551 (moved) which would give us Precise guest VMs for Linux as well but with disabled
FORTIFY_SOURCE. I tend to prefer the former as I fear compatibility risks even in this scenario. While I am pretty sure the bundles would run on Tails systems we'd need some more testing (especially on older 64bit systems) to be sure there are no other issues. And this is not mentioning general yet-to-be-enountered problems following a toolchain switch. 
The other day I happened to start working on adding a Debian plugin to vmbuilder with the intention of giving people the option to use Debian guests with Gitian instead of Ubuntu guests. It seems like this plugin I started on is somewhat of relevance to the discussion in this issue.
It is still very much a work in progress, but I thought you might like to have this on your radar. I was able to reproduce Sqlite for amd64 Linux successfully using a Debian Jessie KVM image with a modified version of gitian-builder (to change some package names and the username from ubuntu to debian, for example). I haven't published the gitian-builder changes, because they need some changes to work generally with both Ubuntu and Debian (e.g. there should probably be a command line option to select between Debian and Ubuntu). The only outstanding issue with my particular test situation is that the fetching of the package manifest doesn't work, so it has to be disabled currently for the build to proceed.
See: https://github.com/josephbisch/vmbuilder.
As you can see, I haven't put much effort yet into getting the particular differences between the Debian suites recorded in the Python files for each suite. I haven't talked to vmbuilder upstream yet about getting the Debian plugin integrated into vmbuilder proper, but I image they would want everything pretty much functional before they would merge it and there is a lot of stuff that isn't strictly necessary to get vmbuilder working with Gitian (like EC2 images). I am in discussions with the gitian-builder maintainer about adding Debian guest support to gitian-builder.
There is also the issue of LXC Debian guests, which I haven't looked at yet, but gitian-builder just seems to use debootstrap, so it should be easy to get LXC Debian guests working.
Trac:
Username: josephbisch
Sponsor: N/A to N/A
