Opened 5 years ago

Closed 4 years ago

#15578 closed defect (fixed)

Ubuntu 10.04 LTS is EOL April 2015

Reported by: mikeperry Owned by: boklm
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-gitian, AffectsTails, TorBrowserTeam201601R
Cc: gk, mcs, intrigeri Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Ubuntu 10.04 turns 5 this month. This means that it will officially be end-of-life. We need to decide what this means for our gitian guest VMs for Linux. I think if we transition the Linux builders to 12.04, we risk incompatibility with Debian/oldstable, Debian/stable and Centos 5.x, due primarily to a slightly newer glibc in 12.04 than those systems (12.04 uses glibc 2.15, where as Debian/stable went with glibc 2.13-and-change).

I think my preferred option long-term is to switch us to Debian guest VM images, but it looks like the only way to do that is to also switch to using a sketch python-vm-builder fork completed as a Summer of Code project way back in 2009: https://wiki.debian.org/VMBuilder

The Bitcoin people must be dealing with this issue as well. We should see if we can find out what they are doing.

This is probably also something we should postpone until after 4.5 is out, due to the potential build incompatibility and build system stability issues.

Child Tickets

Change History (62)

comment:1 Changed 5 years ago by gk

Cc: gk added
Keywords: tbb-gitian added; gitian removed

So, I had been looking a bit at switching to Ubuntu 12.04 (#15551) as the new Debian stable is supposed to come out shortly.

What the Bitcoin people are doing is basically what I did for #10125 (getting Debian as a host OS going) + they use 64 bit Ubuntu Precise: https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-descriptors/gitian-linux.yml. Here is what they did/do with respect to older glibc versions: https://git.thwg.org/wozz/bitcoin/commit/49a3352c1c05319c5fa22c4cb6819371d71de74d, https://git.thwg.org/wozz/bitcoin/commit/d5aab704908d6bea48bd7e7b36a3a49ece7d5c5f, https://git.thwg.org/wozz/bitcoin/commit/ffc6b678b9d12a8f963ab362b952fd6b7e6c4ec7.

Sounds not like anything we could easily do. Switching to Debian guest VMs does not sound trivial either. I wonder how many user are really affected if we switched to Ubuntu Precise after Debian Jessy gets out. I'd be inclined to argue Debian old-stable users (i.e. people using Wheezy then) should upgrade to stable as I can't imagine we want to start switching to old stable for building our Linux packages (not talking about the ones for OS X and Windows). Hmm...

Last edited 5 years ago by gk (previous) (diff)

comment:2 Changed 5 years ago by mcs

Cc: mcs added

comment:3 Changed 5 years ago by mikeperry

Cc: intrigeri added
Keywords: AffectsTails added

Unfortunately, I think the Tails people will not be able to use our builds if we switch to 12.04 before they switch to Jessy. That may not happen at the same time as Debian/Jessy is released.

comment:4 in reply to:  3 ; Changed 5 years ago by intrigeri

Replying to mikeperry:

I think my preferred option long-term is to switch us to Debian guest VM images, but it looks like the only way to do that is to also switch to using a sketch python-vm-builder fork completed as a Summer of Code project way back in 2009: https://wiki.debian.org/VMBuilder

I don't see any RFP (request for package) asking to get python-vm-builder in Debian on https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=wnpp. Of course, just asking likely won't be enough, but if we don't even ask, certainly it won't ever happen :) Shall I file a RFP bug there?

Also, it seems that vmbuilder does something very similar to live-build, so it might be worth looking into it.

Replying to gk:

I'd be inclined to argue Debian old-stable users (i.e. people using Wheezy then) should upgrade to stable

Note that Debian oldstable is supported for a full year after a new stable is out. So users should, and will upgrade, sure, but not necessarily right now: that's what the transition period is for, and I doubt we can force users to go through the upgrade exactly when we want it. If we try to force it, I'm afraid that the end-result, for many Debian users, would simply be "what?! Tor Browser is broken", which isn't very nice UX, even with awesome blog posts explaining them (in English) why they should upgrade their OS.

as I can't imagine we want to start switching to old stable for building our Linux packages

Given Wheezy is pretty solid a release, supported for a full more year (and probably longer, as it seems likely that the LTS effort will apply to Wheezy too), this wouldn't seem _that_ crazy an option to me.

It at least would do the first migration step, away from an unsupported Ubuntu release, to using Debian for the build VMs. And in a year, once Wheezy is officially deprecated (the LTS effort is mostly focussed on server use cases in practice), then we can definitely stop supporting Wheezy users, and then we can use Jessie VMs to build Tor Browser.

Replying to mikeperry:

Unfortunately, I think the Tails people will not be able to use our builds if we switch to 12.04 before they switch to Jessy.

Indeed, that's (unfortunately) very likely. Thanks for keeping the Tails usecase in mind!

But perhaps it would be good to check before giving up, if it's cheap: Wheezy has glibc 2.13, Precise has 2.15 -- the delta perhaps isn't that big => maybe there's no new symbol introduced in 2.13..2.15 that Tor Browser builds would pick up? It might be, in the end, that it's only about one symbol or two, and then perhaps a thin compatibility layer hack, like the bitcoin folks have done, could be an acceptable workaround for a year?

(BTW, I'm pretty sure we'll release Tails based on Jessie by the end of the year, but I doubt it'll happen before November.)

comment:5 in reply to:  4 ; Changed 5 years ago by gk

Replying to intrigeri:

Replying to mikeperry:

I think my preferred option long-term is to switch us to Debian guest VM images, but it looks like the only way to do that is to also switch to using a sketch python-vm-builder fork completed as a Summer of Code project way back in 2009: https://wiki.debian.org/VMBuilder

I don't see any RFP (request for package) asking to get python-vm-builder in Debian on https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=wnpp. Of course, just asking likely won't be enough, but if we don't even ask, certainly it won't ever happen :) Shall I file a RFP bug there?

Also, it seems that vmbuilder does something very similar to live-build, so it might be worth looking into it.

Thanks for the pointer. I think filing an RFP sounds like a good idea if we go with VMBuilder after we patched it in a way to support up-to-date Debian systems (we'll see if we do that).

Replying to gk:

I'd be inclined to argue Debian old-stable users (i.e. people using Wheezy then) should upgrade to stable

Note that Debian oldstable is supported for a full year after a new stable is out. So users should, and will upgrade, sure, but not necessarily right now: that's what the transition period is for, and I doubt we can force users to go through the upgrade exactly when we want it. If we try to force it, I'm afraid that the end-result, for many Debian users, would simply be "what?! Tor Browser is broken", which isn't very nice UX, even with awesome blog posts explaining them (in English) why they should upgrade their OS.

Good point and I agree with you.

as I can't imagine we want to start switching to old stable for building our Linux packages

Given Wheezy is pretty solid a release, supported for a full more year (and probably longer, as it seems likely that the LTS effort will apply to Wheezy too), this wouldn't seem _that_ crazy an option to me.

It at least would do the first migration step, away from an unsupported Ubuntu release, to using Debian for the build VMs. And in a year, once Wheezy is officially deprecated (the LTS effort is mostly focussed on server use cases in practice), then we can definitely stop supporting Wheezy users, and then we can use Jessie VMs to build Tor Browser.

Maybe, but note that Ubuntu Precise will be supported almost until end of 2017 which might cause similar glibc issues if we want to support that long-term Ubuntu distro while pondering switching to Jessie.

Replying to mikeperry:

Unfortunately, I think the Tails people will not be able to use our builds if we switch to 12.04 before they switch to Jessy.

Indeed, that's (unfortunately) very likely. Thanks for keeping the Tails usecase in mind!

But perhaps it would be good to check before giving up, if it's cheap: Wheezy has glibc 2.13, Precise has 2.15 -- the delta perhaps isn't that big => maybe there's no new symbol introduced in 2.13..2.15 that Tor Browser builds would pick up? It might be, in the end, that it's only about one symbol or two, and then perhaps a thin compatibility layer hack, like the bitcoin folks have done, could be an acceptable workaround for a year?

Having Lucid as the OS for the guest VM is exactly due to glibc issues we encountered in the past (a thing the Tor Messenger people had to learn as well). We could check if it is really just one or two symbols but I am not so happy about that idea given our numerous Tor Browser parts and the already fragile build system.

(BTW, I'm pretty sure we'll release Tails based on Jessie by the end of the year, but I doubt it'll happen before November.)

So, sticking to Ubuntu Precise a la #15551 is no option then. The deterministic build system does already have enough kludges and I am not too eager to add another one in the Bitcoin style. Seems Mike was right and we should try to get Gitian working with Debian guest VMs. Exciting.

comment:6 Changed 5 years ago by intrigeri

Another option might be to use whatever OS you prefer (e.g. Debian Jessie) for the build VM, and have the build system download the oldest glibc we want to support somewhere (apt-get source eglibc/oldstable), and export whatever environment variables are needed so that everything is built against that older libc instead of the system one. No idea how it's doable in practice, you'll know better :)

comment:7 in reply to:  5 Changed 5 years ago by gk

Replying to gk:

So, sticking to Ubuntu Precise a la #15551 is no option then. The deterministic build system does already have enough kludges and I am not too eager to add another one in the Bitcoin style. Seems Mike was right and we should try to get Gitian working with Debian guest VMs. Exciting.

Hrm... Maybe I was a bit fast here. Having the switch to Ubuntu Precise as a stop-gap might be working and be worthwhile given that 10.04 is already EOL in two weeks and switching to Debian might not be ready in time. If I take my branch posted in #14730 and make sure I compile everything without FORTIFY_SOURCE then Tor Browser seems already to be working in Tails. The __fdelt_chk function is the only one that is problematic. Quite tempting and might be doable to get fixed properly within the next two/three weeks.

comment:8 Changed 5 years ago by gk

I spent a great deal of my time today investigating this closer and think we should not go the Bitcoin way as we both want to have Debian guest VMs in the long run and I fear this contains some stability risk given that we have much more moving parts not under our control that are (possibly) affected. I think a sane thing to do is trying to get the switch to Debian implemented for 5.0. It might be ambitious due to the unknowns which are always involved when we switch to a new Firefox ESR but I am optimistic.

If that sounds like a proper plan then we have two options for the next one/two releases: We keep Lucid VMs for Linux or we basically switch to the branch I worked on for #15551 which would give us Precise guest VMs for Linux as well but with disabled FORTIFY_SOURCE. I tend to prefer the former as I fear compatibility risks even in this scenario. While I am pretty sure the bundles would run on Tails systems we'd need some more testing (especially on older 64bit systems) to be sure there are no other issues. And this is not mentioning general yet-to-be-enountered problems following a toolchain switch.

comment:9 Changed 5 years ago by mikeperry

Keywords: TorBrowserTeam201505 added; TorBrowserTeam201504 removed

comment:10 Changed 4 years ago by gk

Keywords: TorBrowserTeam201506 GeorgKoppen201506 added; TorBrowserTeam201505 removed

comment:11 Changed 4 years ago by gk

Owner: changed from tbb-team to gk
Status: newassigned

comment:12 Changed 4 years ago by gk

Keywords: TorBrowserTeam201507 GeorgKoppen201507 added; TorBrowserTeam201506 GeorgKoppen201506 removed

comment:13 Changed 4 years ago by gk

Keywords: TorBrowserTeam201508 GeorgKoppen201508 added; TorBrowserTeam201507 GeorgKoppen201507 removed

comment:14 Changed 4 years ago by mikeperry

Keywords: TorBrowserTeam201509 added; TorBrowserTeam201508 removed

Move remaining August tickets to September.

comment:15 Changed 4 years ago by gk

Keywords: GeorgKoppen201509 added; GeorgKoppen201508 removed

comment:16 Changed 4 years ago by gk

Keywords: TorBrowserTeam201510 added; TorBrowserTeam201509 removed

Moving Tor Browser tickets to October 2015.

comment:17 Changed 4 years ago by gk

Keywords: GeorgKoppen201510 added; GeorgKoppen201509 removed

Moving my tickets to October 2015

comment:18 Changed 4 years ago by josephbisch

The other day I happened to start working on adding a Debian plugin to vmbuilder with the intention of giving people the option to use Debian guests with Gitian instead of Ubuntu guests. It seems like this plugin I started on is somewhat of relevance to the discussion in this issue.

It is still very much a work in progress, but I thought you might like to have this on your radar. I was able to reproduce Sqlite for amd64 Linux successfully using a Debian Jessie KVM image with a modified version of gitian-builder (to change some package names and the username from ubuntu to debian, for example). I haven't published the gitian-builder changes, because they need some changes to work generally with both Ubuntu and Debian (e.g. there should probably be a command line option to select between Debian and Ubuntu). The only outstanding issue with my particular test situation is that the fetching of the package manifest doesn't work, so it has to be disabled currently for the build to proceed.

See: https://github.com/josephbisch/vmbuilder.

As you can see, I haven't put much effort yet into getting the particular differences between the Debian suites recorded in the Python files for each suite. I haven't talked to vmbuilder upstream yet about getting the Debian plugin integrated into vmbuilder proper, but I image they would want everything pretty much functional before they would merge it and there is a lot of stuff that isn't strictly necessary to get vmbuilder working with Gitian (like EC2 images). I am in discussions with the gitian-builder maintainer about adding Debian guest support to gitian-builder.

There is also the issue of LXC Debian guests, which I haven't looked at yet, but gitian-builder just seems to use debootstrap, so it should be easy to get LXC Debian guests working.

comment:19 Changed 4 years ago by gk

Keywords: TorBrowserTeam201511 added; TorBrowserTeam201510 removed

comment:20 Changed 4 years ago by gk

Keywords: GeorgKoppen201511 added; GeorgKoppen201510 removed

comment:21 in reply to:  18 ; Changed 4 years ago by gk

Severity: Normal

Replying to josephbisch:

The other day I happened to start working on adding a Debian plugin to vmbuilder with the intention of giving people the option to use Debian guests with Gitian instead of Ubuntu guests. It seems like this plugin I started on is somewhat of relevance to the discussion in this issue.

It is still very much a work in progress, but I thought you might like to have this on your radar. I was able to reproduce Sqlite for amd64 Linux successfully using a Debian Jessie KVM image with a modified version of gitian-builder (to change some package names and the username from ubuntu to debian, for example). I haven't published the gitian-builder changes, because they need some changes to work generally with both Ubuntu and Debian (e.g. there should probably be a command line option to select between Debian and Ubuntu). The only outstanding issue with my particular test situation is that the fetching of the package manifest doesn't work, so it has to be disabled currently for the build to proceed.

See: https://github.com/josephbisch/vmbuilder.

As you can see, I haven't put much effort yet into getting the particular differences between the Debian suites recorded in the Python files for each suite. I haven't talked to vmbuilder upstream yet about getting the Debian plugin integrated into vmbuilder proper, but I image they would want everything pretty much functional before they would merge it and there is a lot of stuff that isn't strictly necessary to get vmbuilder working with Gitian (like EC2 images). I am in discussions with the gitian-builder maintainer about adding Debian guest support to gitian-builder.

There is also the issue of LXC Debian guests, which I haven't looked at yet, but gitian-builder just seems to use debootstrap, so it should be easy to get LXC Debian guests working.

Thanks. I saw that your changes got meanwhile merged into gitian-builder master, great! I started testing your code. It seems I can create neither create Wheezy nor Jessie VMs. I tried both my backported patch and gitian-builder master and both fail for Jessie (that seems the most tested variant which I concentrated on it first to test the whole process).

I tried to create a container with ./bin/make-base-vm --distro debian --suite jessie --arch i386 and as an error I get "Unable to locate package linux-image-i686-pae" I used master your vmbuilder repository. I wonder, though, why this is failing for me given that it is supposed to be working for you. Hrm...

comment:22 in reply to:  21 ; Changed 4 years ago by gk

Replying to gk:

I tried to create a container with ./bin/make-base-vm --distro debian --suite jessie --arch i386 and as an error I get "Unable to locate package linux-image-i686-pae" I used master your vmbuilder repository. I wonder, though, why this is failing for me given that it is supposed to be working for you. Hrm...

This is a gitian-builder bug. It should be FLAVOUR=686-pae instead of FLAVOUR=i686-pae.

comment:23 in reply to:  22 ; Changed 4 years ago by josephbisch

Replying to gk:

Replying to gk:

I tried to create a container with ./bin/make-base-vm --distro debian --suite jessie --arch i386 and as an error I get "Unable to locate package linux-image-i686-pae" I used master your vmbuilder repository. I wonder, though, why this is failing for me given that it is supposed to be working for you. Hrm...

This is a gitian-builder bug. It should be FLAVOUR=686-pae instead of FLAVOUR=i686-pae.

Thanks Georg. I have submitted a PR to gitian-builder to fix the flavour. I also noticed that it should just be 686 instead of 686-pae for Squeeze and earlier, so I also fixed that. I also worked around an apt-get bug in Squeeze and earlier.

There are still some issues on the vmbuilder side that need to be resolved to get Squeeze working. I tried getting Squeeze working, but I am currently at the following bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560417. I need to read through that bug and see how to resolve it. I should have the time this weekend if not sooner to fix Squeeze.

Is Squeeze what you ultimately want to go with?

comment:24 in reply to:  23 Changed 4 years ago by gk

Replying to josephbisch:

Replying to gk:

Replying to gk:

I tried to create a container with ./bin/make-base-vm --distro debian --suite jessie --arch i386 and as an error I get "Unable to locate package linux-image-i686-pae" I used master your vmbuilder repository. I wonder, though, why this is failing for me given that it is supposed to be working for you. Hrm...

This is a gitian-builder bug. It should be FLAVOUR=686-pae instead of FLAVOUR=i686-pae.

Thanks Georg. I have submitted a PR to gitian-builder to fix the flavour. I also noticed that it should just be 686 instead of 686-pae for Squeeze and earlier, so I also fixed that. I also worked around an apt-get bug in Squeeze and earlier.

Thanks.

There are still some issues on the vmbuilder side that need to be resolved to get Squeeze working. I tried getting Squeeze working, but I am currently at the following bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560417. I need to read through that bug and see how to resolve it. I should have the time this weekend if not sooner to fix Squeeze.

Is Squeeze what you ultimately want to go with?

Nah, we don't need it that old. :) We try to get Gitian running with Wheezy. What I am currently hitting is:

2015-11-12 12:45:49,708 INFO    : terminate called after throwing an instance of 'std::out_of_range'
2015-11-12 12:45:49,708 INFO    :   what():  basic_string::compare

when it is trying to do apt-get,install,-y,--force-yes,pciutils,build-essential,git-core,subversion,locales,wget,lsb-release,,grub-pc,openssh-server,cron-

comment:25 Changed 4 years ago by gk

Applying 4f69707c4df4224506a64b760cbd4fd14eb83c8f fixes this for me and I get Wheezy images \o/. In case the feedback on IRC got lost (wrt your fixups) here it is verbatim again:

08:24 < GeKo> re: fe1abd115324d2e706626ed567d1f234fd300e64 i think putting the 686 
              distros in the first elif clause is better
08:24 < GeKo> otherwise you'll always have to update that part if a new debian 
              distro is coming out (assuming they stick to "686-pae" in the future)
08:26 < GeKo> re: 4f69707c4df4224506a64b760cbd4fd14eb83c8f your if caluse is about 
              ubuntu but the comment in it is talking about a debian issue <= 
              squeeze which is confusing
08:27 < GeKo> why does that specific debian issue is affecting the people if they 
              choose to have ubuntu vms one might ask?

comment:26 Changed 4 years ago by gk

So, the first thing that breaks for me when starting the build results in

./bin/gbuild:213:in `[]=': can't convert Array into String (TypeError)
	from ./bin/gbuild:213:in `<main>'

The problem is that distro is an array: [debian] and ENV['DISTRO'] = distro does not like that. Not sure why you are not affected by this.

Working around this problem breaks for Wheezy in the Create package manifest step with

E: Unable to correct problems, you have held broken packages.

Using my Jessie containers seems to work fine then.

comment:27 in reply to:  26 ; Changed 4 years ago by josephbisch

I addressed your comments from IRC in the PR for gitian-builder.

Replying to gk:

So, the first thing that breaks for me when starting the build results in

./bin/gbuild:213:in `[]=': can't convert Array into String (TypeError)
	from ./bin/gbuild:213:in `<main>'

The problem is that distro is an array: [debian] and ENV['DISTRO'] = distro does not like that. Not sure why you are not affected by this.

I added the line "distro: "debian"" (without the outer quotation marks) to my descriptor and I don't encounter this issue. Are you treating it the same as suites, which would make it an array?

Working around this problem breaks for Wheezy in the Create package manifest step with

E: Unable to correct problems, you have held broken packages.

Using my Jessie containers seems to work fine then.

I just fixed the issue in the gitian-builder PR. There were broken packages due to conflicts between grub-pc and grub-legacy. The Debian wiki says to just install grub (it appears to install the correct version of grub for your suite), so I have make-base-vm doing that now for Debian instead of using grub-pc for both distros as gitian-builder uses for Ubuntu.

I am now able to make it through a Wheezy i386 build for a very very simple descriptor.

Last edited 4 years ago by josephbisch (previous) (diff)

comment:28 in reply to:  27 Changed 4 years ago by gk

Replying to josephbisch:

Replying to gk:

So, the first thing that breaks for me when starting the build results in

./bin/gbuild:213:in `[]=': can't convert Array into String (TypeError)
	from ./bin/gbuild:213:in `<main>'

The problem is that distro is an array: [debian] and ENV['DISTRO'] = distro does not like that. Not sure why you are not affected by this.

I added the line "distro: "debian"" (without the outer quotation marks) to my descriptor and I don't encounter this issue. Are you treating it the same as suites, which would make it an array?

Ah, yes, that was it, thanks!

Working around this problem breaks for Wheezy in the Create package manifest step with

E: Unable to correct problems, you have held broken packages.

Using my Jessie containers seems to work fine then.

I just fixed the issue in the gitian-builder PR. There were broken packages due to conflicts between grub-pc and grub-legacy. The Debian wiki says to just install grub (it appears to install the correct version of grub for your suite), so I have make-base-vm doing that now for Debian instead of using grub-pc for both distros as gitian-builder uses for Ubuntu.

I am now able to make it through a Wheezy i386 build for a very very simple descriptor.

Works for me, too, it seems (the build is now broken but that is due to the different toolchain shipped in Wheezy and thus fixable with our descriptors). I'll test things out during the coming days.

One thing I noticed is that you probably want to update the README file (again) as well. All those commands mentioning on-target etc. (basically all of them using TUSER) are broken now as TUSER is not ubuntu anymore but $DISTRO. Thus, something like export TUSER=ubuntu is needed now first to be able to log into Ubuntu VMs (before your changes this worked out of the box).

comment:29 Changed 4 years ago by gk

Alright, the new descriptors are ready (bug_15578_v2 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_15578_v2&id=367dcdcc5f174dac49baa0ae9982ce74f7428929)) and I needed to backport a Mozilla patch (bug_15578 (https://gitweb.torproject.org/user/gk/tor-browser.git/commit/?h=bug_15578&id=3b8008baa5aefd2602b5e14e79dcc29033006bf6)). Now, I need to properly backport the gitian-builder changes to our custom branch and update our instructions to use the enhanced vmbuilder.

comment:30 Changed 4 years ago by gk

josephbisch: Could you tag your vmbuilder changes and sign the tag with a GPG-key?

comment:31 Changed 4 years ago by josephbisch

I'm not sure if you saw, but my changes did get integrated into vmbuilder upstream. So maybe that serves your purpose? If you still need the tag and gpg sig, I can do that tomorrow (though I don't have many sigs on my key, so I'm not sure how useful doing so will be).

comment:32 in reply to:  31 Changed 4 years ago by gk

Replying to josephbisch:

I'm not sure if you saw, but my changes did get integrated into vmbuilder upstream. So maybe that serves your purpose? If you still need the tag and gpg sig, I can do that tomorrow (though I don't have

Yes, I saw it and forgot about it. That might do the trick, thanks. We are doing this already for Debian hosts as there is no python-vm-builder on Debian systems. A good way to detect older vmbuilder installations would be handy as well. But it seems the version number never gets bumped. So, hm.

comment:33 Changed 4 years ago by gk

Status: assignedneeds_review

Ok, this is finally up for review:

tor-browser: see comment:29
tor-browser-bundle: bug_15578_v4 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_15578_v4&id=89f27705959b3c70d29524d786cb53a11ba5d6f9)
gitian-builder: bug_15578_v2 (https://gitweb.torproject.org/user/gk/gitian-builder.git/log/?h=bug_15578_v2, the two most recent commits, basically being a backport from gitian master)

There is one caveat (at least), though: There is no LXC support yet for the Debian images (AFAICT) which breaks our LXC machine (again). This might not be a huge problem as a) We are not able yet to build Tor Browser reproducibly on all platofmrs (see: #12240 for the missing mit) and b) at least until these changes move to stable the latter can still be built and with a bit of energy nightlies/testing builds can be created as well. But still...

Last edited 4 years ago by gk (previous) (diff)

comment:34 Changed 4 years ago by gk

Keywords: TorBrowserTeam201511R added; TorBrowserTeam201511 removed

comment:35 Changed 4 years ago by josephbisch

Okay. I finally got around to adding a tag. It is debian-1.0 (to not confuse people with the versions for upstream vmbuilder). I signed it with the 4096-bit RSA key with fingerprint 5C3F 843D 821C 1A5D 67CC 4AD8 4FAD 7A75 3845 59DB (it's on keyservers).

comment:36 in reply to:  33 Changed 4 years ago by mcs

Status: needs_reviewneeds_revision

Replying to gk:

Ok, this is finally up for review:
...

There are a lot of changes, but I think they look OK other than a couple of small things:
1) In tor-browser-bundle, the files gitian/Makefile and gitian/README.build also use lucid and should be fixed.

2) In tor-browser-bundle - gitian/check-prerequisites.sh, the following line probably should be changed to mention "vmbuilder" instead of "python-vm-builder" for consistency:

echo "The VM tool python-vm-builder is missing."

I have not tried to build with these changes yet.

comment:37 Changed 4 years ago by mikeperry

Keywords: TorBrowserTeam201512R added; TorBrowserTeam201511R removed

comment:38 Changed 4 years ago by gk

Status: needs_revisionneeds_review

comment:39 Changed 4 years ago by boklm

One minor thing: maybe the vmclean rule could still remove the lucid VMs, even if they are not created anymore, to remove the VMs created before the switch.

comment:40 in reply to:  39 Changed 4 years ago by gk

Replying to boklm:

One minor thing: maybe the vmclean rule could still remove the lucid VMs, even if they are not created anymore, to remove the VMs created before the switch.

That might indeed be a good idea, bug_15578_v6 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_15578_v6&id=924ee79ab5043fdb1f104ce1410cd20012d91bdc) has it.

comment:41 Changed 4 years ago by boklm

After updating vmbuilder, I tried an alpha build with the bug_15578_v6 branch (and bug_15578_v2 for the gitian-builder directory), but it failed with the following error:

gzip -f "/home/debian/install/faketime/usr/local/share/man/man1/faketime.1"
make[1]: Leaving directory `/home/debian/build/faketime/man'
install -dm0755 "/home/debian/install/faketime/share/doc/faketime/"
install -m0644 README "/home/debian/install/faketime/share/doc/faketime/README"
install -m0644 NEWS "/home/debian/install/faketime/share/doc/faketime/NEWS"
+ export LD_PRELOAD=/home/debian/install/faketime/usr/local/lib/faketime/libfaketime.so.1
+ LD_PRELOAD=/home/debian/install/faketime/usr/local/lib/faketime/libfaketime.so.1
+ export 'FAKETIME=2000-01-01 00:00:00'
+ FAKETIME='2000-01-01 00:00:00'
+ cd ..
+ cd tor-browser
+ rm -rf .git
+ find -type f -print0
+ xargs -0 touch '--date=2000-01-01 00:00:00'
+ rm -f configure
+ rm -f js/src/configure
+ export LD_PRELOAD=
+ LD_PRELOAD=
+ make -f client.mk configure 'CONFIGURE_ARGS=--with-tor-browser-version=5.5a4 --enable-update-channel=alpha --enable-bundled-fonts'
Traceback (most recent call last):
  File "/home/debian/build/tor-browser/mach", line 148, in <module>
    main(sys.argv[1:])
  File "/home/debian/build/tor-browser/mach", line 76, in main
    mach = get_mach()
  File "/home/debian/build/tor-browser/mach", line 67, in get_mach
    mach = check_and_get_mach(dir_path)
  File "/home/debian/build/tor-browser/mach", line 42, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "/home/debian/build/tor-browser/mach", line 30, in load_mach
    return mach_bootstrap.bootstrap(dir_path)
  File "/home/debian/build/tor-browser/build/mach_bootstrap.py", line 217, in bootstrap
    mach.load_commands_from_file(os.path.join(mozilla_dir, path))
  File "/home/debian/build/tor-browser/python/mach/mach/main.py", line 266, in load_commands_from_file
    imp.load_source(module_name, path)
  File "/home/debian/build/tor-browser/testing/web-platform/mach_commands.py", line 24, in <module>
    from update import updatecommandline
  File "/home/debian/build/tor-browser/testing/web-platform/update/__init__.py", line 17, in <module>
    from wptrunner.update import setup_logging, WPTUpdate
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/update/__init__.py", line 12, in <module>
    from update import WPTUpdate
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/update/update.py", line 8, in <module>
    from metadata import MetadataUpdateRunner
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/update/metadata.py", line 7, in <module>
    from .. import metadata
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/metadata.py", line 18, in <module>
    import testloader
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/testloader.py", line 11, in <module>
    import wpttest
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/wpttest.py", line 10, in <module>
    import mozinfo
  File "/home/debian/build/tor-browser/testing/mozbase/mozinfo/mozinfo/__init__.py", line 54, in <module>
    import mozinfo
  File "/home/debian/build/tor-browser/testing/mozbase/mozinfo/mozinfo/mozinfo.py", line 95, in <module>
    import ctypes
  File "/home/debian/install/python/lib/python2.7/ctypes/__init__.py", line 10, in <module>
    from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes
client.mk:201: /.mozconfig.mk: No such file or directory
Traceback (most recent call last):
  File "/home/debian/build/tor-browser/mach", line 148, in <module>
    main(sys.argv[1:])
  File "/home/debian/build/tor-browser/mach", line 76, in main
    mach = get_mach()
  File "/home/debian/build/tor-browser/mach", line 67, in get_mach
    mach = check_and_get_mach(dir_path)
  File "/home/debian/build/tor-browser/mach", line 42, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "/home/debian/build/tor-browser/mach", line 30, in load_mach
    return mach_bootstrap.bootstrap(dir_path)
  File "/home/debian/build/tor-browser/build/mach_bootstrap.py", line 217, in bootstrap
    mach.load_commands_from_file(os.path.join(mozilla_dir, path))
  File "/home/debian/build/tor-browser/python/mach/mach/main.py", line 266, in load_commands_from_file
    imp.load_source(module_name, path)
  File "/home/debian/build/tor-browser/testing/web-platform/mach_commands.py", line 24, in <module>
    from update import updatecommandline
  File "/home/debian/build/tor-browser/testing/web-platform/update/__init__.py", line 17, in <module>
    from wptrunner.update import setup_logging, WPTUpdate
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/update/__init__.py", line 12, in <module>
    from update import WPTUpdate
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/update/update.py", line 8, in <module>
    from metadata import MetadataUpdateRunner
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/update/metadata.py", line 7, in <module>
    from .. import metadata
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/metadata.py", line 18, in <module>
    import testloader
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/testloader.py", line 11, in <module>
    import wpttest
  File "/home/debian/build/tor-browser/testing/web-platform/harness/wptrunner/wpttest.py", line 10, in <module>
    import mozinfo
  File "/home/debian/build/tor-browser/testing/mozbase/mozinfo/mozinfo/__init__.py", line 54, in <module>
    import mozinfo
  File "/home/debian/build/tor-browser/testing/mozbase/mozinfo/mozinfo/mozinfo.py", line 95, in <module>
    import ctypes
  File "/home/debian/install/python/lib/python2.7/ctypes/__init__.py", line 10, in <module>
    from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes
python2.7 /home/debian/build/tor-browser/config/pythonpath.py -I /home/debian/build/tor-browser/testing/mozbase/mozfile \
	    /home/debian/build/tor-browser/python/mozbuild/mozbuild/controller/clobber.py /home/debian/build/tor-browser 
Usage: clobber.py topsrcdir topobjdir
make: *** [/CLOBBER] Error 1

comment:42 Changed 4 years ago by gk

Status: needs_reviewneeds_revision

Hmmm... I thought I had this fixed, thanks. Probably forgot to change that in the final patch.

comment:43 Changed 4 years ago by gk

Configuring the ctypes module fails with libfaketime enabled due to the sanity check failing. Unfortunately, the modules are configured during the make process and we need libfaketime for that one to get a reproducible Python. There are a bunch of options here:

1) Apply libfaketime only to the commands that insert timestamps (let's assume we figure that out; the Bitcoin people are doing that)
2) Get rid of the custom Python build. We needed that only for Ubuntu Lucid which shipped with a far too old Python version. The one in Wheezy (2.7.3) is okay for the time being.
3) Don't care about Python being not reproducible like we don't care about GCC being reproducible for now.

comment:44 Changed 4 years ago by boklm

If we do 1), it looks like Debian managed to make python reproducible without using libfaketime:
https://reproducible.debian.net/rb-pkg/unstable/amd64/python-defaults.html

But if the version of python in Wheezy is enough, dropping our custom python build looks like a good idea. Unless we have an other reason for keeping it.

comment:45 Changed 4 years ago by gk

Keywords: TorBrowserTeam201512 added; TorBrowserTeam201512R removed

comment:46 in reply to:  44 Changed 4 years ago by boklm

Replying to boklm:

If we do 1), it looks like Debian managed to make python reproducible without using libfaketime:
https://reproducible.debian.net/rb-pkg/unstable/amd64/python-defaults.html

Actually I was looking at the wrong python package. The python2.7 is not reproducible:
https://reproducible.debian.net/rb-pkg/unstable/amd64/python2.7.html

comment:47 Changed 4 years ago by gk

Keywords: TorBrowserTeam201601 added; TorBrowserTeam201512 removed

Tickets for Jan 2016.

comment:48 Changed 4 years ago by gk

Keywords: GeorgKoppen201601 added; GeorgKoppen201511 removed

comment:49 Changed 4 years ago by gk

Keywords: GeorgKoppen201601 removed
Owner: changed from gk to boklm
Status: needs_revisionassigned

I think just dropping it then seems reasonable. boklm, could you do that and test it? Having that switch ready for 6.0a1 by the end of January seems like a good thing to me.

comment:50 Changed 4 years ago by boklm

The bug_15578_v7 branch in my repo has a patch to drop the python build:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/log/?h=bug_15578_v7

I am currently building with those changes to check that it is working.

comment:51 in reply to:  50 ; Changed 4 years ago by boklm

Replying to boklm:

I am currently building with those changes to check that it is working.

Using this patch, the build gets stuck in the firefox descriptor while running "make BUILD_HOSTNAME=gitian -j6 -f client.mk build", with a python2.7 process using 100% of the cpu. I will now try to find what is happening.

comment:52 in reply to:  51 Changed 4 years ago by boklm

Replying to boklm:

I will now try to find what is happening.

The cause was the change to FAKETIME_SKIP_CMDS. I have now corrected this in bug_15578_v8:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/log/?h=bug_15578_v8

And restarted a build.

comment:53 Changed 4 years ago by boklm

In the bug_15578_v9 branch I added the python-dev package to the firefox descriptor. And I now have the following build error:

c++ -o MetaData.o -c -I../../dist/stl_wrappers -I../../dist/system_wrappers -include /home/debian/build/tor-browser/config/gcc_hidden.h -DANDROID_SMP=0 -DLOG_NDEBUG=1 -D_GLIBCXX_OS_DEFINES -DHAVE_SYS_UIO_H -DFAKE_LOG_DEVICE -DSTATIC_EXPORTABLE_JS_API -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DMOZ_GLUE_IN_PROGRAM -DAB_CD=en-US -DNO_NSPR_10_SUPPORT -I/home/debian/build/tor-browser/media/libstagefright -I. -I/home/debian/build/tor-browser/media/libstagefright/binding/include -I/home/debian/build/tor-browser/media/libstagefright/frameworks/av/include -I/home/debian/build/tor-browser/media/libstagefright/frameworks/av/include/media/stagefright/foundation -I/home/debian/build/tor-browser/media/libstagefright/frameworks/av/media/libstagefright/ -I/home/debian/build/tor-browser/media/libstagefright/stubs/empty -I/home/debian/build/tor-browser/media/libstagefright/stubs/include -I/home/debian/build/tor-browser/media/libstagefright/stubs/include/media/stagefright/foundation -I/home/debian/build/tor-browser/media/libstagefright/system/core/include -I../../dist/include   -I/home/debian/build/tor-browser/obj-i686-pc-linux-gnu/dist/include/nspr -I/home/debian/build/tor-browser/obj-i686-pc-linux-gnu/dist/include/nss       -fPIC   -DMOZILLA_CLIENT -include ../../mozilla-config.h -MD -MP -MF .deps/MetaData.o.pp  -Wall -Wempty-body -Woverloaded-virtual -Wsign-compare -Wwrite-strings -Wno-invalid-offsetof -Wcast-align -frandom-seed=tor -fno-exceptions -fno-strict-aliasing -fno-rtti -fno-exceptions -fno-math-errno -std=gnu++0x -pthread -pipe  -DNDEBUG -DTRIMMED -g -freorder-blocks -Os -fomit-frame-pointer -Wno-format -Wno-multichar -Wno-sign-compare -Wno-unused    /home/debian/build/tor-browser/media/libstagefright/frameworks/av/media/libstagefright/MetaData.cpp
hexdump.o
cc1plus: error: -Wformat-security ignored without -Wformat [-Werror=format-security]

Which looks similar to https://bugs.launchpad.net/ubuntu/+source/hardening-wrapper/+bug/1347257

comment:54 Changed 4 years ago by gk

There is a backport of a Mozilla patch in my tor-browser branch that should fix that IIRC: https://gitweb.torproject.org/user/gk/tor-browser.git/commit/?h=bug_15578&id=3b8008baa5aefd2602b5e14e79dcc29033006bf6

comment:55 Changed 4 years ago by boklm

Ah, I missed that you made a tor-browser branch. I will try that, thanks.

comment:56 in reply to:  55 Changed 4 years ago by gk

Replying to boklm:

Ah, I missed that you made a tor-browser branch. I will try that, thanks.

Did you get bundles? If so, could/did you test them on other older systems still in use, like older Red Hat Linux versions?

comment:57 Changed 4 years ago by boklm

Yes, using the bug_15578_v9 branch and your tor-browser branch I was able to get some builds. I will try them on some older distributions.

comment:59 Changed 4 years ago by boklm

Keywords: TorBrowserTeam201601R added; TorBrowserTeam201601 removed
Status: assignedneeds_review

comment:60 Changed 4 years ago by gk

Status: needs_reviewneeds_revision

Looks good. One thing, though, is that you are still checking for PYTHON in

for i in OPENSSL BINUTILS GCC PYTHON PYCRYPTO M2CRYPTO PYTHON_MSI GMP

but there is no PYTHON_PACKAGE etc. anymore. This is both in fetch-inputs.sh and verify-tags.sh. Could you make a fixup here? I'd like to test that in 6.0a1.

comment:61 Changed 4 years ago by boklm

Status: needs_revisionneeds_review

I fixed that in branch bug_15578_v10, after rebasing on master.

comment:62 Changed 4 years ago by gk

Resolution: fixed
Status: needs_reviewclosed

OKay, looks good now with v10. I've update all the three repos (tor-browser-bundle master commit f78282bad0e1e2255adcf65198137ee14d7b2235 and ac9da1d5965169f91973d57737fcd20e06351b08; tor-browser tor-browser-38.6.0esr-6.0-1 commit fcbfae6fc60a23fe79a80d543d12891f0a66b93e and gitian-builder tor-browser-builder-3 commits 5c81bf36119dbecc17e55568ef3bc9f30dce0cff and 15d166d65d006f564bf3c7dbb8780ed0649352ba) and opened #18127 for the LXC support. We'll try this in 6.0a1 and see how it is going, thanks.

Note: See TracTickets for help on using tickets.