Opened 6 years ago

Closed 5 years ago

#10125 closed task (fixed)

Remove Ubuntu as dependency for reproducible builds

Reported by: bastik Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Keywords: gitian, tbb-gitian, TorBrowserTeam201501R, tbb-4.5-alpha-3, boklm201501R
Cc: gk, michael Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

When building TorBrowserBundle 3.x with Gitian one has to use Ubuntu. (Though not even here I succeed as I tried weeks ago, caught in a loop over and over again)

You are not building under Windows, where it would be complicated to migrate away from, this requires Ubunutu, which is based on Debian so it might work with Debian as well. If there are requirements in software that can't be shipped with Debian there are others based on that, like Linux Mint.

In fact I think this process should be agnostic to the underlying Linux distributions as long as all requirements are meet. The free choice of a distribution, even if it is just for building and even if it is just in a virtual machine, is much more important than having an unnecessary dependency. A single-point of failure.

Child Tickets

Change History (24)

comment:1 Changed 6 years ago by erinn

I believe one of the main limitations to doing this on plain Debian is that python-vm-builder (one of the dependencies) is not available there and is only available in Ubuntu. The limitation to doing it for other distros is that they don't have debootstrap which it uses to create the build chroots. The python-vm-builder thing might be surmountable for "porting" it to Debian but the other parts involve extending Gitian in a way that is probably out of scope for the Tor project, although certainly a neat idea that I hope gets more exposure. I think Mozilla is using CentOS or Fedora or some other RPM-based distro and given their recent interest in deterministic builds they might be a good candidate for that part of it.

comment:2 Changed 6 years ago by gk

Cc: gk added
Keywords: gitian added

comment:3 Changed 6 years ago by boklm

Hello,

Many distributions such as Fedora, Mageia, openSUSE provide a debootstrap package to allow creating a Debian/Ubuntu chroot.

About python-vm-builder, it looks like some people plan to add it to Fedora because they want to use gitian :
https://bugzilla.redhat.com/show_bug.cgi?id=964318

comment:4 Changed 6 years ago by gk

There is a pending pull request (https://github.com/defreez/gitian-builder/commit/7ad593397283b07a9006613e16ec95974c724465) that allows to basically get rid of the vmbuilder dependency when using LXC. We should look into that and get it running for TBBs and then we can close this ticket. There is no need to get all ways of building TBB independent from Ubuntu. (That said, using Gentoo + KVM should work as well.)

comment:5 in reply to:  4 Changed 6 years ago by gk

Replying to gk:

There is a pending pull request (https://github.com/defreez/gitian-builder/commit/7ad593397283b07a9006613e16ec95974c724465) that allows to basically get rid of the vmbuilder dependency when using LXC.

Alas, it seems like a ton of work is still needed to get that right as this pull request is still bound to using Ubuntu as the OS. E.g. it needs the ubuntu template which is for instance not available on Debian and so on. Thus, we need a different approach.

comment:6 Changed 5 years ago by erinn

Component: Tor bundles/installationTor Browser
Keywords: tbb-gitian added
Owner: changed from erinn to tbb-team

comment:7 Changed 5 years ago by gk

For a pure Debian approach for the host system see: https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md. We should verify that it works, adapt it to our own setup (currently mainly KVM based) and add it to the HACKING doc/README.

comment:8 Changed 5 years ago by gk

Keywords: TorBrowserTeam201412R added
Status: newneeds_review

I updated check-prerequisites.sh and README.build to incorporate Debian related information and am confident that one can now build Tor Browser on a Debian machine as well (including the creation of the build VMs which was the missing piece before). There might be things we still need to iron out as people are starting to use the instructions. See my bug_10125 for things to review.

I realized that we probably don't need all the dependencies checked in check-prerequisites.sh. I've filed #13992 for that.

comment:9 in reply to:  7 Changed 5 years ago by michael

Replying to gk:

For a pure Debian approach for the host system see: https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md. We should verify that it works, adapt it to our own setup (currently mainly KVM based) and add it to the HACKING doc/README.

If you mean the online hacking document, then a glance at two others might expose additional lack of clarity:

comment:10 in reply to:  8 Changed 5 years ago by michael

Replying to gk:

I realized that we probably don't need all the dependencies checked in check-prerequisites.sh. I've filed #13992 for that.

If we're lucky then Debian systems require less RAM or CPU performance and thus could influence the (yet to be introduced) requirements of check-prerequisites.sh per #11420.

comment:11 Changed 5 years ago by michael

Cc: michael added

comment:12 Changed 5 years ago by mikeperry

Keywords: TorBrowserTeam201501R added; TorBrowserTeam201412R removed

comment:13 Changed 5 years ago by mikeperry

Keywords: 4.5-alpha-3 added

comment:14 Changed 5 years ago by mikeperry

Keywords: tbb-4.5-alpha-3 added

comment:15 Changed 5 years ago by mikeperry

Keywords: 4.5-alpha-3 removed

comment:16 Changed 5 years ago by boklm

Keywords: boklm201501R added

comment:17 in reply to:  8 ; Changed 5 years ago by boklm

Replying to gk:

See my bug_10125 for things to review.

There is a typo in the vm-builder URL: .tar.gzr instead of .tar.gz.

Otherwise, the instructions to install vm-builder are working. I have now started a build on a Debian Wheezy machine to check if that works, and will update the ticket later with the result.

comment:18 in reply to:  17 Changed 5 years ago by gk

Replying to boklm:

Replying to gk:

See my bug_10125 for things to review.

There is a typo in the vm-builder URL: .tar.gzr instead of .tar.gz.

bug_10125_v2 has this fixed.

comment:19 Changed 5 years ago by boklm

Running vmbuilder during the build process failed with the following error:

ImportError: No module named Cheetah.Template

Installing the package python-cheetah fixed this error, so maybe we should add it to the list of packages to check in check-prerequisites.sh on Debian.

comment:20 in reply to:  19 Changed 5 years ago by gk

Replying to boklm:

Running vmbuilder during the build process failed with the following error:

ImportError: No module named Cheetah.Template

Installing the package python-cheetah fixed this error, so maybe we should add it to the list of packages to check in check-prerequisites.sh on Debian.

Done in bug_10125_v3 in my public repo, thanks.

comment:21 Changed 5 years ago by boklm

After looking at the dependencies listed in the Ubuntu vmbuilder package, I think we should also check that the following packages are installed:
debootstrap parted kpartx rsync

comment:22 in reply to:  21 ; Changed 5 years ago by gk

Replying to boklm:

After looking at the dependencies listed in the Ubuntu vmbuilder package, I think we should also check that the following packages are installed:
debootstrap parted kpartx rsync

Might be a good idea, yes. See bug_10125_v4. We can take them out again in #13992 if we don't have to take care of them explicitly ourselves.

comment:23 in reply to:  22 Changed 5 years ago by boklm

Replying to gk:

Replying to boklm:

After looking at the dependencies listed in the Ubuntu vmbuilder package, I think we should also check that the following packages are installed:
debootstrap parted kpartx rsync

Might be a good idea, yes. See bug_10125_v4. We can take them out again in #13992 if we don't have to take care of them explicitly ourselves.

Ok, I see that you added the packages to the "dpkg -s" command. I think they should also be added to the suggested "apt-get install" command.

Other than this, I think it's good to be merged.

comment:24 Changed 5 years ago by mikeperry

Resolution: fixed
Status: needs_reviewclosed

Ok, I fixed the apt line and this is now merged in origin/master. Thanks!

Note: See TracTickets for help on using tickets.