Remove Ubuntu as dependency for reproducible builds
When building TorBrowserBundle 3.x with Gitian one has to use Ubuntu. (Though not even here I succeed as I tried weeks ago, caught in a loop over and over again)
You are not building under Windows, where it would be complicated to migrate away from, this requires Ubunutu, which is based on Debian so it might work with Debian as well. If there are requirements in software that can't be shipped with Debian there are others based on that, like Linux Mint.
In fact I think this process should be agnostic to the underlying Linux distributions as long as all requirements are meet. The free choice of a distribution, even if it is just for building and even if it is just in a virtual machine, is much more important than having an unnecessary dependency. A single-point of failure.
Activity
I believe one of the main limitations to doing this on plain Debian is that python-vm-builder (one of the dependencies) is not available there and is only available in Ubuntu. The limitation to doing it for other distros is that they don't have debootstrap which it uses to create the build chroots. The python-vm-builder thing might be surmountable for "porting" it to Debian but the other parts involve extending Gitian in a way that is probably out of scope for the Tor project, although certainly a neat idea that I hope gets more exposure. I think Mozilla is using CentOS or Fedora or some other RPM-based distro and given their recent interest in deterministic builds they might be a good candidate for that part of it.
Hello,
Many distributions such as Fedora, Mageia, openSUSE provide a debootstrap package to allow creating a Debian/Ubuntu chroot.
About python-vm-builder, it looks like some people plan to add it to Fedora because they want to use gitian : https://bugzilla.redhat.com/show_bug.cgi?id=964318
There is a pending pull request (https://github.com/defreez/gitian-builder/commit/7ad593397283b07a9006613e16ec95974c724465) that allows to basically get rid of the vmbuilder dependency when using LXC. We should look into that and get it running for TBBs and then we can close this ticket. There is no need to get all ways of building TBB independent from Ubuntu. (That said, using Gentoo + KVM should work as well.)
-
Replying to gk:
There is a pending pull request (https://github.com/defreez/gitian-builder/commit/7ad593397283b07a9006613e16ec95974c724465) that allows to basically get rid of the vmbuilder dependency when using LXC.
Alas, it seems like a ton of work is still needed to get that right as this pull request is still bound to using Ubuntu as the OS. E.g. it needs the ubuntu template which is for instance not available on Debian and so on. Thus, we need a different approach.
-
For a pure Debian approach for the host system see: https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md. We should verify that it works, adapt it to our own setup (currently mainly KVM based) and add it to the HACKING doc/README.
-
I updated
check-prerequisites.shandREADME.buildto incorporate Debian related information and am confident that one can now build Tor Browser on a Debian machine as well (including the creation of the build VMs which was the missing piece before). There might be things we still need to iron out as people are starting to use the instructions. See my bug_10125 for things to review.I realized that we probably don't need all the dependencies checked in
check-prerequisites.sh. I've filed #13992 (moved) for that.Trac:
Status: new to needs_review
Keywords: N/A deleted, TorBrowserTeam201412R added 
Replying to gk:
For a pure Debian approach for the host system see: https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md. We should verify that it works, adapt it to our own setup (currently mainly KVM based) and add it to the HACKING doc/README.
If you mean the online hacking document, then a glance at two others might expose additional lack of clarity:
Trac:
Username: michael-
Replying to gk:
I realized that we probably don't need all the dependencies checked in
check-prerequisites.sh. I've filed #13992 (moved) for that.If we're lucky then Debian systems require less RAM or CPU performance and thus could influence the (yet to be introduced) requirements of check-prerequisites.sh per #11420 (moved).
Trac:
Username: michael -
Replying to gk:
See my bug_10125 for things to review.
There is a typo in the vm-builder URL: .tar.gzr instead of .tar.gz.
Otherwise, the instructions to install vm-builder are working. I have now started a build on a Debian Wheezy machine to check if that works, and will update the ticket later with the result.
-
Replying to boklm:
Running vmbuilder during the build process failed with the following error: ImportError: No module named Cheetah.Template
Installing the package python-cheetah fixed this error, so maybe we should add it to the list of packages to check in check-prerequisites.sh on Debian.
Done in bug_10125_v3 in my public repo, thanks.
-
Replying to boklm:
After looking at the dependencies listed in the Ubuntu vmbuilder package, I think we should also check that the following packages are installed: debootstrap parted kpartx rsync
Might be a good idea, yes. See bug_10125_v4. We can take them out again in #13992 (moved) if we don't have to take care of them explicitly ourselves.
-
Replying to gk:
Replying to boklm:
After looking at the dependencies listed in the Ubuntu vmbuilder package, I think we should also check that the following packages are installed: debootstrap parted kpartx rsync
Might be a good idea, yes. See bug_10125_v4. We can take them out again in #13992 (moved) if we don't have to take care of them explicitly ourselves.
Ok, I see that you added the packages to the "dpkg -s" command. I think they should also be added to the suggested "apt-get install" command.
Other than this, I think it's good to be merged.
mentioned in issue #13992 (moved)
mentioned in issue #15578 (moved)
mentioned in issue tpo/applications/tor-browser#13992 (closed)
