Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#18127 closed enhancement (fixed)

Add LXC support for building with Debian guest VMs

Reported by: gk Owned by: boklm
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-gitian, TorBrowserTeam201603R
Cc: josephbisch, boklm, mikeperry, arthuredelstein, mcs Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Now that #15578 is fixed we need to add LXC support for building with Debian guest VMs as well.

Child Tickets

Attachments (2)

Change History (32)

comment:1 Changed 3 years ago by josephbisch

Is there a estimate/roadmap on the release of 6.0a1? I saw in #15578 that you're aiming to use Debian with Gitian for that release. I am just wondering how soon this is needed.

comment:2 in reply to:  1 Changed 3 years ago by gk

Replying to josephbisch:

Is there a estimate/roadmap on the release of 6.0a1? I saw in #15578 that you're aiming to use Debian with Gitian for that release. I am just wondering how soon this is needed.

6.0a1 is coming out next week and has the changes in #15578. But 6.0 will get out end of May.

comment:3 Changed 3 years ago by josephbisch

I have something working (with gitian-builder in general, I haven't tried it with TBB): https://github.com/devrandom/gitian-builder/pull/114.

comment:4 in reply to:  3 Changed 3 years ago by boklm

Replying to josephbisch:

I have something working (with gitian-builder in general, I haven't tried it with TBB): https://github.com/devrandom/gitian-builder/pull/114.

I will try it with TBB and let you know how that works. Thanks!

comment:5 Changed 3 years ago by boklm

Status: newneeds_review

The bug_18127 branch on my tor-browser-bundle.git repo has a few fixes:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/commit/?h=bug_18127

The bug_18127-v2 branch in my gitian-builder repo contains our patches rebased on master, the patch from PR 114, and a few other fixes:
https://github.com/boklm/gitian-builder/commits/bug_18127-v2

Using those changes I have been able to build Tor Browser 6.0a1 with both KVM and LXC, and the binaries produced were identical to the released 6.0a1 builds (except for the Browser/TorBrowser/Docs/sources/bundle.inputs which contains the gitian-builder.git commit hash).

comment:6 Changed 3 years ago by boklm

Keywords: TorBrowserTeam201602R added

comment:7 Changed 3 years ago by gk

Could you test your gitian-builder branch with KVM builds as well? In particular, if we merge your branch with our tor-browser-builder-3 I'd like to be sure that both our stable builds and our alpha builds are still working with KVM: from VM creation to building. My reasoning here is that we so far only cherry-picked particular commits from gitian-builder master and I fear there might be commits we now pull in that work fine with LXC but break our KVM builds.

comment:8 Changed 3 years ago by boklm

I cleaned all VMs and did a build using this gitian-builder branch with KVM and it worked. The resulting files do not match 6.0a1, but that is because of the gitian-builder commit hash in the bundle.inputs file included in the bundles.

comment:9 Changed 3 years ago by boklm

However I only tried this branch with a build of the alpha, not the stable release. I don't know if having this in the tor-browser-builder-3 could cause problems in the release builds.

I see that we checkout tor-browser-builder-3 in fetch-inputs.sh. Is there a reason for doing a checkout of this branch and not of $GITIAN_TAG ?

comment:10 in reply to:  9 Changed 3 years ago by gk

Replying to boklm:

However I only tried this branch with a build of the alpha, not the stable release. I don't know if having this in the tor-browser-builder-3 could cause problems in the release builds.

I see that we checkout tor-browser-builder-3 in fetch-inputs.sh. Is there a reason for doing a checkout of this branch and not of $GITIAN_TAG ?

Well, we do in fact check out the GITIAN_TAG for the non-nightly builds via verify-tags.sh but agreed this is a strange construction and, now that I remember, could haven been responsible for ln5's issues with switching to Wheezy for Linux. Could you make a change for this too (in this ticket is fine)? Because I am thinking about taking your rebased branch as a clean, new tor-browser-builder-4 we are going to use from now on and it would be good to do the right thing with GITIAN_TAG as well.

Last edited 3 years ago by gk (previous) (diff)

comment:11 Changed 3 years ago by boklm

comment:12 Changed 3 years ago by gk

Status: needs_reviewneeds_information

Some things I noted while testing the setup:

I hit:

W: Cannot check Release signature; keyring file not available /usr/share/keyrings/debian-archive-keyring.gpg

while creating the Linux images which scares me. Can we do something about it? Is that an issue?

Further, upon starting the build I always get a bunch of

base-wheezy-i386 already exists, please remove it first
base-wheezy-amd64 already exists, please remove it first

messages. Do we need to do something to get rid of them?

Finally, I've looked at the LXC tips section in the README.md but could not find a hint why we need sudo privs now to create the images. This is not an issue with the tor-browser-builder-3 branch. Any ideas?

comment:13 in reply to:  12 ; Changed 3 years ago by boklm

Replying to gk:

Some things I noted while testing the setup:

I hit:

W: Cannot check Release signature; keyring file not available /usr/share/keyrings/debian-archive-keyring.gpg

while creating the Linux images which scares me. Can we do something about it? Is that an issue?

Ah, I see that Ubuntu has a debian-archive-keyring package which might fix this. So we probably want to add it to check-prerequisites.sh.

Further, upon starting the build I always get a bunch of

base-wheezy-i386 already exists, please remove it first
base-wheezy-amd64 already exists, please remove it first

messages. Do we need to do something to get rid of them?

It seems the image files don't have a .qcow2 extension anymore. So we need to update gitian/make-vms.sh to remove the .qcow2.

Finally, I've looked at the LXC tips section in the README.md but could not find a hint why we need sudo privs now to create the images. This is not an issue with the tor-browser-builder-3 branch. Any ideas?

In tor-browser-builder-3, sudo was used to call vmbuilder. In the new version the same thing is done without vmbuilder, but with different sudo calls to debootstrap, mount, cp, rm. So it is less easy now to allow only specific sudo calls.

If we want to fix that, we can move all the commands from bin/make-base-vm to create an image in the LXC case to a separate file.

comment:14 in reply to:  13 Changed 3 years ago by boklm

Replying to boklm:

Replying to gk:

Some things I noted while testing the setup:

I hit:

W: Cannot check Release signature; keyring file not available /usr/share/keyrings/debian-archive-keyring.gpg

while creating the Linux images which scares me. Can we do something about it? Is that an issue?

Ah, I see that Ubuntu has a debian-archive-keyring package which might fix this. So we probably want to add it to check-prerequisites.sh.

Maybe we should also patch the debootstrap call in bin/make-base-vm, to add a --force-check-gpg to make sure it fails when the keyring file is missing.

I think we have a similar problem with KVM builds where vmbuilder is used, which calls debootstrap too.

comment:15 Changed 3 years ago by gk

Keywords: TorBrowserTeam201602 added; TorBrowserTeam201602R removed
Owner: changed from tbb-team to boklm
Status: needs_informationassigned

Sounds good to me. Could you prepare a patch/patches addressing all the issues?

comment:16 Changed 3 years ago by boklm

The branch bug_18127-v2 has a fix for the first 2 issues:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/log/?h=bug_18127-v2

I'm looking at the options for the 3rd one.

comment:17 Changed 3 years ago by gk

Keywords: TorBrowserTeam201603 added; TorBrowserTeam201602 removed

comment:18 in reply to:  13 Changed 3 years ago by boklm

Replying to boklm:

In tor-browser-builder-3, sudo was used to call vmbuilder. In the new version the same thing is done without vmbuilder, but with different sudo calls to debootstrap, mount, cp, rm. So it is less easy now to allow only specific sudo calls.

The change from vmbuilder to debootstrap was done with this commit:
https://github.com/devrandom/gitian-builder/commit/af56f89a6acffd363c845a489ec163f0d85d30be
For this ticket:
https://github.com/devrandom/gitian-builder/issues/86

I'm not sure what is the best way to fix this problem. The different solutions I can see to fix this are:

  • revert the change to move from vmbuilder to debootstrap. But looking at the ticket it looks like we will have problems because of the kernel and grup packages installed by vmbuilder, so this probably implies patching vmbuilder too.
  • extract the sudo commands from make-base-vm and put them in a script in a directory such as /usr/local/sbin that we add to sudoers, then patch make-base-vm to use this script with sudo if it exists.
  • giving sudoers access to build users to debootstrap, mount, umount, cp, rm commands which is similar to giving full sudoers access

comment:19 Changed 3 years ago by gk

Cc: mikeperry added
Status: assignedneeds_information

Hrm. There is no clear winner for me in the options you collected. Maybe Mike has an idea/preferred solution? I am not concerned with respect to requiring sudo for container creation in general. Rather, our LXC machine usage benefits from not needing that.

comment:20 Changed 3 years ago by arthuredelstein

Cc: arthuredelstein added

comment:21 Changed 3 years ago by mcs

Cc: mcs added

comment:22 Changed 3 years ago by boklm

Regarding our LXC machine, we currently have the following sudo access:

    (root) NOPASSWD: /usr/bin/lxc-execute
    (root) NOPASSWD: /usr/bin/lxc-start
    (root) NOPASSWD: /usr/bin/vmbuilder
    (root) NOPASSWD: /sbin/kpartx
    (root) NOPASSWD: /bin/cp
    (root) NOPASSWD: /bin/chown

So what we would need to add is debootstrap, mount, umount, rm.

comment:23 in reply to:  22 Changed 3 years ago by gk

Replying to boklm:

Regarding our LXC machine, we currently have the following sudo access:

    (root) NOPASSWD: /usr/bin/lxc-execute
    (root) NOPASSWD: /usr/bin/lxc-start
    (root) NOPASSWD: /usr/bin/vmbuilder
    (root) NOPASSWD: /sbin/kpartx
    (root) NOPASSWD: /bin/cp
    (root) NOPASSWD: /bin/chown

So what we would need to add is debootstrap, mount, umount, rm.

I wonder if we would need lxc-start on the LXC machine anymore. That said do we only need them for VM creation like if we are in KVM mode? I am inclined to merge the patches and find a good way of dealing with sudo problems later when needed.

comment:24 Changed 3 years ago by gk

Keywords: TorBrowserTeam201603R added; TorBrowserTeam201603 removed
Resolution: fixed
Status: needs_informationclosed

Okay, this is tor-browser-buidler-4 in the gitian-builder repo and commit 4d625212a6689ebd1b291c5450096aeedad94e28, 9e1ff5bf59a07f9a4277d75e2331c47f297a6921 and 1bc9c54d1eb82b3da499d49fe9c54e0675b0001b on tor-browser-bundle master.

comment:25 Changed 3 years ago by gk

Resolution: fixed
Status: closedreopened

< GeKo> hm.. what was the reasoning for setting GITIAN_HOST_IP=10.0.3.1?
< GeKo> it seems setting it to 10.0.2.2 as it is on devrandom's master branch works for my kvm box
< GeKo> while setting it to 10.0.3.1 does not
< GeKo> aha! see devrandom's make-base-vm: "10.0.2.2 is the host IP as visible from the guest under qemu networking"

comment:26 Changed 3 years ago by gk

Resolution: fixed
Status: reopenedclosed

Okay. This is fixed with commit f1b9924b8928a8954776d7aa96b669e69849a5a5. The patch in the attachment got applied in commit 162a3984a3e1c069c366676cc7fdd0d4bc8a0586 on tor-browser-bundle master.

comment:27 Changed 3 years ago by gk

Resolution: fixed
Status: closedreopened

It seems setting GITIAN_HOST_IP=10.0.3.1 was necessary for LXC builds. I tested a bit more and run into the same issues as Arthur. I guess what we need to do is making sure 10.0.2.2 is getting used for KVM and 10.0.3.1 for LXC then?

comment:28 Changed 3 years ago by boklm

I attached a patch to use 10.0.2.2 for KVM and 10.0.3.1 for LXC.

This fixed the issue on my LXC build machine.

comment:29 Changed 3 years ago by gk

Resolution: fixed
Status: reopenedclosed

Seems to work on all the build machines I have currently available, thanks. This is commit 262ad8f15fc870023b4cfc7399f6908f11056342 on tor-browser-builder-4. Let's hope this bug rests in peace now.

comment:30 Changed 3 years ago by boklm

Yes, let's hope there's nothing more. Sorry for not spotting this issue earlier, I had tested the gitian branch with KVM and I don't know why I did not have the issue with GITIAN_HOST_IP.

Note: See TracTickets for help on using tickets.