Opened 3 years ago

Last modified 5 months ago

#25102 assigned task

Add script to sign nightly build mar files, generate update-responses xml and publish the new version

Reported by: boklm Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, tbb-update, TorBrowserTeam202004, gitlab-tb-tor-browser-build
Cc: boklm, mcs, brade, ln5, tbb-team Actual Points: 4.5
Parent ID: #18867 Points: 3.5
Reviewer: gk Sponsor:

Description

We need a script that will fetch the latest nightly build from the build machine, then sign the mar files and publish them.

Later we can improve it to fetch builds from multiple builders and only do the signing if they match.

Child Tickets

TicketStatusOwnerSummaryComponent
#32768closedboklmCreate a build-infos.json file containing firefox platform_version and buildidApplications/Tor Browser
#32805closedboklmMake creation of downloads.json optionalApplications/Tor Browser
#33380closedboklmAdd build-infos.json to sha256sums-unsigned-build.txtApplications/Tor Browser

Change History (26)

comment:1 Changed 21 months ago by gk

Keywords: tbb-updater added

comment:2 Changed 21 months ago by gk

Keywords: tbb-update added; tbb-updater removed

Renaming keyword to make it a bit broader

comment:3 Changed 14 months ago by gk

Keywords: TorBrowserTeam201909 added
Points: 2

comment:4 Changed 13 months ago by pili

Keywords: TorBrowserTeam201910 added

comment:5 Changed 13 months ago by pili

Keywords: TorBrowserTeam201909 removed

comment:6 Changed 12 months ago by pili

Keywords: TorBrowserTeam201911 added; TorBrowserTeam201910 removed

Moving tickets to November 2019

comment:7 Changed 12 months ago by pili

Cc: tbb-team added
Owner: changed from tbb-team to boklm
Status: newassigned

Assigning tickets to boklm for the next few months

comment:8 Changed 11 months ago by pili

Keywords: TorBrowserTeam201912 added; TorBrowserTeam201911 removed

Moving tickets to December

comment:9 Changed 11 months ago by boklm

Summary: Add script to sign nightly build mar filesAdd script to sign nightly build mar files, generate update-responses xml and publish the new version

I think the same script can be used to generate the update-responses xml, and publish both the mar files and update-responses xml.

comment:11 Changed 11 months ago by boklm

This commit adds the script tools/signing/nightly/sign-nightly:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_25102_v5&id=c2b3fc259d746de78f4c2240b6aee6f1932df8d8

This script downloads the mar files from the latest nightly, sign them, and generate update_responses xml files.

What is still missing:

  • Correctly setting platformVersion and buildID (the current version of the script sets them to 0). This depends on #32768.
  • Uploading of the mar files and update_responses xml somewhere to make them available for users. This depends on #32800.

I also started some ansible scripts to setup a signing machine:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_25102_v5&id=42df2ed18ed5c6409253e2a3651e5fa437150bda

comment:12 Changed 10 months ago by sysrqb

Keywords: TorBrowserTeam202001 added; TorBrowserTeam201912 removed

comment:13 Changed 9 months ago by boklm

Actual Points: 3
Points: 23.5

comment:14 Changed 9 months ago by pili

Keywords: TorBrowserTeam202002 added; TorBrowserTeam202001 removed

Moving tickets to February

comment:15 Changed 9 months ago by boklm

Actual Points: 34

comment:16 Changed 8 months ago by boklm

Actual Points: 44.5

Signed mar files and update xml are now available on https://nightlies.tbb.torproject.org/nightly-updates/

The current version of the script doing that is in branch bug_25102_v8:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_25102_v8&id=025b57bea995db0377a7188b24503c73f8aefa6a

I will set it in needs_review once #33403 and #33402, after checking that updates are working and no other changes are needed.

comment:17 Changed 8 months ago by pili

Keywords: TorBrowserTeam202003 added; TorBrowserTeam202002 removed

We are no longer in February, moving tickets

comment:18 Changed 7 months ago by boklm

Keywords: TorBrowserTeam202003R added; TorBrowserTeam202003 removed
Status: assignedneeds_review

The branch bug_25102_v10 has two commits for review:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_25102_v10&id=709592712db8fc1050c88143b7ca3eea09b8e77f

The first one adds the script tools/signing/nightly/sign-nightly which is downloading, signing and uploading mar files and xml responses to https://nightlies.tbb.torproject.org/nightly-updates/.

The second commits contains ansible scripts to deploy it.

This is what is currently being used, and it seems to be working correctly.

comment:19 Changed 7 months ago by pili

Keywords: TorBrowserTeam202004R added; TorBrowserTeam202003R removed

We are no longer in March

comment:20 Changed 7 months ago by boklm

GeKo was asking on IRC why we are hardcoding the version of martools we are using for signing the mar files, and why we are not just using the martools from the nightly builds.

The reason to do that is to isolate the signing VM from the build VM. The nightly setup looks like this:

  • the nightly build VM is building the nightly, and makes the builds available through http on an onion address. The nightly build is using the git master branch from several components, which means that an attacker who manages to get root access to the git server would also be able to access the build VM.
  • the signing VM fetches the mar files from the build VM through the onion address, sign them (using the martools from a stable Tor Browser version), and upload the signed mars to https://nightlies.tbb.torproject.org/ (using an ssh key). If we were using the martools from the nightly build, an attacker who got access to the build VM could get access to the signing VM too and steal the signing key, and the ssh key to upload malicious mar files to nightlies.tbb.torproject.org.

So I think it is useful to try to keep the build and signing VMs separate. Unfortunately this is not enough to avoid the case where an attacker uses their access to the build VM to produce malicious builds for nightly users. To mitigate this I think what we can do:

  • reinstall the build VM frequently. Keeping build and signing VMs separate means we don't have to rotate mar signing and ssh keys at the same time.
  • require all commits (or at least the top commit on the master branch) to be signed
  • have a second build VM, and check that the builds are matching before signing them.

comment:21 Changed 7 months ago by pili

Reviewer: gk

comment:22 in reply to:  20 Changed 7 months ago by gk

Replying to boklm:

GeKo was asking on IRC why we are hardcoding the version of martools we are using for signing the mar files, and why we are not just using the martools from the nightly builds.

The reason to do that is to isolate the signing VM from the build VM. The nightly setup looks like this:

  • the nightly build VM is building the nightly, and makes the builds available through http on an onion address. The nightly build is using the git master branch from several components, which means that an attacker who manages to get root access to the git server would also be able to access the build VM.

Hey, another attacker is coming out of the weeds, good! Keep 'em coming. :)

  • the signing VM fetches the mar files from the build VM through the onion address, sign them (using the martools from a stable Tor Browser version), and upload the signed mars to https://nightlies.tbb.torproject.org/ (using an ssh key). If we were using the martools from the nightly build, an attacker who got access to the build VM could get access to the signing VM too and steal the signing key, and the ssh key to upload malicious mar files to nightlies.tbb.torproject.org.

So I think it is useful to try to keep the build and signing VMs separate. Unfortunately this is not enough to avoid the case where an attacker uses their access to the build VM to produce malicious builds for nightly users. To mitigate this I think what we can do:

  • reinstall the build VM frequently. Keeping build and signing VMs separate means we don't have to rotate mar signing and ssh keys at the same time.
  • require all commits (or at least the top commit on the master branch) to be signed
  • have a second build VM, and check that the builds are matching before signing them.

Thanks for writing this up, really appreciated. Yes, the last three steps are good things to do. Could you open bugs for the last two (signed commits which we probably should enforce with a git hook and some means on the nightly side to only start building if commits are properly signed; and the second build VM) items so we don't forget about them?

Unfortunately, I fear the issue with mar signing tools breaking will happen in situations where we need that least, that is during transition to a new major release once we do our first nightly builds with the new code. In that moment we do not want to deal with broken mar signing tools, too, so that the whole team is blocked on that. So, how about the following:

a) We do use a hardcoded martools version as you propose, but

b) if that fails, e.g. due to a new major upcoming release, we use the martools that gets shipped with the nightly so that other folks on the team can work on that new upcoming major release while someone from the team fixes the mar signing issue (the former do not get blocked anymore on the latter) (we can then rotate the signing key afterwards and set the VMs newly up to be on the safe side).

comment:23 Changed 7 months ago by gk

Keywords: TorBrowserTeam202004 added; TorBrowserTeam202004R removed
Status: needs_reviewneeds_revision

18:41 <+GeKo> boklm: could you change the uuid for the nightly signing key?
18:41 <+GeKo> it's a role key and not bound to any person in particular
18:42 <+GeKo> tbb-nightly-builds@ something something
18:42 <+GeKo> or something similar

comment:24 in reply to:  20 Changed 6 months ago by boklm

Replying to boklm:

  • require all commits (or at least the top commit on the master branch) to be signed

I opened #34046 for that.

comment:25 Changed 6 months ago by gaba

Owner: changed from boklm to tbb-team
Status: needs_revisionassigned

Release all this tickets back into tbb-team.

comment:26 Changed 5 months ago by gk

Keywords: gitlab-tb-tor-browser-build added

Add magic gitlab keyword.

Note: See TracTickets for help on using tickets.