Sign our MAR files

added TorBrowserTeam201412 TorBrowserTeam201412R component::applications/tor browser owner::mcs priority::high resolution::fixed severity::normal status::closed tbb-no-uplift-60 tbb-security type::defect labels

This has been on my and brade's list of things to do for a while. I think Mozilla only uses signatures on Windows, e.g., see:

http://mxr.mozilla.org/mozilla-central/source/browser/confvars.sh#10

So it is possible this will require code changes as well (if code paths are Windows only or untested).

Trac:
Cc: N/A to brade, mcs

Sounds like a duplicate of #10392 (moved), no?

Trac:
Cc: brade, mcs to brade, mcs, gk

Trac:
Cc: brade, mcs, gk to brade, mcs, gk, boklm

As it turns out, Mozilla has recently made some progress towards adding signature support on all platforms. Here is their tracking bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=973933

Most of the dependent bugs have patches. One thing that is messy is that they want to keep the updater executable as small and standalone as possible, so they are doing things like using OS APIs to verify signatures (on Windows they use CryptoAPI and on Mac OS they use Apple's Security framework). I think they plan to use NSS on Linux, and we may want to use NSS everywhere for trust and auditability reasons.

While thinking about comment:10:ticket:13407 and that it probably is wise to "just" have a role signing key due to just one key for verifying our MARs I was wondering whether it would be feasible to take advantage of reproducibly built MAR files given that no human interaction is interfering here. This is definitely worth a new bug if it is worth one at all (and I am volunteering for coding this actually). Given your knowledge of the MAR signing code Mozilla provides do you think there are general obstacles to extend that to add support for a verification method relying on more than one key?

Replying to gk:

Given your knowledge of the MAR signing code Mozilla provides do you think there are general obstacles to extend that to add support for a verification method relying on more than one key?

I am not sure exactly what you are asking. Mozilla currently supports embedding zero or more signatures in a MAR file. The signatures are added using a program named signmar which is really just a more capable variant of the mar program. signmar requires an NSS certificate database that contains a private key plus a self-signed certificate.

Then, if you configure the Firefox build with --enable-verify-mar, one or two certificates are embedded in the updater program and signatures contained within any MAR file that is downloaded are checked against those certificates. All signatures must be verified using one or the other cert or the MAR file will be rejected; that is, if the MAR file contains two signatures both must be verifiable. And at least one signature must be present when --enable-verify-mar is turned on.

Oh, I wasn't aware that multiple signatures were already supported in this way. If that is the case, we may want to consider having two or three keys: one held by Georg, one by myself, and one on a dist server. Though this has downsides in that it would require Georg and I to always be available to sign builds.. I suppose we could instead share a builders key, and then have the second key live on a signing machine that other people can get access to in an emergency?

Might be too much to deal with for the first rollout, though.

Replying to mcs:

Replying to gk:

Given your knowledge of the MAR signing code Mozilla provides do you think there are general obstacles to extend that to add support for a verification method relying on more than one key?

I am not sure exactly what you are asking. Mozilla currently supports embedding zero or more signatures in a MAR file. The signatures are added using a program named signmar which is really just a more capable variant of the mar program. signmar requires an NSS certificate database that contains a private key plus a self-signed certificate.

Then, if you configure the Firefox build with --enable-verify-mar, one or two certificates are embedded in the updater program and signatures contained within any MAR file that is downloaded are checked against those certificates. All signatures must be verified using one or the other cert or the MAR file will be rejected; that is, if the MAR file contains two signatures both must be verifiable. And at least one signature must be present when --enable-verify-mar is turned on.

Thanks. I was basically asking whether it is easily possible to avoid the bottleneck of just having one signing key. Originally, I was thinking in order to avoid that we somehow need to bolt the verification of the signing and hashing we do for the reproducible builds onto the MAR signing as a kind of additional assurance that everything is okay (like we have it now with a signature for each package and an "advanced verification" via the sah256sums and a couple of builder who sign that file with their own key). But, great that Mozilla supports having multiple signing keys as we may be able to leverage that work to get the same results or at least comparable ones (security-wise).

Replying to mikeperry:

Oh, I wasn't aware that multiple signatures were already supported in this way. If that is the case, we may want to consider having two or three keys: one held by Georg, one by myself, and one on a dist server. Though this has downsides in that it would require Georg and I to always be available to sign builds.. I suppose we could instead share a builders key, and then have the second key live on a signing machine that other people can get access to in an emergency?

Might be too much to deal with for the first rollout, though.

Yes, I think that should be in a new ticket. I've created #13730 (moved) for it. I am especially interested in thinking about needing just a threshold of valid signatures which might release the burden on us to be always available for signing purposes.

I have a design question. The new updater binary has a dependency on various shared libraries that are bundled with the browser (libnss3.so, libnspr4.so, etc.) On Windows, these libraries are found by the OS when the updater is started because of the fix we made for #13594 (moved).

On Mac OS and Linux, the libraries won't be found. Possible solutions:

(1) Modify the browser to set LD_LIBRARY_PATH before launching the updater. This means that while it is running, the updater would use libraries that are possibly going to be updated. I think that is OK because both Linux and Mac OS allow rename and unlink on an open file.

(2) Modify the browser to copy all of the required shared libraries when it makes a copy of the updater binary itself (i.e., we would extend the code here to do more: https://gitweb.torproject.org/tor-browser.git/blob/2822ccdb6d00b563413a285fe63488ab2ca7b460:/toolkit/xre/nsUpdateDriver.cpp#l385 ). To do this, we would need to embed a list of shared libraries inside the browser (which we would then have to maintain).

Kathy and I prefer solution (1) unless someone sees a problem with that approach. Comments?

I think whichever mechanism we use should be the same on all three platforms. I have a quick question about approach 2: would we be copying the old libraries into the update directory, or the new ones from the included mar?

I'm assuming old because the updater hasn't run at this point, and if this is true, it seems to me that option 1 is the winner (simply adding the libraries to the search path is better than trying to maintain a list and copy them, since its the same code either way).

Replying to mikeperry:

I think whichever mechanism we use should be the same on all three platforms. I have a quick question about approach 2: would we be copying the old libraries into the update directory, or the new ones from the included mar?

I'm assuming old because the updater hasn't run at this point, and if this is true, it seems to me that option 1 is the winner (simply adding the libraries to the search path is better than trying to maintain a list and copy them, since its the same code either way).

Correct, we would be copying old libraries. If we can avoid copying them, it seems like a win.

Trac:
Owner: tbb-team to mcs
Cc: brade, mcs, gk, boklm to brade, mcs, gk, boklm, mikeperry
Status: new to assigned

The tor-browser changes are ready for review. The patches may be found on a branch named bug13379-01 in user/brade/tor-browser.git; that is, here:

https://gitweb.torproject.org/user/brade/tor-browser.git/shortlog/refs/heads/bug13379-01

There are two "Brian R. Bondy" commits (from mozilla-central, where the fix for 902761 has landed), one commit to backport some Mozilla patches that have r+ on the Mozilla side but have not yet been committed, and one commit that contains our changes to always use NSS and fix up various things.

The most recent commit (68a488187fde8a1f50e1e85e45b0f0beac15446c) will need to be discarded; it embeds a certificate that brade and I used for testing our own 4.5-alpha-1-ish builds.

Also, these changes cause the NSS certutil command to be built as well as signmar, which is a variant of the mar program that supports signing, verifying signatures, etc. (signmar uses NSS key and cert databases). We have some uncommitted builders/tor-browser-bundle changes that cause certutil and signmar to be included in the mar-tools-linux*.zip archives along with the dependent NSS and NSPR libraries. We will publish those patches soon along with other changes we plan to make to automate the signing process as much as possible (e.g., only prompt once for the NSS password).

Trac:
Keywords: N/A deleted, MikePerry201411R added
Status: assigned to needs_review

Here are a few more details on the signing key and cert.

The certificates files that get embedded in the updater are contained in these files within the tor-browser tree:

toolkit/mozapps/update/updater/dep1.der
toolkit/mozapps/update/updater/dep2.der
toolkit/mozapps/update/updater/nightly_aurora_level3_primary.der
toolkit/mozapps/update/updater/nightly_aurora_level3_secondary.der
toolkit/mozapps/update/updater/release_primary.der
toolkit/mozapps/update/updater/release_secondary.der
toolkit/mozapps/update/updater/xpcshellCertificate.der

dep1.der and dep2.der are no longer used; Mozilla used to use them for "depend" builds (maybe for their try server?).

The nightly_aurora_level3*.der files will be embedded in nightly builds. We need to decide what to do about those, if anything (at the moment, people who run our nightly builds do not expect to receive automated updates).

The release_*.der files will be embedded in our alpha, beta, and release builds. These are the most important ones.

The xpcshellCertificate.der is used by Mozilla for testing; it is embedded in all other builds, e.g., developer builds that lack an update channel.

I generated a test certificate by running these two commands:

./certutil -d .nss -N
./certutil -d .nss -S -x -g 3072 -n marsigner -s "CN=Tor Browser MAR signing key" -t,,

I exported it to a .der file via:

./certutil -d .nss -L -r -n marsigner -o marsigner.der

I then replaced both release_primary.der and release_secondary.der with the contents of marsigner.der (currently, a signature is considered "good" if the key associated with either the primary or the secondary certificates was used to create the signature; other policies could be implemented).

The build changes are here: https://gitweb.torproject.org/user/brade/tor-browser-bundle.git/commit/58acbf5b8d6df5a5e9f8f4760cd8c21836be4cef Please review.

I believe signing should be a reproducible process, but we will need to confirm.

The change to add the --createIncrementalMARs command line to update_responses looks good.

The other changes introduce a single makefile rule to generate the incremental mar files and sign them. I am wondering if we should separate the incremental mar files generation, and the signature, to allow a process like this:

build tor-browser
generate incremental mars
upload sha256sums.incrementals.txt of unsigned mar files
check that sha256sums.txt and sha256sums.incrementals.txt are matching
sign the mar files, update responses xml files and upload

Trac:
Keywords: MikePerry201411R deleted, TorBrowserTeam201411R added

Replying to boklm:

The change to add the --createIncrementalMARs command line to update_responses looks good.

The other changes introduce a single makefile rule to generate the incremental mar files and sign them. I am wondering if we should separate the incremental mar files generation, and the signature, to allow a process like this:

build tor-browser

generate incremental mars

upload sha256sums.incrementals.txt of unsigned mar files

check that sha256sums.txt and sha256sums.incrementals.txt are matching

sign the mar files, update responses xml files and upload

It would be simple to keep 'incrementals' as a separate Make target. The reason I put everything in one script was to make it less likely that things would happen in the wrong order.

gk or mikeperry: What do you think? What will the release process look like once we need to sign the MAR files?

Sign our MAR files

Child items 0

Activity