Okay, I tested quite a bit. Here is the scenarios I covered:
old=BZIP2 new=LZMA
Signing old and new MAR file based on latest esr60 tor browser code with currently used cert
a) used esr60 nightly (just tested old MAR compression)
ERROR: Unknown signature algorithm ID.
ERROR: Unknown signature algorithm ID.
b) used esr52 alpha
i) old worked, updated to esr60 nightly
ii) new did not work, did essentially nothing and gave no errors
Signing old and new MAR file based on latest esr60 tor browser code with new cert
a) esr52 nightly (just tested with old MAR compression)
ERROR: Unknown signature algorithm ID 2.
ERROR: Unknown signature algorithm ID 2.
Taking the result from 1b)i
a) applying old with old cert
ERROR: Unknown signature algorithm ID.
ERROR: Unknown signature algorithm ID.
b) applying new with old cert
ERROR: Unknown signature algorithm ID.
ERROR: Unknown signature algorithm ID.
c) applying old with new cert
ERROR: Error verifying signature.
ERROR: Error verifying signature.
d) applying new with new cert
ERROR: Error verifying signature.
ERROR: Error verifying signature.
Everything looks good except in 3c) and 3d). I had expected that in 3c) nothing happens and in 3d) the update with the new cert works. I tried to debug that and came earlier to the conclusion that I need to replace the nightly certs with the new certs as well for testing purposes. That's already included.
Now, I wonder what is going on. If I use the new mar-tools and create a new nssdb importing the public part of the new cert into it using
certutil -A -d nssdb -n marsigner -t,, -i ../../tor-browser/toolkit/mozapps/update/updater/release_primary.der
and doing now a verification of the signature of the two MAR files used in 3c) and 3d) the check succeeds. I.e.:
returns nothing while importing the second new cert and checking against that one fails (which is expected as the key behind the first one signed the MAR files).
So, this makes me feel optimistic. Still, it would be nice to understand why the update in 3d) failed and why there was a signature verification error in 3c).
Okay, one additional bit: I can't even apply the signed MAR file to the nightly which it is built from but it seems to me that should be possible. I get "ERROR: Error verifying signature." in this case as well.
I essentially used the key generation command as specified in our KeyGeneration doc, just adjusted to the new hash length. I.e. certutil -d nssdb -S -x -g 4096 -Z SHA384 -n marsigner -s "CN=Tor Browser MAR signing key" -t,,
For signing I used the old script we had in the Gitian days, signmars.sh, changed to check for the new cert9.db and to make sure it is using the new mar-tools (i.e. those built with the esr60 nightly).
If you want to inspect the .der certs, I used bug_26045 in my public tor-browser repo for building.
It seems mcs and brade found the problem: when building the nightly not the nightly certificates are included into the build but dep1.der and dep2.der`. The code responsible for that is
and we set the update channel to default for nightlies (see the tor-browser-build repo projects/firefox/config). After copying the new certs over dep1.der and dep2.der scenario 3c) and 3d) in comment:6 behave as epxected: in the former nothing happens after the successful signature verification and in the latter the update works. Thus, we are good with the new key.