Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#14959 closed defect (fixed)

Download missing MAR files for incrementals

Reported by: mikeperry Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Keywords: tbb-gitian, TorBrowserTeam201503R
Cc: boklm, gk, mcs, brade Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We should download the MARs needed to produce an incremental from the appropriate release directory in https://archive.torproject.org/tor-package-archive/torbrowser/.

This may be a little tricky, because soon those MAR files will be signed, so we may need to unsign them before using them to produce the release MARs?

Child Tickets

Change History (10)

comment:1 in reply to:  description ; Changed 5 years ago by gk

Cc: gk added
Keywords: tbb-gitian added; gitian removed

Replying to mikeperry:

This may be a little tricky, because soon those MAR files will be signed, so we may need to unsign them before using them to produce the release MARs?

I don't think so. IIRC mar -x is just extracting the content of the MAR files which should not get altered by appending a signature.

comment:2 in reply to:  1 Changed 5 years ago by mcs

Replying to gk:

I don't think so. IIRC mar -x is just extracting the content of the MAR files which should not get altered by appending a signature.

That is correct. The signatures are in a header block along with some other info, but mar -x just skips past that stuff, so the presence or absence of a signature should make no difference (there is an offset to the MAR file index near the start of the header).

comment:3 Changed 5 years ago by mcs

Cc: mcs brade added

comment:4 Changed 5 years ago by boklm

Keywords: TorBrowserTeam201503R added
Status: newneeds_review

The branch bug14959-v1 in my repository has a patch to download missing versions:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/commit/?h=bug14959-v1&id=418d84e33db2fbf411132f28c5d099ab4ff67eb7

With this patch, when the directory for a version required to generate incremental MARs is not present, we download its sha256sums.txt from archive.tpo, check its signature, then download all .mar files and check that their checksum match the checksum from sha256sums.txt.

It seems we didn't have the gpg public key used for signing Tor Browser releases in the repository, so the patch adds it in gitian/gpg/torbrowser.gpg.

comment:5 Changed 5 years ago by mikeperry

Status: needs_reviewneeds_revision

The sha256sums.txt verification of the .mar files is going to fail here. To avoid repeated re-signging by all of the official builders, we decided that both of the sha256sums.txt files will contain shasums of the *unsigned* .mar files. So this means that you need to do 'signmar -r' (from the mar-tools) on each .mar file before verifying its sha256sum.

I'm surprised you didn't run into this. Did you test this on a 4.5 build or a 4.0 build? We will only use this for 4.5 and later builds, so you don't need to worry about supporting the unsigned mar case at this point.

comment:6 Changed 5 years ago by boklm

Ah, I forgot about this. I tested by adding version 4.0.4 in incremental_from, which is why I didn't see this. I will update the patch to do the "signmar -r".

Now testing with 4.5a4, I see the mismatching shasums. We also need to fix the signature of the sha256sums.txt file for version 4.5a4 for this to work for the next release.

comment:7 Changed 5 years ago by boklm

I uploaded a new version of the patch in branch bug14959-v2:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/commit/?h=bug14959-v2

Now if the shasum doesn't match, we use "signmar -r" to remove the signature on the MAR file and check the shasum again. So it should work for both signed and unsigned MARs (the unsigned case might be useful for the generation of 4.0 -> 4.5 incrementals).

To test this on 4.5a4, I had to uncomment the lines checking the gpg signature of the sha256sums.txt file, as the signature on 4.5a4 is wrong at the moment.

comment:8 Changed 5 years ago by boklm

Status: needs_revisionneeds_review

comment:9 Changed 5 years ago by mikeperry

Resolution: fixed
Status: needs_reviewclosed

Ok, I fixed the signature on 4.5a4. It now looks good to me. I pushed it to master.

Thanks!

comment:10 Changed 5 years ago by mikeperry

(Oh, and btw, the sig was broken because the sha256sums.txt file I signed had a sha256sum of a sha256sums.txt.asc in it already.. I think because I signed one of the earlier builds and accidentally left it in the build dir, perhaps? Anyway, I signed the official sha256sums.txt file, which matched my old one save for the inclusion of that file)

Note: See TracTickets for help on using tickets.