Opened 5 years ago

Closed 5 years ago

#13324 closed task (fixed)

Generation of incremental MAR files

Reported by: boklm Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Keywords: TorBrowserTeam201410, ff31-esr, tbb-4.5-alpha
Cc: mikeperry, gk, mcs, brade Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

The branch update-responses-2 from my git repository at https://git.torproject.org/user/boklm/tor-browser-bundle.git contains a few changes to:

  • make the generated response files deterministic
  • use MAR files from gitian/$version instead of a releases subdir
  • generate incremental MAR files

To be able to generate the incremental MAR files, it requires the tools from mar-tools-$os.zip (in directory gitian-builder/inputs after a build) to be extracted in a directory that is in the PATH.

To speed up the generation of the MAR files, it will generate them in parallel, as many as there are CPUs, or the number in the NUM_PROCS environment variable if it is defined.

The versions from which we generate incremental updates are defined in the config.yml file, in 'incremental_from'.

If you have been running the previous version of the script, you may need to install a few additional perl modules to run this one:
File::Temp IO::CaptureOutput File::Which Parallel::ForkManager

(on Debian: libio-captureoutput-perl libfile-which-perl libparallel-forkmanager-perl)

If everything is working as expected, the files generated in the htdocs directory (which include shasums of the mar files) should match exactly the ones I generated. I will attach to this ticket the content of my htdocs directory.

Child Tickets

Attachments (1)

update_incremental.tar.xz (12.1 KB) - added by boklm 5 years ago.
The htdocs dir I generated

Download all attachments as: .zip

Change History (16)

Changed 5 years ago by boklm

Attachment: update_incremental.tar.xz added

The htdocs dir I generated

comment:1 Changed 5 years ago by boklm

Status: newneeds_review

comment:2 Changed 5 years ago by mcs

Thanks for doing this. Kathy Brade and I reviewed your changes and everything looks OK to us (note that we are not Perl experts). Nice work!

You might consider adding a -q parameter when you exec make_incremental_update.sh (unless you think you need it for debugging). We added support for -q because the Mozilla scripts are very chatty.

I think we need a make target that runs the update_responses program and then verifies against other developers' builds (to check reproducibility).

comment:3 Changed 5 years ago by mcs

I forgot to mention that we did not try to reproduce the files you created (we would need to download a lot of mar files to try).

comment:4 in reply to:  2 Changed 5 years ago by boklm

Replying to mcs:

Thanks for doing this. Kathy Brade and I reviewed your changes and everything looks OK to us (note that we are not Perl experts). Nice work!

Thanks for reviewing this.

You might consider adding a -q parameter when you exec make_incremental_update.sh (unless you think you need it for debugging). We added support for -q because the Mozilla scripts are very chatty.

We are not displaying the output of the script, so it's not a problem that it is chatty.

I think we need a make target that runs the update_responses program and then verifies against other developers' builds (to check reproducibility).

Yes. The new MAR files are added to the gitian/$version directory, so running hash-bundles.sh again should update the sha256sums.txt with the new files. However, it will add a mismatch with other developers who did not generate the incremental MAR files, so I'm wondering if it would be better to add them to a separate sha256sums file.

comment:5 Changed 5 years ago by mikeperry

Keywords: ff31-esr added

comment:6 Changed 5 years ago by mcs

While testing updates using incremental MAR files generated by the revised update_responses script, brade and I discovered that symbolic links will not be created or removed during an update. This happens because the MAR file format itself does not have a way to store a symlink; we just add instructions to add or remove symlinks to the updatev2.manifest and updatev3.manifest files that are inside the MAR files.

That means that after the complete MAR files are extracted, no symlinks are present, Therefore, make_incremental_update.sh does not know to generate any symlink-related instructions. We could fix this by adding support for symlinks to the MAR file format itself, but that may be risky and/or complicated. My preferred solution is to modify the update_responses script to create symlinks when it extracts the complete MAR files. It can do this by looking inside updatev3.manifest for lines that begin with addsymlink. The full syntax is:

addsymlink "link-path" "target-path"

e.g.,

addsymlink "TorBrowser/Tor/PluggableTransports/TorBrowser.app.meek-http-helper/Contents/Resources" "../../../../../Contents/Resources"

(this creates a symlink named Resources in .../TorBrowser.app.meek-http-helper/Contents/ that points to the browser's Resources directory).

comment:7 in reply to:  6 Changed 5 years ago by gk

Status: needs_reviewneeds_revision

Replying to mcs:

My preferred solution is to modify the update_responses script to create symlinks when it extracts the complete MAR files. It can do this by looking inside updatev3.manifest for lines that begin with addsymlink. The full syntax is:

addsymlink "link-path" "target-path"

e.g.,

addsymlink "TorBrowser/Tor/PluggableTransports/TorBrowser.app.meek-http-helper/Contents/Resources" "../../../../../Contents/Resources"

(this creates a symlink named Resources in .../TorBrowser.app.meek-http-helper/Contents/ that points to the browser's Resources directory).

Sounds good to me.

comment:8 Changed 5 years ago by boklm

Sounds good to me too.

So the current TODO list on this branch is:

  • create the symlinks listed in updatev3.manifest after extracting the .mar files before generating the incremental mars
  • extract mar-tools-$os.zip from the latest build in a temporary directory added to the PATH
  • change the incremental .mar file names to end with ".incremental.mar" so they are easier to grep
  • update hash-bundles.sh to put the incremental mar files in a separate sha256sums file
  • add a makefile rule to generate the mar files and match them

comment:9 Changed 5 years ago by boklm

I added this commit which reads updatev3.manifest and creates the symlinks:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/commit/5620ed6cd3beff3edebefea51dddb942983d9ea7

If there is no updatev3.manifest but an updatev2.manifest is present, it will read this file instead.

comment:10 Changed 5 years ago by boklm

This commit extracts the mar-tools*.zip from the latest build:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/commitdiff/a9f8f0a3846998c4ad32f51e95c5bd8544e5d2e8

Currently we use the mar tools that were created during the Linux build to generate the incremental mars for all architectures.

Is it fine to use the Linux mar tools for everything, or should we use the ones created during the OSX and Windows builds to generate the OSX and Windows incremental mars ?

comment:11 in reply to:  10 Changed 5 years ago by gk

Replying to boklm:

This commit extracts the mar-tools*.zip from the latest build:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/commitdiff/a9f8f0a3846998c4ad32f51e95c5bd8544e5d2e8

Currently we use the mar tools that were created during the Linux build to generate the incremental mars for all architectures.

Is it fine to use the Linux mar tools for everything, or should we use the ones created during the OSX and Windows builds to generate the OSX and Windows incremental mars ?

That's fine as people building only, say, Windows bundles won't need to generate incremental MAR files. You could use the ones built during the Tor Browser build for Windows or OS X as well as these tools are HOST tools.

comment:12 Changed 5 years ago by boklm

Status: needs_revisionneeds_review

The branch update-responses-3 contains the commits from update-responses-2 rebased on master, and a new commit to update the gitian/*.sh scripts to handle an sha256sums.incrementals.txt file:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle.git/commitdiff/f1472e59d24945d3f05a740925a05c4bfac4ccc7

comment:13 Changed 5 years ago by mikeperry

Keywords: tbb-4.5-alpha added

comment:14 Changed 5 years ago by mcs

The patches look fine to me. Just a couple of small comments (not showstoppers):

  1. It would be nice to define a function in check-match.sh instead of duplicating the code that checks for a match.
  2. Also in check-match.sh, should a non-zero exit occur if no matches for the incremental mars are found (when a local sha256sums.incrementals.txt file exists)? I think this would be more consistent with behavior for the files listed in sha256sums.txt.

comment:15 Changed 5 years ago by mikeperry

Resolution: fixed
Status: needs_reviewclosed

Ok, I fixed mcs's issue #2, and put an XXX in the check-match scripts for the duplicate code. I went ahead and merged the result. I think we should try for incrementals for the next release (aiming for Thursday, for #13443).

Note: See TracTickets for help on using tickets.