Make rebundling faster by generating multiple bundles in parallel

Rebundling the whole 8.0a10 release on build-sunet-a.torproject.net is taking ~ 12 hours.

I think one part that is taking a lot of time is the creation of the mar/tar.xz/dmg/exe files of the bundles in all the languages, which is done sequentially, with the mar/tar/dmg/nsis tools probably using only one thread. We could probably make that part faster by generating multiple locales in parallel.

added TorBrowserTeam201810R boklm201810 component::applications/tor browser owner::tbb-team priority::very high resolution::fixed severity::normal status::closed tbb-backported tbb-rbm type::task labels

Moving tickets to October

Trac:
Keywords: TorBrowserTeam201809 deleted, TorBrowserTeam201810 added

Trac:
Priority: Medium to Very High

There is a patch for review in branch bug_27218_v3: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27218_v3&id=787cc941f929af286ebbe595adee92051dfd566b

This branch is based on #28002 (moved), so it needs to be merged after #28002 (moved), or cherry-picked.

With this patch, instead of directly generating the bundles, we instead create one separate script to generate the bundle for each locale, and use run_scripts to run those scripts in parallel.

I compared the time to re-bundle an alpha with and without this patch on my build machine:

• linux-x86_64. Without patch: 1:23:46 With patch: 22:25
• linux-i686. Without patch: 1:27:57 With patch: 23:00
• windows-x86_64. Without patch: 57:57 With patch: 17:04
• windows-i686. Without patch: 1:02:18 With patch: 18:41
• osx-x86_64. Without patch: 34:10 With patch: 11:28

So the total rebundle time without this patch was 5 hours and 23 minutes, and it was 1 hour 31 minutes with the patch.

I also compared the build output from this branch and current master. The result was the same, except for the en-US Windows installer, because it included the fix for #28002 (moved). I will now do a build with only the patch for #28002 (moved) to check that I get the same en-US bundle.

Trac:
Keywords: boklm201809, TorBrowserTeam201810 deleted, boklm201810, TorBrowserTeam201810R added
Status: new to needs_review

bug_27218_v4 is rebased on bug_28002_v2: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27218_v4&id=dfa0cc46fc0ef70f78313a651bffe178b4d1c4ee

I am still testing the patch, but it looks good so far. Nice work! One nit in the commit message: s/parrallel/parallel.

Trac:
Keywords: TorBrowserTeam201810R deleted, TorBrowserTeam201810 added
Status: needs_review to needs_revision

Replying to gk:

One nit in the commit message: s/parrallel/parallel.

Which branch are you looking at? There was this typo in bug_27218_v2, but it should already be fixed in bug_27218_v3 and bug_27218_v4.

Trac:
Summary: Make rebundling faster by generating multiple bundles in parrallel to Make rebundling faster by generating multiple bundles in parallel

Replying to boklm:

Replying to gk:

One nit in the commit message: s/parrallel/parallel.

Which branch are you looking at? There was this typo in bug_27218_v2, but it should already be fixed in bug_27218_v3 and bug_27218_v4.

Right, I was still looking at bug_27218_v2, thanks. I merged bug_27218_v4 to master (commit d0bd427db75d980be121b12f30f59f18c4e47325). Let's test it in the next alpha and apply it potentially to our stable branch as well later on.

Trac:
Keywords: TorBrowserTeam201810 deleted, tbb-backport, TorBrowserTeam201810R added
Resolution: N/A to fixed
Status: needs_revision to closed

Cherry-picked to maint-8.0 (commit 2d129f5c6b282c6e9e915339f7b4dfb0f54b5a80).

Trac:
Keywords: tbb-backport deleted, tbb-backported added

closed

mentioned in issue #28002 (moved)

moved to tpo/applications/tor-browser#27218 (closed)