Opened 13 months ago

Closed 11 months ago

Last modified 10 months ago

#27218 closed task (fixed)

Make rebundling faster by generating multiple bundles in parallel

Reported by: boklm Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, boklm201810, TorBrowserTeam201810R, tbb-backported
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Rebundling the whole 8.0a10 release on build-sunet-a.torproject.net is taking ~ 12 hours.

I think one part that is taking a lot of time is the creation of the mar/tar.xz/dmg/exe files of the bundles in all the languages, which is done sequentially, with the mar/tar/dmg/nsis tools probably using only one thread. We could probably make that part faster by generating multiple locales in parallel.

Child Tickets

Change History (8)

comment:1 Changed 12 months ago by gk

Keywords: TorBrowserTeam201810 added; TorBrowserTeam201809 removed

Moving tickets to October

comment:2 Changed 12 months ago by gk

Priority: MediumVery High

comment:3 Changed 12 months ago by boklm

Keywords: boklm201810 TorBrowserTeam201810R added; boklm201809 TorBrowserTeam201810 removed
Status: newneeds_review

There is a patch for review in branch bug_27218_v3:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27218_v3&id=787cc941f929af286ebbe595adee92051dfd566b

This branch is based on #28002, so it needs to be merged after #28002, or cherry-picked.

With this patch, instead of directly generating the bundles, we instead create one separate script to generate the bundle for each locale, and use run_scripts to run those scripts in parallel.

I compared the time to re-bundle an alpha with and without this patch on my build machine:

  • linux-x86_64. Without patch: 1:23:46 With patch: 22:25
  • linux-i686. Without patch: 1:27:57 With patch: 23:00
  • windows-x86_64. Without patch: 57:57 With patch: 17:04
  • windows-i686. Without patch: 1:02:18 With patch: 18:41
  • osx-x86_64. Without patch: 34:10 With patch: 11:28

So the total rebundle time without this patch was 5 hours and 23 minutes, and it was 1 hour 31 minutes with the patch.

I also compared the build output from this branch and current master. The result was the same, except for the en-US Windows installer, because it included the fix for #28002. I will now do a build with only the patch for #28002 to check that I get the same en-US bundle.

comment:5 Changed 11 months ago by gk

Keywords: TorBrowserTeam201810 added; TorBrowserTeam201810R removed
Status: needs_reviewneeds_revision

I am still testing the patch, but it looks good so far. Nice work! One nit in the commit message: s/parrallel/parallel.

comment:6 in reply to:  5 ; Changed 11 months ago by boklm

Summary: Make rebundling faster by generating multiple bundles in parrallelMake rebundling faster by generating multiple bundles in parallel

Replying to gk:

One nit in the commit message: s/parrallel/parallel.

Which branch are you looking at? There was this typo in bug_27218_v2, but it should already be fixed in bug_27218_v3 and bug_27218_v4.

comment:7 in reply to:  6 Changed 11 months ago by gk

Keywords: TorBrowserTeam201810R tbb-backport added; TorBrowserTeam201810 removed
Resolution: fixed
Status: needs_revisionclosed

Replying to boklm:

Replying to gk:

One nit in the commit message: s/parrallel/parallel.

Which branch are you looking at? There was this typo in bug_27218_v2, but it should already be fixed in bug_27218_v3 and bug_27218_v4.

Right, I was still looking at bug_27218_v2, thanks. I merged bug_27218_v4 to master (commit d0bd427db75d980be121b12f30f59f18c4e47325). Let's test it in the next alpha and apply it potentially to our stable branch as well later on.

comment:8 Changed 10 months ago by gk

Keywords: tbb-backported added; tbb-backport removed

Cherry-picked to maint-8.0 (commit 2d129f5c6b282c6e9e915339f7b4dfb0f54b5a80).

Note: See TracTickets for help on using tickets.