Opened 2 years ago

Closed 6 weeks ago

#25101 closed task (fixed)

Generate incremental mar files for nightly builds

Reported by: boklm Owned by: boklm
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, tbb-update, TorBrowserTeam201912R
Cc: boklm, mcs, brade, ln5, tbb-team Actual Points:
Parent ID: #18867 Points: 2
Reviewer: Sponsor:

Description

We have some nightly builds being done at http://f4amtbsowhix7rrf.onion/. However, there is no incremental mar files.

To avoid users downloading a complete mar file each day, we should generate incremental mar files from the previous days. I think we can start with incrementals for the previous 2 days.

Child Tickets

TicketStatusOwnerSummaryComponent
#32738closedboklmIf the linux64 mar-tools does not exist, we should use the other mar-tools availableApplications/Tor Browser

Change History (20)

comment:1 Changed 12 months ago by gk

Keywords: tbb-updater added

comment:2 Changed 12 months ago by gk

Keywords: tbb-update added; tbb-updater removed

Renaming keyword to make it a bit broader

comment:3 Changed 5 months ago by gk

Keywords: TorBrowserTeam201909 added
Points: 2

comment:4 Changed 4 months ago by pili

Keywords: TorBrowserTeam201910 added

comment:5 Changed 4 months ago by pili

Keywords: TorBrowserTeam201909 removed

comment:6 Changed 3 months ago by pili

Keywords: TorBrowserTeam201911 added; TorBrowserTeam201910 removed

Moving tickets to November 2019

comment:7 Changed 3 months ago by pili

Cc: tbb-team added
Owner: changed from tbb-team to boklm
Status: newassigned

Assigning tickets to boklm for the next few months

comment:8 Changed 2 months ago by boklm

Keywords: TorBrowserTeam201911R added; TorBrowserTeam201911 removed
Status: assignedneeds_review

In branch bug_25101_v4 there is a patch adding support for generating incrementals for the nightly build with make incrementals-nightly:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_25101_v4&id=fb35c9e33fea949a91e5461cb201810fbfe3a5d5

comment:9 Changed 2 months ago by gk

Keywords: TorBrowserTeam201911 added; TorBrowserTeam201911R removed
Status: needs_reviewneeds_revision

Just one nit:

  cd [% shell_quote(path(dest_dir)) %]/[%  c("version") %]

One whitespace too much between % and c.

comment:10 in reply to:  9 Changed 2 months ago by boklm

Keywords: TorBrowserTeam201911R added; TorBrowserTeam201911 removed
Status: needs_revisionneeds_review

Replying to gk:

Just one nit:

  cd [% shell_quote(path(dest_dir)) %]/[%  c("version") %]

One whitespace too much between % and c.

I fixed that in branch bug_25101_v5:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_25101_v5&id=d4460a92659808f8d29547197e23cb6defcf2f35

comment:11 Changed 2 months ago by gk

Resolution: fixed
Status: needs_reviewclosed

Thanks! Merged to master (commit d4460a92659808f8d29547197e23cb6defcf2f35).

comment:12 Changed 2 months ago by boklm

Resolution: fixed
Status: closedreopened

The patch which has been merged is adding support for the make incrementals-nightly command, however this is only the first part of this ticket.

The second part that is running it as part of our nightly builds so that the incrementals are generated.

comment:13 Changed 2 months ago by pili

Keywords: TorBrowserTeam201912 added

Moving tickets to December

comment:14 Changed 2 months ago by pili

Keywords: TorBrowserTeam201911R removed

comment:15 Changed 7 weeks ago by boklm

The two tor-browser-bundle-testsuite.git commits from branch bug_25101 should add generation of incremental mars:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle-testsuite.git/commit/?h=bug_25101&id=0d4c6e2db4ee8b9abde916d7be92f1f419af0e4d

With this commit from tor-browser-build.git updating the configuration file, to store the builds in a directory named with the version number:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_25101_v6&id=f350498852967787a3a54df2b4fb1de6ffe97e7e

As I have no other easy way to test that it is working, I have deployed this on the nightly build machine, to see if the incrementals are generated during the next nightly build.

comment:16 in reply to:  15 Changed 7 weeks ago by boklm

Replying to boklm:

As I have no other easy way to test that it is working, I have deployed this on the nightly build machine, to see if the incrementals are generated during the next nightly build.

So after a few fixups, the generation of incrementals worked for the linux64 build. It didn't work for the other archs as the script tools/update-responses/update_responses wants the linux64 mar-tools. I opened #32738 to fix that.

comment:17 Changed 7 weeks ago by boklm

Keywords: TorBrowserTeam201912R added; TorBrowserTeam201912 removed
Status: reopenedneeds_review

There are two tor-browser-bundle-testsuite.git patches for review in my branch bug_25101_v2:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle-testsuite.git/commit/?h=bug_25101_v2&id=4261c714139e0d9896d17c92af54747f34bd9f9b

And two tor-browser-build.git patches for review in my branch bug_25101_v8:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_25101_v8&id=2b0610b784bf0c97e4dde2647a375166d09c3033

The nightly builds from yesterday have been run with those patches:
http://f4amtbsowhix7rrf.onion/reports/r/tor-browser-2019-12-11/tor-browser_build.html

The incrementals were only generated for the linux64 build, but the others should be fixed by #32738.

comment:18 in reply to:  17 Changed 6 weeks ago by boklm

Replying to boklm:

The incrementals were only generated for the linux64 build, but the others should be fixed by #32738.

On the build from 2019-12-14, all incrementals were generated:
http://f4amtbsowhix7rrf.onion/reports/r/tor-browser-2019-12-14/tor-browser_build.html

comment:19 Changed 6 weeks ago by boklm

There are new patches for review, in branch bug_25101_v12:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_25101_v12&id=608fbb6f06237c0e0786b650433ca0c4cc203f72

With the new patches, instead of using the prune-old-builds script which was included in the testsuite repository as a git submodule, we copy it to tor-browser-build. And we then modify it to handle the new directory names based on the nightly version numbers.

In tor-browser-bundle-testsuite.git I added a patch to branch bug_25101_v2, removing the tools/prune-old-builds git submodule:
https://gitweb.torproject.org/user/boklm/tor-browser-bundle-testsuite.git/commit/?h=bug_25101_v2&id=ea0942f9da2bdbaf3f140f3baaed1bb56ee9af69

comment:20 Changed 6 weeks ago by gk

Resolution: fixed
Status: needs_reviewclosed

Okay, this all looks good to me (although I have a hard time testing this, so let's see what comes out here and fix up what needs to get fixed up).

I merged the tor-browser-build changes to master (commit 16e629f680bbbe07a408a3022c3b819acd833bd2) and the tor-browser-bundle-testsuite ones to its respective master as well (commits 679ee355425b0cb4bf77626a8b16755c1c43db60, 4261c714139e0d9896d17c92af54747f34bd9f9b, and ea0942f9da2bdbaf3f140f3baaed1bb56ee9af69).

Note: See TracTickets for help on using tickets.