Opened 4 years ago

Closed 3 years ago

#17858 closed defect (fixed)

Creating incremental MAR files for the hardened builds is broken

Reported by: gk Owned by: mcs
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-gitian, tbb-hardened, GeorgKoppen201512, TorBrowserTeam201608R
Cc: boklm, brade, mcs Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

ASan seems to break the creation of incremental MAR files. mbsdiff is basically frozen while trying to cope with libxul. Using a non-ASan MAR tools archive is working.

Child Tickets

Change History (18)

comment:1 Changed 4 years ago by mcs

Does mbsdiff appear to be in a CPU intensive loop? Or maybe is it using an excessive amount of virtual memory?

It sounds like the non-ASan mbsdiff is working and can produce a diff from two ASan-built copies of libxul? That implies that the problem is not just something like "mbsdiff cannot handle really large binaries" but some flaw in mbsdiff that is caused by it being compiled with ASan.

comment:2 Changed 4 years ago by mcs

Cc: brade mcs added

comment:3 Changed 4 years ago by boklm

Does mbsdiff appear to be in a CPU intensive loop? Or maybe is it using an excessive amount of virtual memory?

Yes, it appears to be in a CPU intensive loop, with no IO-wait. It is also using an excessive amount of virtual memory (top says 20.001t).

comment:4 in reply to:  3 Changed 4 years ago by teor

Replying to boklm:

Does mbsdiff appear to be in a CPU intensive loop? Or maybe is it using an excessive amount of virtual memory?

Yes, it appears to be in a CPU intensive loop, with no IO-wait. It is also using an excessive amount of virtual memory (top says 20.001t).

Increased memory and CPU use a known issue with ASan builds.

comment:5 Changed 4 years ago by gk

Keywords: GeorgKoppen201512 added

comment:6 Changed 4 years ago by mcs

gk, please let me know if you need Kathy and me to look at this. I suppose the first step is to run the mbsdiff command under gdb and see what it is doing. It is possible that diffing libxul just causes a bunch of CPU and memory to be consumed, which is made much worse by ASan.

comment:7 Changed 4 years ago by mcs

Also see http://www.daemonology.net/bsdiff/ for some info about expected memory usage.

comment:8 Changed 4 years ago by gk

Keywords: TorBrowserTeam201601 added; TorBrowserTeam201512 removed

Tickets for Jan 2016.

comment:9 Changed 4 years ago by gk

Owner: changed from tbb-team to mcs
Status: newassigned

comment:10 Changed 4 years ago by gk

Keywords: TorBrowserTeam201602 added; TorBrowserTeam201601 removed

Putting stuff on the radar for February.

comment:11 Changed 4 years ago by gk

Keywords: TorBrowserTeam201603 added; TorBrowserTeam201602 removed

comment:12 Changed 4 years ago by gk

Keywords: TorBrowserTeam201604 added; TorBrowserTeam201603 removed

comment:13 Changed 4 years ago by gk

Keywords: TorBrowserTeam201605 added; TorBrowserTeam201604 removed

Moving tickets

comment:14 Changed 3 years ago by gk

Keywords: TorBrowserTeam201606 added; TorBrowserTeam201605 removed

comment:15 Changed 3 years ago by gk

Keywords: TorBrowserTeam201607 added; TorBrowserTeam201606 removed

comment:16 Changed 3 years ago by gk

Keywords: TorBrowserTeam201608 added; TorBrowserTeam201607 removed

Moving items to August 2016.

comment:17 Changed 3 years ago by mcs

Keywords: TorBrowserTeam201608R added; TorBrowserTeam201608 removed
Status: assignedneeds_review

My ASan-enabled mbsdiff ran for hours without completing while trying to generate diffs for libXUL (eventually I gave up and killed it). The problem appeared to be CPU usage, not VM exhaustion.

I think the simplest solution is to avoid using ASan when generating the MAR files. Here is a patch:
https://gitweb.torproject.org/user/brade/tor-browser.git/commit/?h=bug17858-01&id=2126d94c8191880bdb20370566c5acc13b817ff3

When I tested this, my system took 187 seconds to finish a 'make incrementals-alpha' command and produced a tor-browser-linux64-6.5a1-hardened-6.5a2-hardened_ALL.incremental.mar file of size 23MB (the full MAR file is 110MB).

comment:18 Changed 3 years ago by gk

Resolution: fixed
Status: needs_reviewclosed

Nice find! This is fixed with commit 37e6040d25803771bfe4ef294d90d4e13e4eebb0 on tor-browser-45.3.0esr-6.5-1.

Note: See TracTickets for help on using tickets.