[Asan] Crash in js::AsmJSModule::deserialize / DeserializeSig

Steps to reproduce:

1. Open current tor browser alpha, hardened (6.5a1)
2. surf on facebookcorewwwi.onion
3. click somewhere to start composing a message
4. as soon as you can, try to type (not sure this is required)

What happens: Tor browser crashes.

Date Time [notice] Bootstrapped 100%: Done
Date Time [notice] New control connection opened from 127.0.0.1.
Date Time [notice] New control connection opened from 127.0.0.1.
Time	addons.productaddons	ERROR	Request failed certificate checks: [Exception... "SSL is required and URI scheme is not https."  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: re[/gre/modules/CertUtils.jsm](/gre/modules/CertUtils.jsm) :: checkCert :: line 145"  data: no]
=================================================================
==5252==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f6dfe8c6000 at pc 0x7f6e4c3f2605 bp 0x7f6e009f23f0 sp 0x7f6e009f1ba0
READ of size 9437184 at 0x7f6dfe8c6000 thread T59 (DOM Worker)
ASAN:SIGSEGV
==5252==AddressSanitizer: while reporting a bug found another one. Ignoring.
Date Time [notice] Owning controller connection has closed -- exiting now.
Date Time [notice] Catching signal TERM, exiting cleanly.

added TorBrowserTeam201606 component::applications/tor browser owner::tbb-team priority::very high resolution::fixed severity::critical status::closed tbb-crash tbb-gitian type::defect labels

Trac:
tor-browser-hardened_gdb_reduced.log

A backtrace from gdb attached to tor browser hardened

Trac:
Priority: Medium to Very High
Keywords: N/A deleted, TorBrowserTeam201606 added
Severity: Major to Critical

Bug #19337 (moved) might be related.

Trac:
Severity: Critical to Major
Priority: Very High to Medium

Sorry, I accidentially broke priority/severity.

Trac:
Priority: Medium to Very High
Severity: Major to Critical

It seems I can't get my hand on a Facebook account which is kind of required to reproduce this bug. If anybody has some (dormant) one for testing please let me know (gk [@] torproject.org 35CD 74C2 4A9B 15A1 9E1A 81A1 9437 3AA9 4B7C 3223).

That said there are still things that can be done to try finding more out about this bug.

1. Does saving the whole website locally and using that one (via file:///) crash as well? If so, could I get a copy of that website (see above for contact details)?
2. Does 6.0a5-hardened (or 6.0) crash as well?
3. If you are adventurous, testing a vanilla Firefox ASan build (https://ftp.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64-asan/1450090940/) would be pretty interesting. Pointing it to your local tor should be all that is needed to reach facebookcorewwwi.onion.

Trac:
Cc: N/A to virtualritz, mcs, brade, arthuredelstein
Status: new to needs_information

Replying to gk:

2. Does 6.0a5-hardened (or 6.0) crash as well?

No, it doesn't. Crash happens just since the latest update from 6.0a5-hardened to 6.5a1-hardened or 6.0.1 (see bug #19337 (moved)).

Replying to gk:

3. If you are adventurous, testing a vanilla Firefox ASan build (https://ftp.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64-asan/1450090940/) would be pretty interesting. Pointing it to your local tor should be all that is needed to reach facebookcorewwwi.onion.

That doesn't reproduce the crash. But this FF version is ~6 months old, the change was probably introduced between FF 45 and FF ESR 45.2 (see comment #5 (closed))

Replying to gk:

1. Does saving the whole website locally and using that one (via file:///) crash as well? If so, could I get a copy of that website (see above for contact details)?

No, it doesn't crash. I used a vanilla FF 47 profile to save the page and opened it in TBB 6.5a1-hardened and it didn't crash.

Okay, thanks so much for narrowing this down. As far as I can see, there are three possible sources of the crash:

1. One of our (crash) fixes in 6.0.1 is causing this.
2. It is a bug in Mozilla's code itself.
3. It is a bug caused by new Mozilla code interfering badly with one of our patches.

I've uploaded a test build trying to rule out 1). It only omits the fixes for #19212 (moved) and #19187 (moved) and is the code otherwise the same as the branch used for Tor Browser 6.0.1.

https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-bug19400-hardened_ALL.tar.xz https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-bug19400-hardened_ALL.tar.xz.asc

Does this still crash for you?

Replying to gk:

Okay, thanks so much for narrowing this down. As far as I can see, there are three possible sources of the crash:

1. One of our (crash) fixes in 6.0.1 is causing this.
2. It is a bug in Mozilla's code itself.
3. It is a bug caused by new Mozilla code interfering badly with one of our patches.

I've uploaded a test build trying to rule out 1). It only omits the fixes for #19212 (moved) and #19187 (moved) and is the code otherwise the same as the branch used for Tor Browser 6.0.1.

https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-bug19400-hardened_ALL.tar.xz https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-bug19400-hardened_ALL.tar.xz.asc

Does this still crash for you?

It actually only omits the fix for #19187 (moved). The other build omitting #19212 (moved) is coming shortly.

Here we go:

https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-bug19400_v2-hardened_ALL.tar.xz https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-bug19400_v2-hardened_ALL.tar.xz.asc

has both patches omitted.

The build in comment #10 (closed) doesn't crash. Looks like you found the cause.

Replying to cypherpunks:

The build in comment #10 (closed) doesn't crash. Looks like you found the cause.

Thanks. Could you test the build in comment:8 as well? That way we can be sure which one of the two fixes we shipped with 6.0.1 is causing this.

The build in comment #8 (closed) doesn't crash either.

Thanks! I've confirmed by manual testing that the build in comment:8 only omits the patch for #19187 (moved) and the build in comment:10 omits both. This seems to indicate that the backport of the ASan crash fix is somehow involved which does not make much sense to me. At least according to Mozilla this issue should only affect chrome code. Hm.

Hi cypherpunk: could you test one final build (I think we are pretty close to getting this reproduced) and see whether it is crashing for you:

https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-bug19400_v3-hardened_ALL.tar.xz https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-bug19400_v3-hardened_ALL.tar.xz.asc

Thanks!

Okay, I think I found out some important things:

0. I assume the crashes on mega.nz some of our users observe are caused by the same underlying flaw. I attach a stacktrace from a mega.nz related crash that should be similar enough to justify treating it as the same bug.

1. The first crucial bit that was missing so far was that updating must be involved to reproduce the problem. I.e. I am pretty sure that using a clean, new 6.0.1 or 6.5a1 is working fine (can you confirm this, cypherpunk?). That would explain our problems reproducing the crash I guess.

2. The second crucial bit is that one must have visited e.g. mega.nz once before the update (I guess this applies to Facebook as well but I don't have an account to verify this). "Ideally", you have mega.nz open, apply your update and visit mega.nz again and it crashes.

3. The problem is confined to the Tor Browser profile. More specifically, for some reason there is a https+++mega.nz folder in profile.default/storage/temporary that contains binary asmjs/moduleN files which are different between a clean new profile used to visit mega.nz once and a profile that contains them after the update. Not sure whether that difference is enough to explain the crashes (probably not) but removing https+++mega.nz solves the problem for me.

4. This is no issue with a vanilla Firefox. I tried applying my STR to Firefox 45.1.1esr/45.2.0esr and did not hit this problem.

Things I still don't understand are

a) What role exactly plays the updater here? b) How can it be that these asmjs modules are saved to disk given that we are in PBM? c) Which of our patches is actually causing this problem given 4)?

Trac:
mega_sigsev