Investigate usage of alternate memory allocators and memory hardening options
One thing we can do to improve security of TBB is to build it with an alternate semi-hardened malloc implementation that attempts to randomize the allocation pattern and performs some minimal checks to guard against heap overflows an reference count issues in Firefox (perhaps by also enabling some additional reference count debugging features already in Firefox).
Such allocator behavior may make exploitation of various use-after-free vulnerabilities more difficult, as it would be harder to predict the location of reallocated regions during exploitation in order to get a target object to overlay an incorrectly freed object.
The downside is this will likely come at the performance costs of loss of locality, increased fragmentation, and additional overhead of reference count checks, but this may be an acceptable cost for improved hardening against exploits.
The first question is: are there any existing drop-in replacement memory allocators we can use in place of Firefox's current jemalloc implementation?
The second question is will any of the Firefox refcounting checks actually help, or will they just increase runtime for no real benefit?
Activity
The memory.free_dirty_pages pref may have some effect here in terms of encouraging the default jemalloc allocator to minimize the use of free lists, but according to https://bugzilla.mozilla.org/show_bug.cgi?id=805855, we may also have to emit the 'memory-pressure' observer event for it to have effect.
-
It turns out Chrome has a special hardened allocator called ParitiionAlloc for this purpose: https://code.google.com/p/chromium/issues/detail?id=340947 https://chromium.googlesource.com/chromium/blink/+/master/Source/wtf/PartitionAlloc.h https://chromium.googlesource.com/chromium/blink.git/+/master/Source/wtf/PartitionAlloc.cpp
Trac:
Cc: gk, nickm, mcs, brade, ioerror to gk, nickm, mcs, brade, ioerror, tom@ritter.vg-
I spoke with cevans and he helped me pull the newest version of ctmalloc from http://src.chromium.org/blink/trunk/Source/wtf/PartitionAlloc.h, which has the usable_size function needed. I commented out a bunch of macros, successfully built, and it this may get past the sqlite3 problem. It crashed in a new spot though. Progress.
Trac:
Cc: gk, nickm, mcs, brade, ioerror, tom@ritter.vg to gk, nickm, mcs, brade, ioerror, tom@ritter.vg, arthuredelstein-
Just a note to mention that the memory tags from about:memory may be useful for initial partitioning with PartitionAlloc, but we may need to special case some of that. For instance, ArrayBuffers are a popular target for setting up UAF exploitation, and probably need a special partition, as do certain DOM/iframe objects (see http://robert.ocallahan.org/2010/10/mitigating-dangling-pointer-bugs-using_15.html).
-
For lack of a better place to store these, here are some good blog posts on IE's technique(s):
- http://researchcenter.paloaltonetworks.com/2014/07/beginning-end-use-free-exploitation/
- https://labs.mwrinfosecurity.com/blog/2014/06/20/isolated-heap-friends---object-allocation-hardening-in-web-browsers/
- http://blog.fortinet.com/post/is-use-after-free-exploitation-dead-the-new-ie-memory-protector-will-tell-you
Trac:
Cc: gk, nickm, mcs, brade, ioerror, tom@ritter.vg, arthuredelstein to gk, nickm, mcs, brade, ioerror, tom@ritter.vg, arthuredelstein, intrigeri-
As an update.
Mozilla is working on moving to jemalloc3 which will have a partitioned heap. I believe they are hoping to have this for the next ESR, but it's not guaranteed. More: http://guilherme-pg.com/2014/10/15/Partitioned-heap-in-Firefox-pt1.html
I have a ctmalloc build compiling on Linux 4.5 alpha. My patches are: https://github.com/tomrittervg/tor-browser/commits/tor-browser-31.2.0esr-4.5-1-build5-ctmalloc https://github.com/tomrittervg/tor-browser-builder/commits/tjr-ctmalloc
Unfortunately, my implementation of memalign is wrong I need to hack at PartitionAlloc to support that, which will be a bit risky and tricky. Also, once jemalloc3 comes along, the advantages of using PartitionAlloc are much fewer, although some of the work (e.g. random partitioning based on callsite) could likely be ported over as defense in depth.
-
Two related Mozilla bugs worth tracking: https://bugzilla.mozilla.org/show_bug.cgi?id=1052573 https://bugzilla.mozilla.org/show_bug.cgi?id=1052575
There's also a tree of jemalloc3 bugs here: https://bugzilla.mozilla.org/show_bug.cgi?id=762449
-
Gave it a shot in https://lists.torproject.org/pipermail/tbb-dev/2015-October/000309.html
An update: I tried building tor-browser.git with --enable-jemalloc flag, and it was able to build and run with no problem. This activates jemalloc4. As Tom Ritter points out in his experiments, jemalloc potentially makes it possible to partition the heap into "arenas", but it's going to require a lot of changes that ideally will be carried out by Mozilla.
Replying to mikeperry:
One thing we can do to improve security of TBB is to build it with an alternate semi-hardened malloc implementation that attempts to randomize the allocation pattern and performs some minimal checks to guard against heap overflows an reference count issues in Firefox (perhaps by also enabling some additional reference count debugging features already in Firefox).
Such allocator behavior may make exploitation of various use-after-free vulnerabilities more difficult, as it would be harder to predict the location of reallocated regions during exploitation in order to get a target object to overlay an incorrectly freed object.
The downside is this will likely come at the performance costs of loss of locality, increased fragmentation, and additional overhead of reference count checks, but this may be an acceptable cost for improved hardening against exploits.
The first question is: are there any existing drop-in replacement memory allocators we can use in place of Firefox's current jemalloc implementation?
The second question is will any of the Firefox refcounting checks actually help, or will they just increase runtime for no real benefit?
Trac:
Username: waxmiguel
I backported the patch from https://bugzilla.mozilla.org/show_bug.cgi?id=1277704 (and previous related patches) and ran a try build with them based on ESR45:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a567af3611e
There is a lot of test bustage visible. But if one compares that to a vanilla ESR45 build
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d3fa3b49cda3
then it seems to me the test bustage is not related to enabling jemalloc4. Moreover, (intermittent) test failures related to jemalloc 4.2.1 were reported for x86-64 win which we don't support currently. Thus, I think for the alpha series it could make sense to backport 4.2.1 and start playing with hardening it.
To start playing: https://bugzilla.mozilla.org/show_bug.cgi?id=1262164. Only for hardened, please: we have enough weird crashes on alphas...
Here is a patch (please review) that uses jemalloc4 in Tor Browser ESR45 (Linux). It also activates jemalloc redzones, and aborts if the redzones are found to have been corrupted.
https://github.com/arthuredelstein/tor-browser/commit/10281+2
(I'm still working on two alternative patches that (1) uses the DieHarder memory allocator and (2) uses jemalloc4 with randomized arenas, but I think we may want to give the redzone patch a try for now.)
Trac:
Keywords: TorBrowserTeam201612 deleted, TorBrowserTeam201612R added-
Two other things I am wondering: should we have this on the hardened series first, only? (sounds like a good idea to me) Are we okay with the jemalloc4 ESR45 ships or do we think we should need to backport the patches already landed on m-c (or are about to land)? There was work done in comment:29 for that already and I am wondering about the severity of all the bugfixes that landed in jemalloc4 meanwhile...
Replying to gk:
How did you test this? Do you have a test site showing the differences between the current behavior and the intended one?
I don't have a test page, but I made a test patch that deliberately causes a buffer overflow. The patch is here:
https://github.com/arthuredelstein/tor-browser/commits/10281+2_check
In this code, I malloc a 5-byte array, but write 32 bytes:
+ char* test = (char*) malloc(5); + for (int i = 0; i < 32; ++i) { + test[i] = i; + printf("test[%d] value: %d\n", i, (int) test[i]); + } + free(test);(I arbitrarily put this code in a function that always runs on startup, loading of the XUL window.)
arthur@instance-3 ~/tor-browser (10281+2_check)> ./mach run 0:02.27 /home/arthur/tor-browser/obj-x86_64-unknown-linux-gnu/dist/bin/firefox -no-remote -profile /home/arthur/tor-browser/obj-x86_64-unknown-linux-gnu/tmp/scratch_user test[0] value: 0 test[1] value: 1 test[2] value: 2 test[3] value: 3 test[4] value: 4 test[5] value: 5 test[6] value: 6 test[7] value: 7 test[8] value: 8 test[9] value: 9 test[10] value: 10 test[11] value: 11 test[12] value: 12 test[13] value: 13 test[14] value: 14 test[15] value: 15 test[16] value: 16 test[17] value: 17 test[18] value: 18 test[19] value: 19 test[20] value: 20 test[21] value: 21 test[22] value: 22 test[23] value: 23 test[24] value: 24 test[25] value: 25 test[26] value: 26 test[27] value: 27 test[28] value: 28 test[29] value: 29 test[30] value: 30 test[31] value: 31 <jemalloc>: Corrupt redzone 0 bytes after 0x7ff663ea6570 (size 8), byte=0x8 <jemalloc>: Corrupt redzone 1 byte after 0x7ff663ea6570 (size 8), byte=0x9 <jemalloc>: Corrupt redzone 2 bytes after 0x7ff663ea6570 (size 8), byte=0xa <jemalloc>: Corrupt redzone 3 bytes after 0x7ff663ea6570 (size 8), byte=0xb <jemalloc>: Corrupt redzone 4 bytes after 0x7ff663ea6570 (size 8), byte=0xc <jemalloc>: Corrupt redzone 5 bytes after 0x7ff663ea6570 (size 8), byte=0xd <jemalloc>: Corrupt redzone 6 bytes after 0x7ff663ea6570 (size 8), byte=0xe <jemalloc>: Corrupt redzone 7 bytes after 0x7ff663ea6570 (size 8), byte=0xf <jemalloc>: Corrupt redzone 8 bytes after 0x7ff663ea6570 (size 8), byte=0x10 <jemalloc>: Corrupt redzone 9 bytes after 0x7ff663ea6570 (size 8), byte=0x11 <jemalloc>: Corrupt redzone 10 bytes after 0x7ff663ea6570 (size 8), byte=0x12 <jemalloc>: Corrupt redzone 11 bytes after 0x7ff663ea6570 (size 8), byte=0x13 <jemalloc>: Corrupt redzone 12 bytes after 0x7ff663ea6570 (size 8), byte=0x14 <jemalloc>: Corrupt redzone 13 bytes after 0x7ff663ea6570 (size 8), byte=0x15 <jemalloc>: Corrupt redzone 14 bytes after 0x7ff663ea6570 (size 8), byte=0x16 <jemalloc>: Corrupt redzone 15 bytes after 0x7ff663ea6570 (size 8), byte=0x17So the free(...) call resulted in jemalloc detecting that a redzone had been overwritten. Note that 16 bytes of overwriting were detected, because allocation is chunked to 16 bytes here.
Replying to gk:
Two other things I am wondering: should we have this on the hardened series first, only? (sounds like a good idea to me)
That's OK with me! I'm not sure how it will interact with ASan, though.
Are we okay with the jemalloc4 ESR45 ships or do we think we should need to backport the patches already landed on m-c (or are about to land)? There was work done in comment:29 for that already and I am wondering about the severity of all the bugfixes that landed in jemalloc4 meanwhile...
Unfortunately, when I include just the 4 patches from comment:29 and use
MOZ_JEMALLOC4=1, I get build errors. I think jemalloc didn't actually build in comment:29, because the patches that implemented --enable-jemalloc=4 weren't included. I'm working on backporting more patches to try to get the latest jemalloc to build. It's a bit complex.Replying to arthuredelstein:
Unfortunately, when I include just the 4 patches from comment:29 and use
MOZ_JEMALLOC4=1, I get build errors.Apparently I was doing something wrong here. When I wiped everything again, the build worked fine with the 4 patches from comment:29. Sorry for my confusion!
Here's a new version, using the 4 patches from comment:29 to upgrade to jemalloc 4.3.1. Then my additional patch enables jemalloc4 as the memory allocator and activates aborts on redzones.
https://github.com/arthuredelstein/tor-browser/commits/10281+3
And here is the same extra test patch, again. I get the same results as in comment:39:
https://github.com/arthuredelstein/tor-browser/commit/10281+3_check
Something Yawning pointed out is that redzones will no longer be available in jemalloc 5: https://github.com/jemalloc/jemalloc/issues/369 But given that Firefox 52 still uses jemalloc 4.x, we should be OK for all of 2017. I see redzones as a stopgap while we continue to look for better options.
-
Replying to arthuredelstein:
Replying to gk:
Two other things I am wondering: should we have this on the hardened series first, only? (sounds like a good idea to me)
That's OK with me! I'm not sure how it will interact with ASan, though.
Hm. Looking closer at what we have in
.mozconfig-asanit seems we might want to try the alpha series:ac_add_options --enable-address-sanitizer ac_add_options --disable-jemallocUsing the hardened series instead would at least merit some investigation first I think but we are probably lacking that time right now.
mcs/brade: could you give the patches a look as well?
I did not look at the backported patches; do I need to? They are large; I guess if they merged cleanly they are probably okay.
Enabling the redzones and abort options might make TB slower and less stable. But I guess finding lurking bugs is a good thing as long as performance is not reduced too much.
Replying to mcs:
I did not look at the backported patches; do I need to? They are large; I guess if they merged cleanly they are probably okay.
I think those are OK -- they're just pulling in the update patches from jemalloc.
Enabling the redzones and abort options might make TB slower and less stable. But I guess finding lurking bugs is a good thing as long as performance is not reduced too much.
Both these comments are true. Seems like the alphas is the right place until we observe performance and how crashy the browser is. One thing I can try in the near future is to push this patch to Mozilla's try server and look at crashes and talos (performance) results.
Trac:
Parent: N/A to #20955 (moved)-
This ticket is done. Further items to work on will be detailed in child tickets to #20955 (moved).
Trac:
Resolution: N/A to fixed
Status: needs_review to closed -
I'm pasting this here from an email I wrote in Sept 2016.
The original problem with PartitionAlloc was the lack of memalign functions and the need to implement them. They don't seem to have been added, so that problem persists.
See https://github.com/struct/HardenedPartitionAlloc/issues/1
There seem to be some security features that are in OpenBSD's allocator that aren't/can't be in PartitionAlloc, but not all of them.
- No inline metadata - PartitionAlloc has this feature as well.
- "It is guaranteed to abort for pointers that are not active malloc allocations." - not sure about this, but http://struct.github.io/partition_alloc.html has a patch for PartitionAlloc that does check the freelist for double frees. I'm not sure if this is a comprehensive equality of features though
- "sets the allocator to abort on out-of-memory by default" - This is probably pretty easy to do. (Just a NULL check and an abort() no?)
- "Fine-grained randomization is performed for small allocations by choosing a random pool to satisfy requests" - okay, but this is like 'choose a random partition' except not as powerful, cause you're in the same heap
- "and then choosing a random free slot within a page provided by that pool" - PartitionAlloc does not have this, but it could. Here's a patch: http://struct.github.io/partition_alloc.html
- "Freed small allocations are quarantined before being put back into circulation via a randomized delayed allocation pool" - okay, PArtitionAlloc doesn't have this
- "CopperheadOS uses a ring buffer..." PartitionAlloc doesn't have this
- "Small allocations are filled with junk data upon being released." - this is easily added
- "Canaries can be placed at the end of small allocations to absorb small overflows and catch various forms of heap corruption upon free. This was a successfully upstreamed CopperheadOS extension. " - PartitionAlloc doesn't have this
I looked at OpenBSD's allocator, which AFAICT is in openbsd's src/lib/libc/stdlib/malloc.c It contains an implementation of the needed functions: malloc, posix_memalign, calloc, realloc, free, and usable_size It does not contain an implementation of the following, but they should be simple enough to implement: memalign, alligned_alloc, valloc, good_size
So I think it would be possible to get OpenBSD's allocator in without a ton of pain... But the main things OpenBSD's allocator seems to lack is any sort of partitioning. So the real gains that Chrome saw and made was moving Layout Objects and Buffer Objects to their own partitions.
So that brings us to jemalloc. As far as integration goes: Mozilla has merged in jemalloc 4, but it seems to have a lot of bugs filed against it. Some try results seemed to pass on 4.1.1 but failed on 4.2. It seems the best thing to do to figure out it's stability is to sit down with Mike Hommey and ask: if we want to enable jemalloc 3, or 4; how stable is it?
