Opened 5 years ago

Closed 2 years ago

#10281 closed enhancement (fixed)

Investigate usage of alternate memory allocators and memory hardening options

Reported by: mikeperry Owned by: arthuredelstein
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-security, tbb-hardened, TorBrowserTeam201612R
Cc: gk, nickm, mcs, brade, ioerror, tom@…, arthuredelstein, intrigeri Actual Points:
Parent ID: #20955 Points:
Reviewer: Sponsor: SponsorU

Description

One thing we can do to improve security of TBB is to build it with an alternate semi-hardened malloc implementation that attempts to randomize the allocation pattern and performs some minimal checks to guard against heap overflows an reference count issues in Firefox (perhaps by also enabling some additional reference count debugging features already in Firefox).

Such allocator behavior may make exploitation of various use-after-free vulnerabilities more difficult, as it would be harder to predict the location of reallocated regions during exploitation in order to get a target object to overlay an incorrectly freed object.

The downside is this will likely come at the performance costs of loss of locality, increased fragmentation, and additional overhead of reference count checks, but this may be an acceptable cost for improved hardening against exploits.

The first question is: are there any existing drop-in replacement memory allocators we can use in place of Firefox's current jemalloc implementation?

The second question is will any of the Firefox refcounting checks actually help, or will they just increase runtime for no real benefit?

Child Tickets

Change History (51)

comment:1 Changed 5 years ago by mikeperry

Cc: ioerror added

comment:2 Changed 5 years ago by mikeperry

Summary: Investigate usage of alternate memory allocators and meory hardening optionsInvestigate usage of alternate memory allocators and memory hardening options

comment:3 Changed 5 years ago by mikeperry

The memory.free_dirty_pages pref may have some effect here in terms of encouraging the default jemalloc allocator to minimize the use of free lists, but according to https://bugzilla.mozilla.org/show_bug.cgi?id=805855, we may also have to emit the 'memory-pressure' observer event for it to have effect.

comment:5 Changed 5 years ago by tom

Cc: tom@… added

comment:6 Changed 5 years ago by erinn

Keywords: tbb-firefox-patch added

comment:7 Changed 5 years ago by erinn

Component: Firefox Patch IssuesTor Browser

comment:8 Changed 5 years ago by tom

I spoke with cevans and he helped me pull the newest version of ctmalloc from http://src.chromium.org/blink/trunk/Source/wtf/PartitionAlloc.h, which has the usable_size function needed. I commented out a bunch of macros, successfully built, and it this may get past the sqlite3 problem. It crashed in a new spot though. Progress.

comment:9 Changed 5 years ago by arthuredelstein

Cc: arthuredelstein added

comment:10 Changed 5 years ago by mikeperry

Just a note to mention that the memory tags from about:memory may be useful for initial partitioning with PartitionAlloc, but we may need to special case some of that. For instance, ArrayBuffers are a popular target for setting up UAF exploitation, and probably need a special partition, as do certain DOM/iframe objects (see http://robert.ocallahan.org/2010/10/mitigating-dangling-pointer-bugs-using_15.html).

comment:12 Changed 5 years ago by intrigeri

Cc: intrigeri added

comment:13 Changed 4 years ago by tom

As an update.

Mozilla is working on moving to jemalloc3 which will have a partitioned heap. I believe they are hoping to have this for the next ESR, but it's not guaranteed. More: http://guilherme-pg.com/2014/10/15/Partitioned-heap-in-Firefox-pt1.html

I have a ctmalloc build compiling on Linux 4.5 alpha. My patches are:
https://github.com/tomrittervg/tor-browser/commits/tor-browser-31.2.0esr-4.5-1-build5-ctmalloc
https://github.com/tomrittervg/tor-browser-builder/commits/tjr-ctmalloc

Unfortunately, my implementation of memalign is wrong I need to hack at PartitionAlloc to support that, which will be a bit risky and tricky. Also, once jemalloc3 comes along, the advantages of using PartitionAlloc are much fewer, although some of the work (e.g. random partitioning based on callsite) could likely be ported over as defense in depth.

comment:15 Changed 3 years ago by gk

Keywords: TorBrowserTeam201510 added
Sponsor: SponsorU

We should come up with a write-up showing were we are right now.

Last edited 3 years ago by gk (previous) (diff)

comment:17 Changed 3 years ago by gk

Keywords: tbb-hardening added

comment:18 Changed 3 years ago by gk

Keywords: tbb-hardened added; tbb-hardening removed

comment:19 Changed 3 years ago by mikeperry

Keywords: TorBrowserTeam201510 removed

comment:20 Changed 3 years ago by bugzilla

Keywords: tbb-security added; gitian tbb-security-hardening tbb-firefox-patch removed
Severity: Normal

This field is too complex that even M$ is still trying to write MM without bugs. So, it's better to think twice before taking the responsibility for it.

comment:21 Changed 3 years ago by isabela

Sponsor: SponsorUSponsorU-can

comment:22 Changed 3 years ago by gk

Keywords: TorBrowserTeam201606 added

comment:23 Changed 3 years ago by gk

Sponsor: SponsorU-canSponsorU

comment:24 Changed 3 years ago by gk

Keywords: TorBrowserTeam201607 added; TorBrowserTeam201606 removed

comment:25 Changed 3 years ago by arthuredelstein

An update: I tried building tor-browser.git with --enable-jemalloc flag, and it was able to build and run with no problem. This activates jemalloc4. As Tom Ritter points out in his experiments, jemalloc potentially makes it possible to partition the heap into "arenas", but it's going to require a lot of changes that ideally will be carried out by Mozilla.

comment:26 Changed 3 years ago by gk

Keywords: TorBrowserTeam201608 added; TorBrowserTeam201607 removed

Moving items to August 2016.

comment:27 Changed 2 years ago by gk

Keywords: TorBrowserTeam201609 added; TorBrowserTeam201608 removed

Tickets for September.

comment:28 in reply to:  description Changed 2 years ago by waxmiguel

Replying to mikeperry:

One thing we can do to improve security of TBB is to build it with an alternate semi-hardened malloc implementation that attempts to randomize the allocation pattern and performs some minimal checks to guard against heap overflows an reference count issues in Firefox (perhaps by also enabling some additional reference count debugging features already in Firefox).

Such allocator behavior may make exploitation of various use-after-free vulnerabilities more difficult, as it would be harder to predict the location of reallocated regions during exploitation in order to get a target object to overlay an incorrectly freed object.

The downside is this will likely come at the performance costs of loss of locality, increased fragmentation, and additional overhead of reference count checks, but this may be an acceptable cost for improved hardening against exploits.

The first question is: are there any existing drop-in replacement memory allocators we can use in place of Firefox's current jemalloc implementation?

The second question is will any of the Firefox refcounting checks actually help, or will they just increase runtime for no real benefit?

comment:29 Changed 2 years ago by gk

I backported the patch from https://bugzilla.mozilla.org/show_bug.cgi?id=1277704 (and previous related patches) and ran a try build with them based on ESR45:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a567af3611e

There is a lot of test bustage visible. But if one compares that to a vanilla ESR45 build

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d3fa3b49cda3

then it seems to me the test bustage is not related to enabling jemalloc4. Moreover, (intermittent) test failures related to jemalloc 4.2.1 were reported for x86-64 win which we don't support currently. Thus, I think for the alpha series it could make sense to backport 4.2.1 and start playing with hardening it.

comment:30 Changed 2 years ago by gk

Keywords: TorBrowserTeam201610 added; TorBrowserTeam201609 removed

Moving SponsorU items to October.

comment:31 Changed 2 years ago by gk

Owner: set to arthuredelstein
Priority: HighVery High
Status: newassigned

comment:32 Changed 2 years ago by bugzilla

To start playing: https://bugzilla.mozilla.org/show_bug.cgi?id=1262164.
Only for hardened, please: we have enough weird crashes on alphas...

comment:33 Changed 2 years ago by gk

Keywords: TorBrowserTeam201611 added; TorBrowserTeam201610 removed

Moving tickets over to November.

comment:34 Changed 2 years ago by gk

Do we have a patch somewhere up for review, Arthur? I suppose it could take some days for the back-and-forth of the review process and 6.5a6 is scheduled for next week.

comment:35 Changed 2 years ago by gk

Keywords: TorBrowserTeam201612 added; TorBrowserTeam201611 removed

Moving tickets to December.

comment:36 Changed 2 years ago by arthuredelstein

Keywords: TorBrowserTeam201612R added; TorBrowserTeam201612 removed

Here is a patch (please review) that uses jemalloc4 in Tor Browser ESR45 (Linux). It also activates jemalloc redzones, and aborts if the redzones are found to have been corrupted.

https://github.com/arthuredelstein/tor-browser/commit/10281+2

(I'm still working on two alternative patches that (1) uses the DieHarder memory allocator and (2) uses jemalloc4 with randomized arenas, but I think we may want to give the redzone patch a try for now.)

comment:37 Changed 2 years ago by gk

Status: assignedneeds_information

How did you test this? Do you have a test site showing the differences between the current behavior and the intended one?

Last edited 2 years ago by gk (previous) (diff)

comment:38 Changed 2 years ago by gk

Two other things I am wondering: should we have this on the hardened series first, only? (sounds like a good idea to me) Are we okay with the jemalloc4 ESR45 ships or do we think we should need to backport the patches already landed on m-c (or are about to land)? There was work done in comment:29 for that already and I am wondering about the severity of all the bugfixes that landed in jemalloc4 meanwhile...

comment:39 in reply to:  37 Changed 2 years ago by arthuredelstein

Replying to gk:

How did you test this? Do you have a test site showing the differences between the current behavior and the intended one?

I don't have a test page, but I made a test patch that deliberately causes a buffer overflow. The patch is here:

https://github.com/arthuredelstein/tor-browser/commits/10281+2_check

In this code, I malloc a 5-byte array, but write 32 bytes:

+  char* test = (char*) malloc(5);
+  for (int i = 0; i < 32; ++i) {
+    test[i] = i;
+    printf("test[%d] value: %d\n", i, (int) test[i]);
+  }
+  free(test);

(I arbitrarily put this code in a function that always runs on startup, loading of the XUL window.)

arthur@instance-3 ~/tor-browser (10281+2_check)> ./mach run
 0:02.27 /home/arthur/tor-browser/obj-x86_64-unknown-linux-gnu/dist/bin/firefox -no-remote -profile /home/arthur/tor-browser/obj-x86_64-unknown-linux-gnu/tmp/scratch_user
test[0] value: 0
test[1] value: 1
test[2] value: 2
test[3] value: 3
test[4] value: 4
test[5] value: 5
test[6] value: 6
test[7] value: 7
test[8] value: 8
test[9] value: 9
test[10] value: 10
test[11] value: 11
test[12] value: 12
test[13] value: 13
test[14] value: 14
test[15] value: 15
test[16] value: 16
test[17] value: 17
test[18] value: 18
test[19] value: 19
test[20] value: 20
test[21] value: 21
test[22] value: 22
test[23] value: 23
test[24] value: 24
test[25] value: 25
test[26] value: 26
test[27] value: 27
test[28] value: 28
test[29] value: 29
test[30] value: 30
test[31] value: 31
<jemalloc>: Corrupt redzone 0 bytes after 0x7ff663ea6570 (size 8), byte=0x8
<jemalloc>: Corrupt redzone 1 byte after 0x7ff663ea6570 (size 8), byte=0x9
<jemalloc>: Corrupt redzone 2 bytes after 0x7ff663ea6570 (size 8), byte=0xa
<jemalloc>: Corrupt redzone 3 bytes after 0x7ff663ea6570 (size 8), byte=0xb
<jemalloc>: Corrupt redzone 4 bytes after 0x7ff663ea6570 (size 8), byte=0xc
<jemalloc>: Corrupt redzone 5 bytes after 0x7ff663ea6570 (size 8), byte=0xd
<jemalloc>: Corrupt redzone 6 bytes after 0x7ff663ea6570 (size 8), byte=0xe
<jemalloc>: Corrupt redzone 7 bytes after 0x7ff663ea6570 (size 8), byte=0xf
<jemalloc>: Corrupt redzone 8 bytes after 0x7ff663ea6570 (size 8), byte=0x10
<jemalloc>: Corrupt redzone 9 bytes after 0x7ff663ea6570 (size 8), byte=0x11
<jemalloc>: Corrupt redzone 10 bytes after 0x7ff663ea6570 (size 8), byte=0x12
<jemalloc>: Corrupt redzone 11 bytes after 0x7ff663ea6570 (size 8), byte=0x13
<jemalloc>: Corrupt redzone 12 bytes after 0x7ff663ea6570 (size 8), byte=0x14
<jemalloc>: Corrupt redzone 13 bytes after 0x7ff663ea6570 (size 8), byte=0x15
<jemalloc>: Corrupt redzone 14 bytes after 0x7ff663ea6570 (size 8), byte=0x16
<jemalloc>: Corrupt redzone 15 bytes after 0x7ff663ea6570 (size 8), byte=0x17

So the free(...) call resulted in jemalloc detecting that a redzone had been overwritten. Note that 16 bytes of overwriting were detected, because allocation is chunked to 16 bytes here.

comment:40 in reply to:  38 ; Changed 2 years ago by arthuredelstein

Replying to gk:

Two other things I am wondering: should we have this on the hardened series first, only? (sounds like a good idea to me)

That's OK with me! I'm not sure how it will interact with ASan, though.

Are we okay with the jemalloc4 ESR45 ships or do we think we should need to backport the patches already landed on m-c (or are about to land)? There was work done in comment:29 for that already and I am wondering about the severity of all the bugfixes that landed in jemalloc4 meanwhile...

Unfortunately, when I include just the 4 patches from comment:29 and use MOZ_JEMALLOC4=1, I get build errors. I think jemalloc didn't actually build in comment:29, because the patches that implemented --enable-jemalloc=4 weren't included. I'm working on backporting more patches to try to get the latest jemalloc to build. It's a bit complex.

comment:41 in reply to:  40 Changed 2 years ago by arthuredelstein

Replying to arthuredelstein:

Unfortunately, when I include just the 4 patches from comment:29 and use MOZ_JEMALLOC4=1, I get build errors.

Apparently I was doing something wrong here. When I wiped everything again, the build worked fine with the 4 patches from comment:29. Sorry for my confusion!

Here's a new version, using the 4 patches from comment:29 to upgrade to jemalloc 4.3.1. Then my additional patch enables jemalloc4 as the memory allocator and activates aborts on redzones.

https://github.com/arthuredelstein/tor-browser/commits/10281+3

And here is the same extra test patch, again. I get the same results as in comment:39:

https://github.com/arthuredelstein/tor-browser/commit/10281+3_check

Something Yawning pointed out is that redzones will no longer be available in jemalloc 5:
https://github.com/jemalloc/jemalloc/issues/369
But given that Firefox 52 still uses jemalloc 4.x, we should be OK for all of 2017. I see redzones as a stopgap while we continue to look for better options.

comment:42 Changed 2 years ago by arthuredelstein

Status: needs_informationneeds_review

comment:43 in reply to:  40 Changed 2 years ago by gk

Replying to arthuredelstein:

Replying to gk:

Two other things I am wondering: should we have this on the hardened series first, only? (sounds like a good idea to me)

That's OK with me! I'm not sure how it will interact with ASan, though.

Hm. Looking closer at what we have in .mozconfig-asan it seems we might want to try the alpha series:

ac_add_options --enable-address-sanitizer
ac_add_options --disable-jemalloc

Using the hardened series instead would at least merit some investigation first I think but we are probably lacking that time right now.

mcs/brade: could you give the patches a look as well?

Last edited 2 years ago by gk (previous) (diff)

comment:44 Changed 2 years ago by mcs

I did not look at the backported patches; do I need to? They are large; I guess if they merged cleanly they are probably okay.

Enabling the redzones and abort options might make TB slower and less stable. But I guess finding lurking bugs is a good thing as long as performance is not reduced too much.

comment:45 in reply to:  44 Changed 2 years ago by arthuredelstein

Replying to mcs:

I did not look at the backported patches; do I need to? They are large; I guess if they merged cleanly they are probably okay.

I think those are OK -- they're just pulling in the update patches from jemalloc.

Enabling the redzones and abort options might make TB slower and less stable. But I guess finding lurking bugs is a good thing as long as performance is not reduced too much.

Both these comments are true. Seems like the alphas is the right place until we observe performance and how crashy the browser is. One thing I can try in the near future is to push this patch to Mozilla's try server and look at crashes and talos (performance) results.

comment:46 Changed 2 years ago by gk

Okay I applied the commits to tor-browser-45.5.1esr-6.5-1:

d8a70881bc02d75b001aa41f728e54e7386eac17
3262a05cc8b55af727e779f5460583c8e5d740c9
dcd174d057a14a9c5b3086e92b3005a3cf7d69fe
1f3117d9feafd34582214479ce85752514c7b34e
bd00b14abd3c9dace3ccbce2a4597602bb84a3c7

comment:47 Changed 2 years ago by arthuredelstein

Parent ID: #20955

comment:48 Changed 2 years ago by gk

Resolution: fixed
Status: needs_reviewclosed

This ticket is done. Further items to work on will be detailed in child tickets to #20955.

comment:49 Changed 2 years ago by tom

I'm pasting this here from an email I wrote in Sept 2016.

The original problem with PartitionAlloc was the lack of memalign
functions and the need to implement them. They don't seem to have been
added, so that problem persists.

See https://github.com/struct/HardenedPartitionAlloc/issues/1

There seem to be some security
features that are in OpenBSD's allocator that aren't/can't be in
PartitionAlloc, but not all of them.

  • No inline metadata - PartitionAlloc has this feature as well.
  • "It is guaranteed to abort for pointers that are not active malloc allocations." - not sure about this, but http://struct.github.io/partition_alloc.html has a patch for PartitionAlloc that does check the freelist for double frees. I'm not sure if this is a comprehensive equality of features though
  • "sets the allocator to abort on out-of-memory by default" - This is probably pretty easy to do. (Just a NULL check and an abort() no?)
  • "Fine-grained randomization is performed for small allocations by choosing a random pool to satisfy requests" - okay, but this is like 'choose a random partition' except not as powerful, cause you're in the same heap
  • "and then choosing a random free slot within a page provided by that pool" - PartitionAlloc does not have this, but it could. Here's a patch: http://struct.github.io/partition_alloc.html
  • "Freed small allocations are quarantined before being put back into circulation via a randomized delayed allocation pool" - okay, PArtitionAlloc doesn't have this
  • "CopperheadOS uses a ring buffer..." PartitionAlloc doesn't have this
  • "Small allocations are filled with junk data upon being released." - this is easily added
  • "Canaries can be placed at the end of small allocations to absorb small overflows and catch various forms of heap corruption upon free. This was a successfully upstreamed CopperheadOS extension. " - PartitionAlloc doesn't have this

I looked at OpenBSD's allocator, which AFAICT is in openbsd's
src/lib/libc/stdlib/malloc.c
It contains an implementation of the needed functions: malloc,
posix_memalign, calloc, realloc, free, and usable_size
It does not contain an implementation of the following, but they should
be simple enough to implement: memalign, alligned_alloc, valloc, good_size

So I think it would be possible to get OpenBSD's allocator in without a
ton of pain... But the main things OpenBSD's allocator seems to lack is
any sort of partitioning. So the real gains that Chrome saw and made
was moving Layout Objects and Buffer Objects to their own partitions.

So that brings us to jemalloc. As far as integration goes: Mozilla has
merged in jemalloc 4, but it seems to have a lot of bugs filed against
it. Some try results seemed to pass on 4.1.1 but failed on 4.2. It
seems the best thing to do to figure out it's stability is to sit down
with Mike Hommey and ask: if we want to enable jemalloc 3, or 4; how
stable is it?

comment:50 Changed 2 years ago by adrelanos

Resolution: fixed
Status: closedreopened

Since Tor Browser 7.0a2 it does no longer start. (Qubes-Whonix)

user@host:~/.tb/tor-browser$ ./start-tor-browser.desktop --debug
Launching './Browser/start-tor-browser --detach --debug'...
Using system Tor process.
<jemalloc>: Corrupt redzone 0 bytes after 0x7f0503ede9d0 (size 80), byte=0x0
<jemalloc>: Corrupt redzone 1 byte after 0x7f0503ede9d0 (size 80), byte=0x0
<jemalloc>: Corrupt redzone 2 bytes after 0x7f0503ede9d0 (size 80), byte=0x0
<jemalloc>: Corrupt redzone 3 bytes after 0x7f0503ede9d0 (size 80), byte=0x0
<jemalloc>: Corrupt redzone 4 bytes after 0x7f0503ede9d0 (size 80), byte=0x0
<jemalloc>: Corrupt redzone 5 bytes after 0x7f0503ede9d0 (size 80), byte=0x0
<jemalloc>: Corrupt redzone 6 bytes after 0x7f0503ede9d0 (size 80), byte=0x0
<jemalloc>: Corrupt redzone 7 bytes after 0x7f0503ede9d0 (size 80), byte=0x0
./Browser/start-tor-browser: line 368:  2593 Segmentation fault      TOR_CONTROL_PASSWD=${TOR_CONTROL_PASSWD} ./firefox --class "Tor Browser" -profile TorBrowser/Data/Browser/profile.default "${@}" < /dev/null
user@host:~/.tb/tor-browser$ echo $?
139

Something in the system config or environment seems to trigger this. Any idea?

comment:51 Changed 2 years ago by adrelanos

Resolution: fixed
Status: reopenedclosed

Created #21804 for it.

Note: See TracTickets for help on using tickets.