Opened 3 years ago

Last modified 3 years ago

#20100 needs_information defect

persistent libxul.so bug crashing TBB Linux/64 (but probably a bug in locally linked shared object)

Reported by: sjamaan Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Major Keywords: tbb-crash
Cc: gk Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I have a TBB (Linux x86/64) that crashes every now and then (not reproducible as it seems quite random yet it's happening over multiple TBB versions): tor-browser-linux64-6.0.4_en-US.tar.xz and about 10 earlier versions. Both Ubuntu and Debian latest stables.

Linux kernel log reveals the issue in libxul.so, always at the same mmap pointer [segment+4a4a000] Same segment in previous TBBs. The mmap range for libxul.so always is 04a4a000 - 04f25000.

Kernel message:

firefox[1202] trap stack segment ip:7f6032255894 sp:7ffc6cc8aae0 error:0 in libxul.so[7f60305a9000+4a4a000]

Now, libxul.so links to 69 locally installed shared libs. In future for tor security, I believe this is interesting to look at because they don't come with TBB. ldd libxul.so and see for yourself :)

I don't see a reason in sharing my libs outside TBB because at the moment I can't narrow it down more specifically (problem of reproduction). Any tips on how to do this are welcome (strace, ptrace have not wielded any clues and with gdb I don't know where to start as it appears random). So different debugging tips are welcome ;)

My question is simple: I want to figure out of there is a problem with my Linux install... So did more people with the tor-browser-linux64-6.0.4_en-US (and earlier) encountered crashes through libxul.so (specifically with the 4a4000 offset which is where the current TBB x64/64 libxul loads) ? With 69 linked shared objects on the local system, I think it's better to raise the general question on libxul crashes before diving into the depths.

Child Tickets

Change History (11)

comment:1 Changed 3 years ago by gk

Component: - Select a componentApplications/Tor Browser
Keywords: tbb-crash added; libxul.so stack segment trap removed
Owner: set to tbb-team

comment:2 Changed 3 years ago by gk

That's the first crash of this type I've seen. So, no, we don't know of any other user having this problem. What I've been doing for a while now to catch hard to reproduce crashes is running Tor Browser inside GDB the whole time (see: https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Hacking#Usinggdb). I basically start Firefox from inside GDB.

comment:3 Changed 3 years ago by ronvandaal

Bump. This problem exists outside TBB. Seems to be a (remote) vulnerability in libxul not TBB. Surely managed to hit Iceweasel binaries on disk.. Segfaults, Iceweasel (default Debian Jessie/amd64 builds) basically stopped working, corrupted new binary in place :( Hopefully more details for debugging on next occurrence.

Modified object name: /usr/lib/iceweasel/libxul.so

Property: Expected Observed
------------- ----------- -----------

  • CRC32 ANY2Ld BF2LwO
  • MD5 Bbag4ohsGADzwEm2fKZ88P BiEha5mIlm4mmKifQ0AuZC

-ronvandaal

Last edited 3 years ago by ronvandaal (previous) (diff)

comment:4 Changed 3 years ago by ronvandaal_

Priority: MediumHigh
Severity: NormalMajor

Another libxul.so oddity. Fresh system, stock Debian using TBB now. This one affects tor-browser-linux64-6.0.6_en-US.tar.xz

Fresh unpack and run of TBB. Crashes after surfing to a popular website.

Here only one byte changed:

before crash: 1d66170 -- bytes: e800 f39a fec4 8948 48ea c389 8948 4cc7
after crash: 1d66170 -- bytes: e900 f39a fec4 8948 48ea c389 8948 4cc7

0xe8 turned 0xe9 (elf64-x86-64 format)

1d66171: e8 9a f3 c4 fe callq 9b5510 <moz_xmalloc@plt>
1d66171: e9 9a f3 c4 fe jmpq 9b5510 <moz_xmalloc@plt>

I have to ask: is it normal for libxul.so binaries to change when run? (not a coder here, sorry)

If not, maybe TBB could implement a integrity checker on the libs?

I believe there's an issue with libxul in general, which also may affect the Tor Browser Bundle, Tails, etc.

The difference here is 1 byte. It didn't expand like in other settings. With JMPQ there's no need for the CALL routine to return using RET, JMPQ discards proper control using addresses pushed on the stack. This behaviour could explain the segfaults in the other configurations.

Right now TBB startup scripts don't check hashes of packaged libs. And why not? It's an easy feat to add to the start-tor-browser-desktop script. Call it an early warning system if you will. Now TBB runs REGARDLESS if a modded libxul.so is residing in the Browser/ directory. This is also a way to find users with similar problems (in case of no segfault, when TBB appears to "just" exit)

Just my 2 cts

ronvandaal

comment:5 in reply to:  2 Changed 3 years ago by ronvandaal_

Replying to gk:

That's the first crash of this type I've seen. So, no, we don't know of any other user having this problem.

That's probably because most of this goes under the radar since there's no validation on the lib during startup. Some modded ie. patched (see previous post) libxul.so just runs fine without the user noticing. Besides that, when TBB suddenly exits it doesn't always generate a segfault. Not all users investigate and report bugs, etc. I would like so suggest checksum validation on at least libxul.so as a enhancement for your project. (submitted Enhancement ticket #20743)

Last edited 3 years ago by ronvandaal_ (previous) (diff)

comment:6 Changed 3 years ago by cypherpunks

is it normal for libxul.so binaries to change when run?

On disk? No, of course.
What hash of your libxul?

comment:7 in reply to:  6 Changed 3 years ago by ronvandaal_

Replying to cypherpunks:

is it normal for libxul.so binaries to change when run?

On disk? No, of course.
What hash of your libxul?

The question was for my own sanity. Thank you ;-)

See attached files:
tor-browser-linux64-6.0.6_en-libxul.so-STOCK-VALID eaa0cb53cea58ced3a8182d81c9b48c9
tor-browser-linux64-6.0.6_en-libxul.so-POWNED cc2fc415dda6632cf67b285ae41808b4

For records: tor-browser-linux64-6.0.6_en-US.tar.xz e71f92bc24cc8325a328561c2f6d6aa7

These don't relate to TBB but Debian's Iceweasel (patchlevel unsure, did reinstall after):
libxul.so-debian-jessie-amd64-STOCK cad4639bff84298c3af0340c19714e46
libxul.so-debian-jessie-amd64-POWN 3eb2b8139fd24211e5a787f2510a3aff (changed while browsing)

I'm certain when the TBB init scripts (same goes for Tails) run a simple hashchecker, perhaps with the option to feedback a mismatch, more background info on the bug can be obtained. Just an idea.

comment:8 Changed 3 years ago by ronvandaal_

OK, so no attachments due to huge filesizes and limits on the trac.
The 1 byte difference and it's assembly + address is already described above.

Please know the problem existed in tor-browser-linux64-6.0.4_en-US.tar.xz and earlier ones as described in FIPO. It affects Iceweasel's core lib and the only way to get info, I believe, is to implement a hash checker before runtime and exit when a compromised lib is detected. This way you'd get more feedback from users that have similar issues.

I don't know why it always comes back with new setups and builds and the latest 1-byte change (same filesize!) appears to me quite as odd... 1 byte is too little to investigate.

Last edited 3 years ago by ronvandaal_ (previous) (diff)

comment:9 Changed 3 years ago by ronvandaal_

By the way. Also see. #20746 Question: what are MLF files (found in /tmp/) ?

comment:10 Changed 3 years ago by gk

Cc: gk added
Status: newneeds_information

Following up on comment:2 would still be helpful.

comment:11 in reply to:  10 Changed 3 years ago by ronvandaal_

Replying to gk:

Following up on comment:2 would still be helpful.

Will do that but problem occurs at random. Goes well for months, then suddenly a 2 crashes, a week no issues, another crash, etc. Will setup a new environment and let it run in gdb if it doesn't slow down the user experience.

Besides that I wonder what your thoughts are on adding a hash check at the Bundles startup script? Libxul.so is quite vast with alot of shared library dependencies. I can't belief there isn't any kind of integrity checking done on this in Tor Browser Bundle. Bugs like these go under the radar, how many users would unpack the TBB bundle and notice a 1 byte change from a CALLQ to a JMPQ which basically has the same functionality and doesn't crash until further exploitation. It's still a wild guess why this happens, but integrity checking the binaries seems common sense.

Note: See TracTickets for help on using tickets.