Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#13359 closed defect (fixed)

Linux: update fails because bundled libstdc++.so.6 is not found

Reported by: mcs Owned by: gk
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Keywords: ff31-esr
Cc: brade, mikeperry, boklm Actual Points:
Parent ID: #12460 Points:
Reviewer: Sponsor:

Description

When an ESR31-based browser tries to restart after applying an update on Linux, it fails to restart because the bundled copy of libstdc++.so.6 is not found. Here is the full error message:

XPCOMGlueLoad error for file /home/brade/Desktop/tbb-4.0a3-ru/Browser/libxul.so:
/usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required
  by /home/brade/Desktop/tbb-4.0a3-ru/Browser/libxul.so)
Couldn't load XPCOM.
Tor Browser exited abnormally.  Exit code: 255

Although LD_LIBRARY_PATH is set inside start-tor-browser to avoid this problem, apparently something within the updater code clears it before restarting Firefox. Relying on LD_LIBRARY_PATH also means that the following will fail too:

cd Browser
./firefox

Can we use -rpath at link time instead or will that cause other problems?

Child Tickets

Change History (16)

comment:1 in reply to:  description ; Changed 3 years ago by gk

Replying to mcs:

Can we use -rpath at link time instead or will that cause other problems?

My first guess is that it will break one part of our hardening. We had the RPATH for a while available, see #9150. Then some jerk showed up on the forum and explained how this may easily be exploited: https://blog.torproject.org/blog/tor-browser-365-and-40-alpha-2-are-released#comment-74540. If your solution leads to the same issue I'd rather avoid that.

I wonder how tor is starting properly up at all after the update as it definitely needs the LD_LIBRARY_PATH. Or are we just lucky to have the libs on our system and thus, we don't see this problem during testing? Assuming the LD_LIBRARY_PATH is somehow working for tor but not Tor Browser, what is happening if you put the libstdc++ into the directory where all the libs are that tor needs and do the update then (assuming we put the library there and not into /Browser)? Could you test that scenario?

Last edited 3 years ago by gk (previous) (diff)

comment:2 in reply to:  1 ; Changed 3 years ago by mcs

Replying to gk:

Replying to mcs:

Can we use -rpath at link time instead or will that cause other problems?

My first guess is that it will break one part of our hardening. We had the RPATH for a while available, see #9150. Then some jerk showed up on the forum and explained how this may easily be exploited: https://blog.torproject.org/blog/tor-browser-365-and-40-alpha-2-are-released#comment-74540. If your solution leads to the same issue I'd rather avoid that.

Make sense, although I am not sure what the security impact is if we were to use $ORIGIN with -rpath. On the other hand, I found this: http://seclists.org/fulldisclosure/2010/Oct/257 (setuid program vulnerability caused by use of $ORIGIN).

I wonder how tor is starting properly up at all after the update as it definitely needs the LD_LIBRARY_PATH. Or are we just lucky to have the libs on our system and thus, we don't see this problem during testing? Assuming the LD_LIBRARY_PATH is somehow working for tor but not Tor Browser, what is happening if you put the libstdc++ into the directory where all the libs are that tor needs and do the update then (assuming we put the library there and not into /Browser)? Could you test that scenario?

I did not test that scenario, because I confirmed that without LD_LIBRARY_PATH set on my system, all of the tor dependencies are satisfied by system libraries (probably not a good thing):

cd tor-browser/Browser
LD_DEBUG=libs ./TorBrowser/Tor/tor
     31204:	find library=libz.so.1 [0]; searching
     31204:	 search cache=/etc/ld.so.cache
     31204:	  trying file=/lib/x86_64-linux-gnu/libz.so.1
     31204:	
     31204:	find library=libm.so.6 [0]; searching
     31204:	 search cache=/etc/ld.so.cache
     31204:	  trying file=/lib/x86_64-linux-gnu/libm.so.6
     31204:	
     31204:	find library=libevent-2.0.so.5 [0]; searching
     31204:	 search cache=/etc/ld.so.cache
     31204:	  trying file=/usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
     31204:	
     31204:	find library=libssl.so.1.0.0 [0]; searching
     31204:	 search cache=/etc/ld.so.cache
     31204:	  trying file=/lib/x86_64-linux-gnu/libssl.so.1.0.0
...

Ideally, running tor or firefox without setting LD_LIBRARY_PATH would do the right thing. On the other hand, we don't want to introduce an exploit path.

comment:3 Changed 3 years ago by mikeperry

The problem with our -rpath setting was that it was to a fixed path that wasn't actually in use (/home/ubuntu/install/lib, IIRC). The argument by the commenter was that an "ubuntu" username on the machine might be able to create this path with a library in it, to compromise other users. I don't think the same argument applies to a relative rpath.

If we can set a "$ORIGIN" -rpath relative to the current directory, we may have people complaining about things like #12736, but ultimately I don't think it is any worse than always LD_LIBRARY_PATH to the same directory.

If I had to guess, there's probably code somewhere in the installation of updates in Firefox that is calling execle or execvpe with a blank environment. We could add LD_LIBRARY_PATH back in to this environment.

I think a relative rpath is probably a cleaner solution though?

comment:4 in reply to:  2 Changed 3 years ago by gk

Replying to mcs:

I did not test that scenario, because I confirmed that without LD_LIBRARY_PATH set on my system, all of the tor dependencies are satisfied by system libraries (probably not a good thing):

I gave it a test. Here is what I did:

  1. I got rid of my system libevent.
  2. I downloaded a 4.0-alpha-2 and commented out the LD_LIBRARY_PATH settings in the start script.
  3. On starting 4.0-alpha-2 tor complained about the missing libevent and did not start.
  4. I removed the comments in the start script, started and updated to 4.0-alpha-3.
  5. A restart worked as it should. Thus, for tor LD_LIBRARY_PATH is somehow still present.

If that is true, I wonder whether a short-term workaround would be to put the libstdc++ into the directory with the libs tor is dependent on.

comment:5 in reply to:  3 ; Changed 3 years ago by gk

Replying to mikeperry:

The problem with our -rpath setting was that it was to a fixed path that wasn't actually in use (/home/ubuntu/install/lib, IIRC). The argument by the commenter was that an "ubuntu" username on the machine might be able to create this path with a library in it, to compromise other users. I don't think the same argument applies to a relative rpath.

If we can set a "$ORIGIN" -rpath relative to the current directory, we may have people complaining about things like #12736, but ultimately I don't think it is any worse than always LD_LIBRARY_PATH to the same directory.

If I had to guess, there's probably code somewhere in the installation of updates in Firefox that is calling execle or execvpe with a blank environment. We could add LD_LIBRARY_PATH back in to this environment.

I think a relative rpath is probably a cleaner solution though?

I tend to agree. My main concern currently is that we are running short of time implementing and testing this properly for the upcoming release: 1) Getting it to work at all seems not so easy as passing "$ORIGIN" to the proper flags AND having it in the RPATH after linking turns out to be non-trivial. Seems I need to find the proper spell here... (we could do the workaround mentioned in https://enchildfone.wordpress.com/2010/03/23/a-description-of-rpath-origin-ld_library_path-and-portable-linux-binaries/ but I'd rather avoid that). 2) But even then we have our testsuite complaining about the now acceptable RPATH. 3) There are not all Firefox libs in /Browser but some are in /Browser/components. Thus, we might need some patches to the build logic to make sure everything has the proper RPATH. 4) There is pluggable transports build logic (fte comes to mind) that needs to get adapted and the result tested. So, I'd favor a stopgap not involving this change one or two days before we start our release building.

I'd test the one mentioned in comment:4 but I currently lack the update testing setup. mcs: can you think of trying this idea (does not matter if you still have your tor related system libs installed in this case)? Would a patch from my side be of help (it would basically copy the libstdc++ into the tor lib directory instead of the browser directory)?

Other cheap workaround proposals are welcome.

comment:6 Changed 3 years ago by gk

Replace the concern mentioned in 3) with the issue that we need to make sure that the proper flags get passed down to all the parts of the Mozilla build system.

comment:7 Changed 3 years ago by boklm

Cc: boklm added

comment:8 Changed 3 years ago by gk

Cc: boklm removed

I fixed 1) and am now working on 3).

comment:9 Changed 3 years ago by gk

Cc: bolkm added

comment:10 in reply to:  5 ; Changed 3 years ago by mcs

I'd test the one mentioned in comment:4 but I currently lack the update testing setup. mcs: can you think of trying this idea (does not matter if you still have your tor related system libs installed in this case)? Would a patch from my side be of help (it would basically copy the libstdc++ into the tor lib directory instead of the browser directory)?

Kathy and I tried this for a 4.0-alpha-3 to nightly upgrade. Our quick and dirty approach was to move libstdc++.so from Browser/ to Browser/TorBrowser/Tor/ after the update was applied (but before clicking "Restart to Update"). The browser (and tor) restarted without error. In fact, if you set devtools.chrome.enabled = true, open the browser console, and then type the following you can see LD_LIBRARY_PATH:

alert(Cc@mozilla.org/process/environment;1.getService(Ci.nsIEnvironment).get("LD_LIBRARY_PATH"));

After successfully restarting, the above shows that LD_LIBRARY_PATH is still set to /home/brade/Desktop/tor-browser/Browser/TorBrowser/Tor/ (the same value it had in 4.0-alpha-3 before we applied the update). So the update process is not touching LD_LIBRARY_PATH; the real problem is that the LD_LIBRARY_PATH value used in 4.0-alpha-3 is not sufficient for the ESR31-based browser.

It would still be nice to embed a relative rpath but it seems that moving libstdc++.so is an acceptable workaround if that proves to be too messy.

comment:11 Changed 3 years ago by gk

An update: First, 4) in comment:5 is no urgent issue either as an update with FTE enabled along the lines of the steps I outlined in comment:4 is working as well. Apart from that is no pluggable transport currently compiled with GCC 4.9.1, so there is no new libstdc++ requirement as in Firefox related code. Second, I compiled the browser with

export LDFLAGS='-Wl,-rpath,\$${ORIGIN}

This gives me for tor a proper RPATH (i.e. ${ORIGIN}) and is working. For the browser, however, I get that as RUNPATH. This is likely due to the gold linker. See: https://sourceware.org/bugzilla/show_bug.cgi?id=15098 and the likely culprit: https://sourceware.org/git/?p=binutils.git;a=commit;h=0e0646ad6f21c4d6230c31f452b73f133f4b5777. We might want to make sure this issue does not affect us. Examining the libs I realized that we want a more fine-grained approach, too. E.g. there are libraries that have the RUNPATH set although there is no need to as there are no libstdc++ dependencies and there are the libraries in /components that have the wrong RUNPATH: ${ORIGIN} is not correct in this case as the libstdc++ is not in /components. Hence, the system libstdc++ is used.

comment:12 in reply to:  10 Changed 3 years ago by gk

Replying to mcs:

I'd test the one mentioned in comment:4 but I currently lack the update testing setup. mcs: can you think of trying this idea (does not matter if you still have your tor related system libs installed in this case)? Would a patch from my side be of help (it would basically copy the libstdc++ into the tor lib directory instead of the browser directory)?

Kathy and I tried this for a 4.0-alpha-3 to nightly upgrade. Our quick and dirty approach was to move libstdc++.so from Browser/ to Browser/TorBrowser/Tor/ after the update was applied (but before clicking "Restart to Update"). The browser (and tor) restarted without error. In fact, if you set devtools.chrome.enabled = true, open the browser console, and then type the following you can see LD_LIBRARY_PATH:

alert(Cc@mozilla.org/process/environment;1.getService(Ci.nsIEnvironment).get("LD_LIBRARY_PATH"));

After successfully restarting, the above shows that LD_LIBRARY_PATH is still set to /home/brade/Desktop/tor-browser/Browser/TorBrowser/Tor/ (the same value it had in 4.0-alpha-3 before we applied the update). So the update process is not touching LD_LIBRARY_PATH; the real problem is that the LD_LIBRARY_PATH value used in 4.0-alpha-3 is not sufficient for the ESR31-based browser.

Ah, yes, that makes sense. Thanks for trying.

It would still be nice to embed a relative rpath but it seems that moving libstdc++.so is an acceptable workaround if that proves to be too messy.

Given the issues I mentioned in my previous comment and our lack of time I am in favor of moving libstdc++. Seems less risky than hotfixing this with -rpath although that is preferable in the longer run.

comment:13 Changed 3 years ago by gk

bug_13359 in my public tor-browser-bundle repo has the "move-libstdc++-and-go-back-to-old-LD_LIBRARY_PATH-value"-fix. I am currently building Linux bundles with it to test my changes.

comment:14 Changed 3 years ago by gk

Resolution: fixed
Status: newclosed

This got fixed by commit fb0ffbecae5c102230ee6b789dcc02c11b862689. #13373 is the ticket for using a RPATH/RUNPATH instead of LD_LIBRARY_PATH now.

comment:15 Changed 3 years ago by gk

Cc: boklm added; bolkm removed

comment:16 Changed 3 years ago by mikeperry

Parent ID: #12460
Note: See TracTickets for help on using tickets.