Opened 2 years ago

Closed 2 years ago

#22041 closed defect (fixed)

libmozsandbox.so error on Debian jessie amd64 after upgrading to 7.0a3

Reported by: dcf Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Critical Keywords: ff52-esr, tbb-7.0-must-alpha, TorBrowserTeam201705R
Cc: mcs, brade, asn, arthuredelstein Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

This may be related to #21907.

Tor Browser downloaded some update yesterday. Today I started it and it showed a progress bar and a message like "Please wait a few moments while we apply the updates" and then the browser didn't start. Trying two start it manually in the command line causes:

$ ./start-tor-browser.desktop -v
Launching './Browser/start-tor-browser --detach -v'...
XPCOMGlueLoad error for file /home/david/tor-browser_en-US/Browser/libxul.so:
libmozsandbox.so: cannot open shared object file: No such file or directory
Couldn't load XPCOM.

I would have expected that the update was for 7.0a3, released yesterday. Bug I am not sure what exact version it thinks it is, because I see 7.0a1, 7.0a2, and 7.0a3 in various places when I grep:

$ grep -rF '7.0a'
...
Browser/TorBrowser/Docs/sources/versions.moz-backup:TORBROWSER_VERSION=7.0a1
Browser/TorBrowser/Docs/sources/versions:TORBROWSER_VERSION=7.0a2
...
Browser/TorBrowser/Data/Browser/profile.default/prefs.js:user_pref("browser.startup.homepage_override.torbrowser.version", "7.0a2");
Browser/TorBrowser/Data/Browser/profile.default/prefs.js:user_pref("extensions.lastTorBrowserVersion", "7.0a2");
Browser/TorBrowser/Data/Browser/profile.default/prefs.js:user_pref("extensions.torbutton.lastBrowserVersion", "7.0a2");
Browser/TorBrowser/UpdateInfo/updates/0/update.version:7.0a3
...

Child Tickets

Change History (14)

comment:1 Changed 2 years ago by gk

Cc: mcs brade added
Keywords: tbb-7.0-must-alpha TorBrowserTeam201704 added
Priority: MediumHigh
Status: newneeds_information

Huh. Do you have some update related logs somewhere in your Browser dir? Like last-update.log or update.log that could shine some light on this issue? And I guess there is no libmozsandbox.so in your Browser dir either?

comment:2 in reply to:  1 Changed 2 years ago by dcf

Replying to gk:

Huh. Do you have some update related logs somewhere in your Browser dir? Like last-update.log or update.log that could shine some light on this issue? And I guess there is no libmozsandbox.so in your Browser dir either?

I sent a copy of the directory to gk and mcs and brade. There's no libmozsandbox.so. There are a bunch of .moz-backup files and a Browser/TorBrowser/UpdateInfo/updates/0/ directory.

comment:3 Changed 2 years ago by gk

Thanks. So it seems you tried to apply an incremental update indicating you were on 7.0a2 before (there is a UPDATE TYPE partial in your update.log). But that fails with

EXECUTE PATCH TorBrowser/Tor/PluggableTransports/snowflake-client
unable to open destination file: TorBrowser/Tor/PluggableTransports/snowflake-client, err: 2
### execution failed

and indeed there is no snowflake-client available for some reason. I wonder why it did not fall back to a full update then. Hmm. And there are a bunch of errors that seem to indicate restoring the backup did not work which is probably the reason your bundle is in the broken state.

mcs/brade could you have a closer look at that bug?

comment:4 Changed 2 years ago by gk

Priority: HighVery High
Severity: NormalCritical
Status: needs_informationassigned

asn reported the same error message today. The update.log file shows a non-applying partial update as well. Trying to reproduce the bug with a non-applying partial update on my system fails, though: the updater falls back to the full update and applies it properly.

comment:5 Changed 2 years ago by gk

Cc: asn added

comment:6 Changed 2 years ago by mcs

One thing about dcf's setup that is different from most people's is that he has app.update.staging.enabled = false. Maybe this bug is easily reproducible if staging is disabled. Kathy and I try that and see.

I think the real problem here is that after the partial update failed to apply, "unwinding" the partially applied update failed badly (hence there are a lot of .moz-backup files left behind). Then, since firefox could not start due to the botched partial update that was not backed out, there was no chance for the update service to notice the failure and fallback to a full update.

I don't understand why the updater was unable to rename the.moz-backup files in order to restore the original (7.0a2) files. The update.log file is full of messages like:

backup_restore: cannot get info for backup file: plugin-container.moz-backup

That message comes from https://gitweb.torproject.org/tor-browser.git/tree/toolkit/mozapps/update/updater/updater.cpp?h=tor-browser-45.8.0esr-7.0-1#n1028

Unfortunately, errno is not logged after the lstat call fails.

comment:7 Changed 2 years ago by dcf

I have sent a backup of the directory from before the attempted upgrade to gk, mcs, and brade. It indeed has no snowflake-client, I'm not sure why. It is possible that I was making manual changes inside the installation directory.

comment:8 Changed 2 years ago by mcs

dcf – thanks for the files.
Kathy and I have not been able to prove it yet, but our current theory is that there is an error in a patch we applied to the backup_restore() function within toolkit/mozapps/update/updater/updater.cpp. We added an lstat call but passed it the path of the original file (e.g., /home/mcs/tor-browser_en-US/Browser/firefox) instead of the path to the backup file that is supposed to be restored (/home/mcs/tor-browser_en-US/Browser/firefox.moz-backup). This will cause backup_restore() to return early and not restore the backed up file. Some files are successfully patched during the failed incremental update, so this bug causes the original (7.0a2) files to not be put back in place. We will try to confirm this tomorrow.

We also need to think about the implications, e.g., how many people are affected? If we are correct in our analysis, this is not a new bug. But it is possible that older browsers managed to start up and run even with mixed up files. However, I find that somewhat unlikely, so maybe most people have left staged updates enabled (no .moz-backup files are needed in that case). Windows users are not affected because the code we added is there to support symlinks (OSX and Linux).

comment:9 Changed 2 years ago by mcs

Keywords: TorBrowserTeam201704R added; TorBrowserTeam201704 removed
Status: assignedneeds_review

Here is a patch for review:

https://gitweb.torproject.org/user/brade/tor-browser.git/commit/?h=bug22041-01&id=b1dd51db5d8489b67a47344dbcffe21aaa083378

The main problem was a botched test of the lstat return value.
Embarrassingly, it appears that this bug has been present since we first added symlink support to the updater (a long time ago). Kathy and I believe the impact is relatively low because most Linux and OSX users have not turned off staged updates plus made changes to the browser installation. Another way to look at it is that no one reported a bug until now.

Users who experienced this bug may have several .moz-backup files under their Browser/ directory, but after a full update is successfully applied the backup files will be removed. If they have .moz-backup files and successfully apply an incremental update they may have some backup files left, but they will tend to be deleted over time (as files are patched for each new release).

Since we cannot patch the updater programs that have already been released, we have to live with this bug for one more update.

comment:10 Changed 2 years ago by gk

Cc: arthuredelstein added

Thanks, looks good to me. Arthur, what do you think?

comment:11 Changed 2 years ago by gk

Tested as well and it fixes the problem. Still waiting for Arthur before merging.

comment:12 Changed 2 years ago by arthuredelstein

Looks good to me too. Sorry I missed this earlier.

comment:13 Changed 2 years ago by gk

Keywords: TorBrowserTeam201705R added; TorBrowserTeam201704R removed

Moving review tickets to May.

comment:14 Changed 2 years ago by gk

Resolution: fixed
Status: needs_reviewclosed

Fixed with commit d70009f425f9da6a74413b286ef768ba97c899cd, thanks.

Note: See TracTickets for help on using tickets.