Updates are not getting properly applied when trying to update to 6.5a4(-hardened) on Linux

added TorBrowserTeam201611R component::applications/tor browser owner::tbb-team priority::immediate resolution::fixed severity::critical status::closed type::defect labels

That's what I get with a test bundle

AUS:SVC Downloader:onStopRequest - status: 0, current fail: 0, max fail:
10, retryTimeout: 2000
AUS:SVC Downloader:_verifyDownload called
AUS:SVC Downloader:_verifyDownload downloaded size == expected size.
AUS:SVC Downloader:_verifyDownload hashes match.
AUS:SVC Downloader:onStopRequest - setting state to: pending
AUS:SVC Downloader:onStopRequest - attempting to stage update:
Tor-Browser 6.5a4
AUS:SVC readStatusFile - status: failed: 6, path:
/home/thomas/Arbeit/Tor/tor-browser-bundle/tor-browser_de/Browser/TorBrowser/UpdateInfo/updates/0/update.status
AUS:SVC handleFallbackToCompleteUpdate - install of partial patch
failed, downloading complete patch

IIRC the complete update showed the same error message.

I found signed mar files here: https://people.torproject.org/~gk/builds/6.5a4/ and was able to reproduce this problem by hosting an incremental mar file on my own server. My last-update.log file contained the following:

Performing a staged update
PATCH DIRECTORY /home/brade/Desktop/tb-test/Browser/TorBrowser/UpdateInfo/updates/0
INSTALLATION DIRECTORY /home/brade/Desktop/tb-test/Browser
WORKING DIRECTORY /home/brade/Desktop/tb-test/Browser/updated
ensure_copy: failed to open the file for reading: /home/brade/Desktop/tb-test/Browser/TorBrowser/Data/Tor/control.socket, err: 6
failed: 6
calling QuitProgressUI

It seems that the updater code is not smart enough to skip Unix domain sockets when copying the installation directory during the staged update process. I was able to update successfully on a 64-bit Linux system after I used about:config to set app.update.staging.enabled to false. The bad news is that as far as I know we cannot disable staged updates from the update manifest (server-side XML). Using the full mar file will result in the same problem as gk experienced.

I will try to think of a workaround but at the moment I do not have any ideas. I will ask Kathy about this problem in the morning.

A small bit of good news: I did not do any testing, but this problem should not occur on Windows (no Unix domain sockets and no staged updates either).

On OSX, I am able to successfully apply the update (the control port Unix domain socket is in the TorBrowser-Data directory, which is outside the scope of what the updater touches, so the ensure_copy problem does not occur). But on the OSX system that I used for testing, after the update was applied and the browser restarted, tor failed to start. This happened because quotes are missing from the ControlPort unix:... argument which was passed on the tor command line by Tor Launcher (I have spaces in my path). But we fixed that problem in Tor Launcher! After a little more investigation I learned that Tor Launcher was not updated in the profile or in the TorBrowser.app/Contents/Resources/distribution/extensions directory. I don't know why yet though, and we might need a new ticket for OSX. The dmg files have the correct Tor Launcher. I have not checked the complete mar files and I have not done enough analysis to determine if the incremental mar files are flawed or if the updater failed to update the xpi properly.

Replying to mcs:

I found signed mar files here: https://people.torproject.org/~gk/builds/6.5a4/ and was able to reproduce this problem by hosting an incremental mar file on my own server. My last-update.log file contained the following: {{{ Performing a staged update PATCH DIRECTORY /home/brade/Desktop/tb-test/Browser/TorBrowser/UpdateInfo/updates/0 INSTALLATION DIRECTORY /home/brade/Desktop/tb-test/Browser WORKING DIRECTORY /home/brade/Desktop/tb-test/Browser/updated ensure_copy: failed to open the file for reading: /home/brade/Desktop/tb-test/Browser/TorBrowser/Data/Tor/control.socket, err: 6 failed: 6 calling QuitProgressUI }}}

It seems that the updater code is not smart enough to skip Unix domain sockets when copying the installation directory during the staged update process. I was able to update successfully on a 64-bit Linux system after I used about:config to set app.update.staging.enabled to false. The bad news is that as far as I know we cannot disable staged updates from the update manifest (server-side XML). Using the full mar file will result in the same problem as gk experienced.

I will try to think of a workaround but at the moment I do not have any ideas. I will ask Kathy about this problem in the morning.

A small bit of good news: I did not do any testing, but this problem should not occur on Windows (no Unix domain sockets and no staged updates either).

Thanks for the whole analysis so far! Just to reply to this bit: you are right. boklm and I tested updating to 6.5a4 on Windows and it worked both by using the partial and the full .mar. We have therefore enabled auto-updating for Windows users again.

Trac:
Summary: Alpha updates are not getting applied when trying to update to 6.5a4(-hardened) to Updates are not getting properly applied when trying to update to 6.5a4(-hardened) on OS X and Linux

Replying to mcs:

On OSX, I am able to successfully apply the update (the control port Unix domain socket is in the TorBrowser-Data directory, which is outside the scope of what the updater touches, so the ensure_copy problem does not occur). But on the OSX system that I used for testing, after the update was applied and the browser restarted, tor failed to start. This happened because quotes are missing from the ControlPort unix:... argument which was passed on the tor command line by Tor Launcher (I have spaces in my path). But we fixed that problem in Tor Launcher! After a little more investigation I learned that Tor Launcher was not updated in the profile or in the TorBrowser.app/Contents/Resources/distribution/extensions directory. I don't know why yet though, and we might need a new ticket for OSX. The dmg files have the correct Tor Launcher. I have not checked the complete mar files and I have not done enough analysis to determine if the incremental mar files are flawed or if the updater failed to update the xpi properly.

Interesting. I made a couple of tests taking a fresh 6.0.5 and a fresh 6.5a3. I tested updating the stable both in a desktop and an /Applications location and it worked: the extensions and everything seemed to be fine. Note, though, that we only have full updates right now for the stable series.

For the alpha I tested just installing it on the desktop and applying the MAR files manually. Both the incremental and the full MAR file applied correctly and the extensions were updated, too.

Replying to gk:

Replying to mcs:

On OSX, I am able to successfully apply the update (the control port Unix domain socket is in the TorBrowser-Data directory, which is outside the scope of what the updater touches, so the ensure_copy problem does not occur). But on the OSX system that I used for testing, after the update was applied and the browser restarted, tor failed to start. This happened because quotes are missing from the ControlPort unix:... argument which was passed on the tor command line by Tor Launcher (I have spaces in my path). But we fixed that problem in Tor Launcher! After a little more investigation I learned that Tor Launcher was not updated in the profile or in the TorBrowser.app/Contents/Resources/distribution/extensions directory. I don't know why yet though, and we might need a new ticket for OSX. The dmg files have the correct Tor Launcher. I have not checked the complete mar files and I have not done enough analysis to determine if the incremental mar files are flawed or if the updater failed to update the xpi properly.

Interesting. I made a couple of tests taking a fresh 6.0.5 and a fresh 6.5a3. I tested updating the stable both in a desktop and an /Applications location and it worked: the extensions and everything seemed to be fine. Note, though, that we only have full updates right now for the stable series.

For the alpha I tested just installing it on the desktop and applying the MAR files manually. Both the incremental and the full MAR file applied correctly and the extensions were updated, too.

I tested now putting 6.5a3 to /Applications (getting a space in the user dir path) and applying the incremental update manually worked as well. So, hm.

Replying to gk:

For the alpha I tested just installing it on the desktop and applying the MAR files manually. Both the incremental and the full MAR file applied correctly and the extensions were updated, too.

I tested now putting 6.5a3 to /Applications (getting a space in the user dir path) and applying the incremental update manually worked as well. So, hm.

This morning I tried updating an existing/older 6.5a3 installation on another OSX system and it worked fine. Based on this and the fact that all of your tests worked, I think the OSX updates are probably okay. Kathy and I will try again on the OSX system that failed and let you know what we find, but I think you should re-enable updates for OSX.

Replying to mcs:

Kathy and I will try again on the OSX system that failed and let you know what we find, but I think you should re-enable updates for OSX.

It turns out that problem was a false alarm. The alpha install I tested with was an older one, and last night it updated from 6.5a2 to 6.5a3 (not a4), which led to the "oh no, things do not work due to a space in the path" behavior. So, never mind.

Trac:
Summary: Updates are not getting properly applied when trying to update to 6.5a4(-hardened) on OS X and Linux to Updates are not getting properly applied when trying to update to 6.5a4(-hardened) on Linux

It is possible that we missed something, but after reading the updater and update service code, Kathy and I have concluded that the only workarounds are:

Users can set app.update.staging.enabled to false before attempting the update.
Users can disable the control port Unix domain socket by setting extensions.torlauncher.control_port_use_socket to false and restarting their browser before attempting the update.

The other thing to think about is "what will happen during the next update, i.e., 6.5a4 to 6.5a5?" The answer is that updates may fail (since the 6.5a4 updater which we already shipped is flawed in the same way as the 6.5a3 one). There is some good news though: because of the fix for #20185 (moved), users who have XDG_RUNTIME_DIR set (most people?) will not encounter this bug because the Unix domain sockets will be outside of the TB installation directory.

If XDG_RUNTIME_DIR is not set, similar workarounds will be needed for the 6.5a4 to 6.5a5 update. Note that the "disable Unix domain socket" prefs in 6.5a4 are extensions.torlauncher.control_port_use_ipc and extensions.torlauncher.socks_port_use_ipc (both would need to be set to false).

There are also prefs that control the location of the Unix domain sockets; these could be used to ensure that the sockets are created somewhere outside of the installation directory.

Also, I thought we had an open ticket for "create an automated test for the updater" but I cannot find it at the moment.

Replying to mcs:

Also, I thought we had an open ticket for "create an automated test for the updater" but I cannot find it at the moment.

#17662 (moved).

Replying to mcs:

It is possible that we missed something, but after reading the updater and update service code, Kathy and I have concluded that the only workarounds are:

Users can set app.update.staging.enabled to false before attempting the update.

Users can disable the control port Unix domain socket by setting extensions.torlauncher.control_port_use_socket to false and restarting their browser before attempting the update.

The other thing to think about is "what will happen during the next update, i.e., 6.5a4 to 6.5a5?" The answer is that updates may fail (since the 6.5a4 updater which we already shipped is flawed in the same way as the 6.5a3 one). There is some good news though: because of the fix for #20185 (moved), users who have XDG_RUNTIME_DIR set (most people?) will not encounter this bug because the Unix domain sockets will be outside of the TB installation directory.

If XDG_RUNTIME_DIR is not set, similar workarounds will be needed for the 6.5a4 to 6.5a5 update. Note that the "disable Unix domain socket" prefs in 6.5a4 are extensions.torlauncher.control_port_use_ipc and extensions.torlauncher.socks_port_use_ipc (both would need to be set to false).

There are also prefs that control the location of the Unix domain sockets; these could be used to ensure that the sockets are created somewhere outside of the installation directory.

Thanks for looking into it. So, what do you suggest? Enabling the updates for Linux as well and doing a final update to our blog post? It seems in the worst case users are downloading the update again and again until they get the "Update failed x times" dialog. Can we say something sensibly on it conveying the current issue (with some attribute in the XML files)? I guess not? Or would we just leave the updater disabled and getting the onion to flash getting users to find the "Check for Tor Browser Update..." in the Torbutton menu? I guess this is suboptimal, at least for the reason that it is probably an unusal update flow even for alpha users.

Replying to gk:

Thanks for looking into it. So, what do you suggest? Enabling the updates for Linux as well and doing a final update to our blog post?

Yes, because we do want users to get the update.

It seems in the worst case users are downloading the update again and again until they get the "Update failed x times" dialog. Can we say something sensibly on it conveying the current issue (with some attribute in the XML files)? I guess not? Or would we just leave the updater disabled and getting the onion to flash getting users to find the "Check for Tor Browser Update..." in the Torbutton menu? I guess this is suboptimal, at least for the reason that it is probably an unusal update flow even for alpha users.

I don't think we can add text to that dialog, but there is a "billboard" feature that we have never used that might be helpful in this case. (note that Mozilla recently remove that feature; see https://bugzilla.mozilla.org/show_bug.cgi?id=1296207). I tried it and it seems to work. You can create an HTML page that is displayed instead of the standard update prompt, and there is also a way to force the prompt to be shown even for automatic/background updates. Here is a sample update manifest; note the showPrompt and billboard attributes:

<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="6.5a4" appVersion="6.5a4"
            platformVersion="45.5.0"
            buildID="20000101000000"
            showPrompt="true"
            billboardURL="https://example.com/tor/update_2/alpha/billboard.html"
            detailsURL="https://example.com/tor/details.html">
        <patch type="complete"
               URL="http://example.com/tor/update_2/alpha/mars/tor-browser-linux64-6.5a4_en-US.mar"
               hashFunction="SHA512"
               hashValue="ce9e233cac4495b803b08243603ccaf3c968c0ecb48f725e9ccb1a2b9d61c0b78cd71ecd324ed3b1a71875a94ecaeff0018ad059c139eaf61f37ad432fe8befe"
               size="89297856"/>
    </update>
</updates>

The other thing that took Kathy and me some time to figure out is that the HTML file must include a billboard attribute on the body, e.g.,

<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Billboard Test</title>
</head>
<body billboard="true">
This is a test.
<p>
Useful info should be placed here.
</body>
</html>

I will attach a screenshot that shows what this looks like. The text should include "Tor Browser 6.5a4 is available" plus whatever you include in the blog to help people work around this update issue (which most people will be affected by).

Trac:

billboard sample

Billboard sample:

Thanks. I guess we could try that billboard feature. But I won't have the energy to work on that anymore today. boklm: If you have, feel free to do so. I guess testing with the hardened series first is the way to go.

Kathy and I created a patch for the underlying updater bug. Please review. https://gitweb.torproject.org/user/brade/tor-browser.git/commit/?h=bug20691-01&id=d2225a513147e33a9d50476174ca053cb9107880

Trac:
Keywords: TorBrowserTeam201611 deleted, TorBrowserTeam201611R added
Status: new to needs_review

Updates are not getting properly applied when trying to update to 6.5a4(-hardened) on Linux

Child items ...

Activity