Opened 14 months ago

Closed 13 days ago

#27604 closed defect (fixed)

Relocating the Tor Browser directory is broken with Tor Browser 8

Reported by: gk Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Major Keywords: tbb-8.0-issues, tbb-regression, tbb-8.0.1-can, TorBrowserTeam201910R, tbb-backport
Cc: user6969, catalyst, salah, mcs, brade Actual Points: 3
Parent ID: Points: 0.5
Reviewer: Sponsor:

Description

In Tor Browser 7 users were able to extract Tor Browser in directory foo run it and copy it over to bar and run it. Starting with Tor Browser 8 this does not seem to be possible anymore. See, for instance, the respective report on our blog: https://blog.torproject.org/comment/277011#comment-277011.

Child Tickets

Attachments (1)

error.png (13.6 KB) - added by morar 3 weeks ago.

Download all attachments as: .zip

Change History (39)

comment:1 Changed 14 months ago by gk

Keywords: tbb-8.0.1-can added

Marking for 8.0.1 can.

comment:2 Changed 14 months ago by mcs

Kathy and I cannot reproduce the crash, but any action that causes the browser profile to be renamed or moved breaks things. The specific problem which we observed on both macOS and Linux is that Torbutton and Tor Launcher are not loaded after a rename or move.

A workaround is to delete the extensions.json file, which contains full paths. Probably that file is supposed to be recreated automatically when the profile is moved. Webextensions do not seem to suffer the same fate even though extensions.json contains their paths as well (which are outdated after the profile move).

comment:3 Changed 13 months ago by traumschule

#28199 is a duplicate.

comment:4 Changed 9 months ago by gk

Summary: Relocating the Tor Browser directory is broken with Tor Browser 8 on LinuxRelocating the Tor Browser directory is broken with Tor Browser 8

FWIW, this happens on Windows as well it seems, see: https://blog.torproject.org/comment/279768#comment-279768

comment:5 Changed 8 months ago by gk

Cc: user6969 added

Closed #27574 and #29711 as duplicates.

comment:6 Changed 7 months ago by cypherpunks

win x64, tb 8.0.8 and still... the same...

comment:7 Changed 7 months ago by catalyst

Cc: catalyst added

comment:8 Changed 7 months ago by ayylmao

why isn't this fixed already? my tor browser is up to date and if i move the folder somewhere else than the location where i first installed it and try to launch the program i can't get it to work. i thought tor was supposed to be portable

comment:9 in reply to:  8 Changed 7 months ago by gk

Replying to ayylmao:

why isn't this fixed already? my tor browser is up to date and if i move the folder somewhere else than the location where i first installed it and try to launch the program i can't get it to work. i thought tor was supposed to be portable

It isn't fixed yet because nobody has worked on it so far. But it could be you to solve this, helping us out! It would be really appreciated.

comment:10 Changed 5 months ago by teconmoon

A more permanent workaround for me was to delete extensions.json, create an empty 0 KB file in its place, and mark it as read-only. That forced the browser to use a temporary extensions.json for each session, and kept me from having to delete the file every time the directory changed.

comment:11 in reply to:  10 Changed 5 months ago by gk

Keywords: TorBrowserTeam201906 added

Replying to teconmoon:

A more permanent workaround for me was to delete extensions.json, create an empty 0 KB file in its place, and mark it as read-only. That forced the browser to use a temporary extensions.json for each session, and kept me from having to delete the file every time the directory changed.

Wow, that's painful. I guess what we could do is patching XPIProviderUtils.js's parseDB() and add a sanity check making sure the path of the extensions in the DB exists. If not we need to recreate the DB. I suspect that would be something Mozilla would take if we upstreamed it.

We could as well just delete the json file during browser shutdown. However, that seems to be a bit too excessive.

comment:12 Changed 5 months ago by Thorin

FWIW, John Haller from PortableApps applied a fix (see https://portableapps.com/node/60299 , and comment 9), but I'm not entirely sure what he did, but also had an additional issue (see https://portableapps.com/node/60496 ) with addonStartup.json.lz4 (edit: which was added in FF67).

Let me know if you would like me to reach out to John Halter and ask what he did, but I think you know what needs to be done already, and getting this done upstream is the best solution IMO.

Update: https://portableapps.com/comment/240579#comment-240579 -> John's reply

Last edited 5 months ago by Thorin (previous) (diff)

comment:15 Changed 4 months ago by gk

Keywords: TorBrowserTeam201907 added; TorBrowserTeam201906 removed

Moving tickets to July

comment:16 Changed 6 weeks ago by gk

Cc: salah added

#31894 is a duplicate.

Last edited 6 weeks ago by gk (previous) (diff)

comment:17 Changed 5 weeks ago by gk

Keywords: TorBrowserTeam201910 GeorgKoppen201910 added; TorBrowserTeam201907 removed
Points: 0.5

So, this was broken in 8.x in a way that makes the whole experience failing hard which is kind of good. I wonder how the breakage will look like in 9.0 given that Tor Launcher and Torbutton are now no separate extensions anymore. The fear here is that the user retains now a somewhat usable Tor Browser but broken in subtle and dangerous ways. We should double-check that and fix it if necessary.

comment:18 Changed 4 weeks ago by gk

Cc: mcs brade added
Keywords: GeorgKoppen201910 removed

mcs and brade are picking that one up.

comment:19 Changed 4 weeks ago by mcs

This problem still exists with Tor Browser 9.0a8. After renaming the profile directory, HTTPS-E and NoScript are both broken. On macOS we had to remove the following three files in order to make things work again (I assume this is also true for Windows and Linux, although the file paths are different):

TorBrowser-Data/Browser/XXX.default/extensions.json
TorBrowser-Data/Browser/Caches/XXX.default/startupCache/webext.sc.lz4
TorBrowser-Data/Browser/XXX.default/addonStartup.json.lz4

Looking at the code, each is handled in a different place.

For extensions.json, the solution described here should work: https://bugzilla.mozilla.org/show_bug.cgi?id=1429838#c11

For webext.sc.lz4, we could use a similar solution: we could modify _readData() inside components/extensions/ExtensionParent.jsm to check extension paths and ignore the cached data if one or more files pointed to by the paths no longer exist.

For addonStartup.json.lz4, we could again use a similar solution: we could modify loadExtensionState() inside toolkit/mozapps/extensions/internal/XPIProvider.jsm to check extension paths and ignore the cached data if paths have changed.

All of the above said, it is too risky to fix this in Tor Browser 9.0. Maybe something for 9.5alpha?

comment:20 in reply to:  19 ; Changed 4 weeks ago by gk

Replying to mcs:

This problem still exists with Tor Browser 9.0a8. After renaming the profile directory, HTTPS-E and NoScript are both broken. On macOS we had to remove the following three files in order to make things work again

What does that mean? Is the browser usable now but not as expected because NoScript and HTTPS-E are missing?

comment:21 in reply to:  20 ; Changed 4 weeks ago by mcs

Replying to gk:

Replying to mcs:

This problem still exists with Tor Browser 9.0a8. After renaming the profile directory, HTTPS-E and NoScript are both broken. On macOS we had to remove the following three files in order to make things work again

What does that mean? Is the browser usable now but not as expected because NoScript and HTTPS-E are missing?

Sorry for not providing more detail. The browser is usable but HTTPS-E and NoScript are not working correctly. For example, JS is executed on regular (non-SSL) pages even when the security level is set to Safer. I am 98% sure that the extension code is not loaded at all because this is visible on the browser console after startup:

Error while loading 'jar:file:///Users/XXX/Desktop/Test1/TorBrowser-Data/Browser/l98sbeb6.default/extensions/https-everywhere-eff@eff.org.xpi!/manifest.json' (NS_ERROR_FILE_NOT_FOUND) Extension.jsm:513

(/Users/XXX/Desktop/Test1 is the original folder name, prior to renaming).

The Tor Launcher and the Torbutton functionality (such as circuit display) are not broken in any obvious way.

Other things: after the rename, about:addons shows both HTTPS-E and NoScript but the extension icons show up as broken images. The same broken image are shown on about:debugging, and debugging does not work (no surprise). And if I add one or both of the HTTPS-E/NoScript icons to the browser toolbar before renaming the directory that contains TorBrowser-Data, the toolbar icons are gone after the rename.

comment:22 in reply to:  21 Changed 4 weeks ago by gk

Priority: MediumHigh
Severity: NormalMajor

Replying to mcs:

Replying to gk:

Replying to mcs:

This problem still exists with Tor Browser 9.0a8. After renaming the profile directory, HTTPS-E and NoScript are both broken. On macOS we had to remove the following three files in order to make things work again

What does that mean? Is the browser usable now but not as expected because NoScript and HTTPS-E are missing?

Sorry for not providing more detail. The browser is usable but HTTPS-E and NoScript are not working correctly. For example, JS is executed on regular (non-SSL) pages even when the security level is set to Safer. I am 98% sure that the extension code is not loaded at all because this is visible on the browser console after startup:

Error while loading 'jar:file:///Users/XXX/Desktop/Test1/TorBrowser-Data/Browser/l98sbeb6.default/extensions/https-everywhere-eff@eff.org.xpi!/manifest.json' (NS_ERROR_FILE_NOT_FOUND) Extension.jsm:513

(/Users/XXX/Desktop/Test1 is the original folder name, prior to renaming).

The Tor Launcher and the Torbutton functionality (such as circuit display) are not broken in any obvious way.

Other things: after the rename, about:addons shows both HTTPS-E and NoScript but the extension icons show up as broken images. The same broken image are shown on about:debugging, and debugging does not work (no surprise). And if I add one or both of the HTTPS-E/NoScript icons to the browser toolbar before renaming the directory that contains TorBrowser-Data, the toolbar icons are gone after the rename.

Okay, that's what I feared, thanks. Could you pick this up for our alpha series?

comment:23 Changed 3 weeks ago by mcs

Actual Points: 0.6

Added brade/mcs points so far (0.6).

comment:24 Changed 3 weeks ago by morar

I think that another problem is the torrc file where there is:

DataDirectory C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor
GeoIPFile C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip
GeoIPv6File C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip6

When Relocating the Tor Browser directory, this are not updated.

comment:25 in reply to:  24 ; Changed 3 weeks ago by Thorin

Replying to morar:

I think that another problem is...

I was going to post something about this. With the release of ESR68 based stable, and since I need to keep various releases around (FP tests, regressions etc), I decided to cleanup my folder naming convention for the ESR60 based stable -> e.g. to TB60, TB60-32bit. Once I did this, the browser opens but Tor doesn't. It doesn't even try to connect. Not even a failed message or dialog or anything.

I guess morar has nailed the root of the issue

Last edited 3 weeks ago by Thorin (previous) (diff)

comment:26 in reply to:  25 ; Changed 3 weeks ago by mcs

Replying to Thorin:

I was going to post something about this. With the release of ESR68 based stable, and since I need to keep various releases around (FP tests, regressions etc), I decided to cleanup my folder naming convention for the ESR60 based stable -> e.g. to TB60, TB60-32bit. Once I did this, the browser opens but Tor doesn't. It doesn't even try to connect. Not even a failed message or dialog or anything.

I guess morar has nailed the root of the issue

Are you sure? Tor Launcher passes values for those torrc parameters in the arg list / command line when it starts tor, which means they should automatically be corrected the next time you open the browser. Did you delete extensions.json? In your ESR60-based browsers, Tor Launcher is possibly not being loaded at all.

comment:27 Changed 3 weeks ago by morar

I made some tests:
1) Relocating the Tor Browser 7.5.6, restarted and loaded Tor Browser 7.5.6, torrc not updated.

2) Relocating the Tor Browser 8.5.5 after delete extensions.json, restarted and loaded Tor Browser 8.5.5, torrc not updated.

3) Relocating the Tor Browser 9.0, restarted and loaded Tor Browser 9.0, torrc not updated.

In all three cases

DataDirectory C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor
GeoIPFile C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip
GeoIPv6File C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip6

are not updated with the new folder.

Last edited 3 weeks ago by morar (previous) (diff)

comment:28 in reply to:  27 Changed 3 weeks ago by mcs

Replying to morar:

I made some tests:
...
In all three cases

DataDirectory C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor
GeoIPFile C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip
GeoIPv6File C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip6

are not updated with the new folder.

Thanks for testing. In those cases, did Tor Browser open and work correctly (after you applied workarounds including deleting the extensions.json file)? I realized yesterday that even though new values for config parameters such as DataDirectory are passed to tor via args (and therefore used), the torrc file will not be overwritten unless you make other config changes. In other words, tor doesn't write out an updated torrc in response to new command line args.

Last edited 3 weeks ago by mcs (previous) (diff)

Changed 3 weeks ago by morar

Attachment: error.png added

comment:29 Changed 3 weeks ago by morar

With Tor Browser 7.5.6 and Tor Browser 8.5.5 i can surf the internet.
But i think that some command don't work like ExitNodes {DE}.

Today, after replacing Tor Browser 9.0, i have a critical error.
When Tor Browser starts, i only see a window with a gray background and error:

XML Parsing Error: undefined entity
Location: chrome://browser/content/browser.xul
Line Number 60, Column 1:

<window id="main-window"
^

See attachment.

If I put the folder back in the original place, the error disappears.

Last edited 3 weeks ago by morar (previous) (diff)

comment:30 Changed 3 weeks ago by Thorin

#32302 is a duplicate

comment:31 in reply to:  26 ; Changed 2 weeks ago by Thorin

Replying to mcs:

Are you sure? ... Did you delete extensions.json?...

I didn't delete or do anything special, just renamed the dir. After the rename, I tried several times but it just continued to skip the tor connection part/dialog completely. I ended up just deleting those TB instances, and set up two (64/32bit) brand new legacy 8.5.5's with updates disabled. I also set up two version 9 stables, and I have my two alphas.

FWIW, on all six, I cannot replicate: make a copy, rename dir, start, and I always get success (i.e tor connection dialog etc, browsing works). Maybe I'm weird, because the extensions (NS, HTTPS-E) also load and work the first time.

So IDK what went on, and I no longer have the originals which were initially clean v8.0 setups with updates. Chalk it up to experience :)

comment:32 Changed 2 weeks ago by acat

Keywords: TorBrowserTeam201910R added; TorBrowserTeam201910 removed
Status: newneeds_review

Patch for review: https://github.com/acatarineu/tor-browser/commit/27604

I think there are (at least) three different issues here:

One is the problem of extensions being broken that mcs mentioned, which I think was introduced by https://bugzilla.mozilla.org/show_bug.cgi?id=1512436. More concretely, the problem seems to be the new rootURI field in addonStartup.json which is absolute, unlike the path one which is serialized as relative: https://searchfox.org/mozilla-esr68/rev/22bae08f58d48ff86e02d5bbd12e5630af148d6f/toolkit/mozapps/extensions/internal/XPIProvider.jsm#555. I think this was needed to have built-in addons with resource:// rootURI like resource://gre/modules/themes/default/. Fixed this by always overriding it here if this.file exists (should not happen with addons with resource:// rootURIs).

The other is some special treatment that langpacks get due to https://bugzilla.mozilla.org/show_bug.cgi?id=1492459. These are flagged as missing early here (currentModifiedTime is set here only for langpacks) if the (old) path of the extension xpi does not exist. If that's the case the langpack is removed in https://searchfox.org/mozilla-esr68/rev/715f10032bb8be971ff0e9846e12be58afad4950/toolkit/mozapps/extensions/internal/XPIDatabase.jsm#3143. That seems to fallback to English, but on browser restart it's completely broken, showing the same error message mentioned in comment:29 and in bugzilla. I'm not so sure what's the best way to fix this, in the proposed patch I'm checking for the (new) extension file existing before flagging it as missing.

Third is the issue of extensions.json (and possibly webext.sc.lz4) paths not being updated, but I'm not so sure this has functionality impact if the two previous issues are fixed. In any case, I think making scanForChanges return true when the path of some addon location has changed will do the trick here and trigger an update of extensions.json. I verified that webext.sc.lz4 paths are also updated, although I did not investigate what exactly in the code is updating the latter.

With the patch I cannot reproduce the issues anymore, verified on Linux and Windows. I also verified that a profile which is in the bad state of comment:29 is fixed.

I think it's worth filing bugzillas or updating the existing ones and try to get this fixed in Firefox (and also get some feedback on the suggested patches).

comment:33 in reply to:  31 ; Changed 2 weeks ago by acat

Replying to Thorin:

Replying to mcs:

Are you sure? ... Did you delete extensions.json?...

I didn't delete or do anything special, just renamed the dir. After the rename, I tried several times but it just continued to skip the tor connection part/dialog completely. I ended up just deleting those TB instances, and set up two (64/32bit) brand new legacy 8.5.5's with updates disabled. I also set up two version 9 stables, and I have my two alphas.

FWIW, on all six, I cannot replicate: make a copy, rename dir, start, and I always get success (i.e tor connection dialog etc, browsing works). Maybe I'm weird, because the extensions (NS, HTTPS-E) also load and work the first time.

So IDK what went on, and I no longer have the originals which were initially clean v8.0 setups with updates. Chalk it up to experience :)

These are the steps I followed to reproduce in Linux.

  1. Extract tor-browser 9 to a new folder, so that tor-browser_{locale} is the only one there.
  2. Run tor-browser.
  3. Close it.
  4. Rename tor-browser_{locale} to something else.
  5. Run again.
  6. Check that the extension icons in Customize... are gone (and the other issues mcs mentioned in comment:21).

Can you confirm whether you still cannot reproduce with these steps?

comment:34 Changed 2 weeks ago by acat

Actual Points: 0.63

comment:35 in reply to:  33 Changed 2 weeks ago by Thorin

Replying to acat:

  1. Rename tor-browser_{locale} to something else.

Can you confirm whether you still cannot reproduce with these steps?

I wasn't thinking in my re-test, it helps to not leave the original dir lying around :facepalm: - on Windows with 9.0 (and I did not delete any files)

First run after dir rename

  • the extension icons are gone in customize and about:addons
  • console errors eg Error while loading 'jar:file:///D:/TBTest/TB1/Browser/TorBrowser/Data/Browser/profile.default/extensions/...
  • torcc file shows the original path TB1, not the new path TB2

Second (and subsequent) run after dir rename:

  • NoScript seems fine: no console errors, icons present, test with safest setting did not execute JS
  • HTTPS-E is not: same console error, missing icon in customize and about:addons
  • torcc file updated to new path TB2

see below the first line is still looking for TB1

Error while loading 'jar:file:///D:/TBTest/TB1/Browser/TorBrowser/Data/Browser/profile.default/extensions/https-everywhere-eff@eff.org.xpi!/manifest.json' (NS_ERROR_FILE_NOT_FOUND) Extension.jsm:513
    readJSON resource://gre/modules/Extension.jsm:513
    onStopRequest resource://gre/modules/NetUtil.jsm:128
    _openNetworkSettings jar:file:///D:/TBTest/TB2/Browser/browser/omni.ja!/chrome/torlauncher/components/tl-process.js:751
    _controlTor jar:file:///D:/TBTest/TB2/Browser/browser/omni.ja!/chrome/torlauncher/components/tl-process.js:586
    TorStartAndControlTor jar:file:///D:/TBTest/TB2/Browser/browser/omni.ja!/chrome/torlauncher/components/tl-process.js:333
    observe jar:file:///D:/TBTest/TB2/Browser/browser/omni.ja!/chrome/torlauncher/components/tl-process.js:135
Last edited 2 weeks ago by Thorin (previous) (diff)

comment:36 in reply to:  32 Changed 13 days ago by gk

Keywords: TorBrowserTeam201910 added; TorBrowserTeam201910R removed
Status: needs_reviewneeds_revision

Replying to acat:

Patch for review: https://github.com/acatarineu/tor-browser/commit/27604

I think there are (at least) three different issues here:

One is the problem of extensions being broken that mcs mentioned, which I think was introduced by https://bugzilla.mozilla.org/show_bug.cgi?id=1512436. More concretely, the problem seems to be the new rootURI field in addonStartup.json which is absolute, unlike the path one which is serialized as relative: https://searchfox.org/mozilla-esr68/rev/22bae08f58d48ff86e02d5bbd12e5630af148d6f/toolkit/mozapps/extensions/internal/XPIProvider.jsm#555. I think this was needed to have built-in addons with resource:// rootURI like resource://gre/modules/themes/default/. Fixed this by always overriding it here if this.file exists (should not happen with addons with resource:// rootURIs).

The other is some special treatment that langpacks get due to https://bugzilla.mozilla.org/show_bug.cgi?id=1492459. These are flagged as missing early here (currentModifiedTime is set here only for langpacks) if the (old) path of the extension xpi does not exist. If that's the case the langpack is removed in https://searchfox.org/mozilla-esr68/rev/715f10032bb8be971ff0e9846e12be58afad4950/toolkit/mozapps/extensions/internal/XPIDatabase.jsm#3143. That seems to fallback to English, but on browser restart it's completely broken, showing the same error message mentioned in comment:29 and in bugzilla. I'm not so sure what's the best way to fix this, in the proposed patch I'm checking for the (new) extension file existing before flagging it as missing.

Third is the issue of extensions.json (and possibly webext.sc.lz4) paths not being updated, but I'm not so sure this has functionality impact if the two previous issues are fixed. In any case, I think making scanForChanges return true when the path of some addon location has changed will do the trick here and trigger an update of extensions.json. I verified that webext.sc.lz4 paths are also updated, although I did not investigate what exactly in the code is updating the latter.

With the patch I cannot reproduce the issues anymore, verified on Linux and Windows. I also verified that a profile which is in the bad state of comment:29 is fixed.

I think it's worth filing bugzillas or updating the existing ones and try to get this fixed in Firefox (and also get some feedback on the suggested patches).

Yes, please do update https://bugzilla.mozilla.org/show_bug.cgi?id=1429838 so we can get some feedback there as well.

That said this looks pretty good. One thing we should fix is getting

JavaScript error: resource://gre/modules/ExtensionCommon.jsm, line 2318: Error: listener not re-registered
JavaScript error: resource://gre/modules/ExtensionCommon.jsm, line 2318: Error: listener not re-registered
JavaScript error: resource://gre/modules/ExtensionCommon.jsm, line 2318: Error: listener not re-registered
JavaScript error: resource://gre/modules/ExtensionCommon.jsm, line 2318: Error: listener not re-registered
JavaScript error: resource://gre/modules/ExtensionCommon.jsm, line 2318: Error: listener not re-registered
JavaScript error: resource://gre/modules/ExtensionCommon.jsm, line 2318: Error: listener not re-registered
JavaScript error: resource://gre/modules/ExtensionCommon.jsm, line 2318: Error: listener not re-registered
JavaScript error: resource://gre/modules/ExtensionCommon.jsm, line 2318: Error: listener not re-registered

after applying your patches. (I blew the startup cache in the profile directory away before I started a Tor Browser modified with your patches) Re-locating does not matter I start to get those JS errors even if I keep the current location.

Last edited 13 days ago by gk (previous) (diff)

comment:37 Changed 13 days ago by acat

I can't reproduce... Which exact files did you remove to get these?

comment:38 in reply to:  37 Changed 13 days ago by gk

Keywords: TorBrowserTeam201910R tbb-backport added; TorBrowserTeam201910 removed
Resolution: fixed
Status: needs_revisionclosed

Replying to acat:

I can't reproduce... Which exact files did you remove to get these?

Hrm. It seems I can't reproduce that anymore with clean bundles, sorry for the noise. I wonder what I did do differently before, though... Anyway, nice work! I cherry-picked you patch onto tor-browser-68.2.0esr-9.5-1 (commit 7d5f879bba1e81ee64576e882f9d8916a2bc471e).

It will be interesting to see what Mozilla folks are saying. One thing we should not forget when upstreaming is including a test that makes sure this scenario is still working. It seems over time more and more functionality crept in that broke the relocation scenario.

We miiiight want to backport the changes at some point but not sure. This will probably be sysrqb's decision.

Note: See TracTickets for help on using tickets.