Opened 6 years ago

Closed 5 years ago

#9114 closed defect (fixed)

Reorganize bundle directory structure for TBB 3.0

Reported by: mikeperry Owned by: brade
Priority: Medium Milestone:
Component: Applications/Tor bundles/installation Version:
Severity: Keywords: tbb-3.0, MikePerry201310
Cc: erinn, Sebastian, mcs, brade, g.koppen@… Actual Points:
Parent ID: #4234 Points:
Reviewer: Sponsor:

Description

I think our updaters (Thandy and/or Firefox's) and a few other issues (#6457) would be simplified if we tried to reorganize our bundle structure a bit so that more of the bundle's executable bits (the parts that need updating) were grouped together with Firefox.

Doing this will require updates to Tor Launcher and the launch script, and probably several config files.

Of course, this might not matter for our updater, so maybe we don't care.

Child Tickets

Attachments (1)

0001-Bug-9114-Reorganize-bundle-directory-structure-for-T.patch (380.7 KB) - added by mcs 5 years ago.
tor-browser-bundle patch

Download all attachments as: .zip

Change History (16)

comment:1 Changed 6 years ago by gk

Cc: g.koppen@… added

comment:2 Changed 6 years ago by brade

Owner: changed from erinn to brade
Parent ID: #4234
Status: newassigned

comment:3 Changed 6 years ago by brade

Cc: erinn added

The Firefox updater will need to be modified to update files above the updater itself (there does not seem to be a way to avoid that because of the need for the "start browser" program/scripts and -- presumably -- a desire to keep the Docs and sources directories at the top level).

For the bundle directory structure, our proposal is to more closely align the platforms by changing the directory structure to be as follows:

  • All platforms will have a Tor directory which contains tor and its libraries.
  • All platforms will have a Data directory which contains two subdirectories: Tor (for torrc, geoip, etc.) and profile (for browser profile settings, addons, etc.)
  • All platforms will have a Docs directory (ChangeLog.txt)
  • All platforms will have a sources directory (bundle.inputs and versions).
  • Linux and Windows will have a "Browser" directory for Firefox and related files while Mac will keep the Contents/MacOS hierarchy it currently uses.

Below is a "hierarchical" view. (Lines ending with '>' are directories.)

** Windows / Linux **
TBB >
   Start Tor Browser.exe / start-tor-browser
   Browser >
   Tor >
   Data >
   Docs >
   sources >

** Mac **
TBB.app >
    Contents >
        MacOS >
            TorBrowser.app >
    Tor >
    Data >
    Docs >
    sources >

comment:4 Changed 6 years ago by mikeperry

Actually, we can reorganize more radically if it will minimize our changes to the updater.

We could for example put everything under the Firefox directory, if it would help..

In fact, we may want to do that anyways for #9173 (and #9091). If you agree this will simplify changes to the updater, we can call #9173 a child ticket of #4234, too, and you can count those hours as part of the updater work.

comment:5 Changed 6 years ago by mikeperry

Oh, some things to keep in mind: We will want to exempt the user's user_prefs.js, any extensions the user has installed, the torrc file, and any generated Firefox profile files, tor files, and state.

Tor has support for multiple torrc files via the --defaults-torrc option, so what we'll want to do is make sure pref changes get written to a torrc that is exempt from updating, and only update the main defaults.torrc that we ship.

My guess is that exempting these things will be easier than support for traversing beyond the parent dir, and probably cleaner anyways.. But I just wanted to give you a heads up.

comment:6 Changed 6 years ago by mcs

We made some progress on Tor Launcher:

  Bug 9114: Reorganize bundle directory structure for TBB 3.0.

  Use new default paths for tor, torrc, and tordatadir.
  Add support for torrc-defaults file.

https://gitweb.torproject.org/user/brade/tor-launcher.git/commitdiff/036281c532dd8daac953fa0da5324358c081c60a

There is no need to review this in detail now since we have not yet made the necessary changes to the packaging process that go with this -- but hopefully soon.

comment:7 Changed 6 years ago by mcs

We made one additional change to Tor Launcher (to avoid problems with use of relative paths in torrc):

Bug 9114: Reorganize bundle directory structure for TBB 3.0.

Include GeoIPFile (path) in tor command line args.

https://gitweb.torproject.org/user/brade/tor-launcher.git/commitdiff/700a2e08b2637093e6cfbab428a61a9711360347

comment:8 Changed 6 years ago by mcs

Status: assignedneeds_review

A few explanations about the patch I just attached:

  • Compared to our earlier proposal, sources was moved under Docs.
  • Under the Data directory we have:
      Data/
        Browser/
          profile.default/
          profiles.ini
        Tor/
          geoip
          torrc
          torrc-defaults
    
  • Most of the contents of torrc were moved to torrc-defaults (DataDirectory and GeoIPFile were removed; their paths are passed in on the tor command line by Tor Launcher).
  • On Mac OS, the bookmarks.html file was missing from the browser profile. We added it.
  • On Mac OS, we switched the top-level bundle's icon set to the "globe" icon (firefox.icns).
  • Some files were simply moved (not modified): bookmarks.html, extension-overrides.js, profiles.ini.
  • For some reason, our Windows builds lack webapprt, which is why we made the re-dzip conditional in our gitian-firefox.yml changes:
       #   cd $INSTDIR
    -  ~/build/re-dzip.sh App/Firefox/omni.ja
    -  ~/build/re-dzip.sh App/Firefox/webapprt/omni.ja
    -  cd ..
    -  ~/build/dzip.sh tor-browser-win32-gbuilt.zip ./FirefoxPortable/
    +  ~/build/re-dzip.sh Browser/omni.ja
    +  if [ -f Browser/webapprt/omni.ja ]; then
    +    ~/build/re-dzip.sh Browser/webapprt/omni.ja
    +  fi
    +  ~/build/dzip.sh tor-browser-win32-gbuilt.zip Browser
       cp tor-browser-win32-gbuilt.zip $OUTDIR
    

If this is just a problem with our build process, those changes should be ignored.

With these changes, the TBB package structure will be consistent across all platforms. That should help keep developers and users sane, and also make updater work easier (e.g., #4234).

comment:9 Changed 6 years ago by mikeperry

Keywords: MikePerry201308 added

comment:10 Changed 6 years ago by mikeperry

Keywords: MikePerry201309 added; MikePerry201308 removed

comment:11 Changed 5 years ago by mikeperry

Keywords: MikePerry201310 added; MikePerry201309 removed

mcs,brade: Sorry for the delay here, but is it possible for you guys to rebase your patch on top of the latest origin/master? We've changed some things in a few of the pref files and the descriptors that caused a lot of conflicts in the git-am.

comment:12 in reply to:  11 Changed 5 years ago by mcs

Replying to mikeperry:

mcs,brade: Sorry for the delay here, but is it possible for you guys to rebase your patch on top of the latest origin/master? We've changed some things in a few of the pref files and the descriptors that caused a lot of conflicts in the git-am.

Sure. Is that tor-browser-17.0.9esr-1?

comment:13 Changed 5 years ago by mikeperry

Status: needs_reviewneeds_revision

Actually this patch is against https://git.torproject.org/builders/tor-browser-bundle.git, the gitian build directory, not the browser source tree. fc14747c24c87085f6ca502d1643dd2e7a0ea7a0 is its latest commit.

comment:14 Changed 5 years ago by mcs

We rebased both patches.

For Tor Launcher, you need the following two commits from
https://git.torproject.org/user/brade/tor-launcher.git

371d37d4d83932980eef6493084b55a71c76cc8c
f644a38729a2687d095eaa992e1ea00a1d3bb0ed

(both from the tip of the bug9114 branch in that repo).

Since we don't yet have a developer repo on git.torproject.org for tor-browser-bundle, I very quickly set one up on our web server. You need the following commit from
https://pearlcrescent.com/git/user/brade/tor-browser-bundle.git

23688accb6e34c3c2af14f0cc684ace80d092ff7

(from the tip of the bug9114 branch in that repo).
I will go ahead an update the tor-browser-bundle patch that is attached to this bug as well, just in case.

Changed 5 years ago by mcs

tor-browser-bundle patch

comment:15 in reply to:  14 Changed 5 years ago by mcs

Resolution: fixed
Status: needs_revisionclosed

Replying to mcs:

...
Since we don't yet have a developer repo on git.torproject.org for tor-browser-bundle, I very quickly set one up on our web server. You need the following commit from
https://pearlcrescent.com/git/user/brade/tor-browser-bundle.git

23688accb6e34c3c2af14f0cc684ace80d092ff7

(from the tip of the bug9114 branch in that repo).

This is now in a torproject.org repo at:
https://gitweb.torproject.org/user/brade/tor-browser-bundle.git/commit/23688accb6e34c3c2af14f0cc684ace80d092ff7

(recorded here for posterity).

Mike already merged this into the main repo, so I am resolving this bug as fixed.

Note: See TracTickets for help on using tickets.