Opened 3 years ago

Closed 11 months ago

#4261 closed task (fixed)

Use .dmg rather than .zip for Mac OS X

Reported by: ioerror Owned by: hellias
Priority: major Milestone:
Component: Tor bundles/installation Version:
Keywords: tbb-usability, tbb-3.0, tbb-easy Cc: erinn, hellais, g.koppen@…, Sebastian, ioerror, gnorcie@…
Actual Points: Parent ID:
Points:

Description

Arturo and I have been discussing the creation of a normal looking Mac OS X .dmg for OS X users - this would include the big pretty background that users expect. We believe that this will normalize the OS X install process to be more like all other OS X software.

Helix expressed some concern about size and said that we have a 25MB limit with gmail. It seems that we are already over this limit with .zip as well as .dmg.

Helix further expressed concern about having to upload two sets of files. I believe that this is resolved by only uploading .dmg files rather than .zip files.

If we need to send TBB for OS X with gettor - we will need to reduce the size in either case. I do not understand how we currently send .zip files of TBB for OS X with the size limit.

Many users have expressed serious issues with the .zip - it is very confusing for them. The user complaint is that it is against all their OS X user experience and training.

In any case, I think that gettor could easily be modified to zip up a .dmg and it would prevent the need to upload two sets of packages. In no case does Helix need to upload two sets of packages. However, I do not see how the current set of packages actually works with Gettor if we are already over the 25MB limit.

If the size restriction only comes from gettor/gmail - I think we should weigh the total number of gettor downloads (~400-500 a day for all platforms) against the the total number of HTTP/HTTPS (~10000s a day for all platforms) downloads.

Helix - does that about sum things up?

Arturo - does that sound about correct?

What can we do to move this forward and to offer .dmgs? A patch to make a pretty .dmg? A way to zip up a .dmg for gettor? Anything?

Child Tickets

Attachments (3)

tor-browser-bundle-OSX-dmg.diff (4.4 KB) - added by hellais 3 years ago.
Patch for making TBB on OS X a .dmg
update_dsstore.applescript (4.8 KB) - added by ioerror 3 years ago.
chrome apple script
osx-dmg.patch (42.2 KB) - added by hellais 3 years ago.
DMG writing for Tor Browser Bundle

Download all attachments as: .zip

Change History (39)

comment:1 Changed 3 years ago by hellais

So I tinkered a little bit with different compression algorithms and different hdiutil commands and in ended up with these conclusions.

The Tor Browser Bundle compressed to .dmg with the following command:

hdiutil convert TorBrowserBundle.dmg -format UDZO -imagekey zlib-level=9 -o TorBrowserBundle-compressed.dmg

has a total file size of 3.2364 KB.

This is compressed using zlib level of 9, so optimized memory consumption minimized speed.

If I compress the original bundle using bzip2 I actually end up with a smaller file than the current .zip: 2.9976 KB.

comment:2 Changed 3 years ago by ioerror

So other than file size, what are the constraints? It seems like file size is taken care of in all cases here?

comment:3 Changed 3 years ago by mikeperry

  • Cc erinn hellais added; helix hellais removed
  • Component changed from Tor Browser to Tor bundles/installation
  • Owner changed from mikeperry to erinn

Trac fail. Erinn didn't see the mails for this bug. Wrong owner/component and wrong CC (her trac username is erinn).

comment:4 Changed 3 years ago by mikeperry

Also, can you paste the bzip2 hdiutil command? It looks like it is UBDZ instead of UBZO, and maybe another switch or two for max compression level?

comment:5 Changed 3 years ago by ioerror

We were chatting on IRC Mike - so Erinn has seen the bugs.

Changed 3 years ago by hellais

Patch for making TBB on OS X a .dmg

comment:6 Changed 3 years ago by hellais

I have made a patch for the Makefile to build the TBB for OSX. I haven't tested it in the whole process, but I think it is correct.

You need to place TorBundleTemplate.dmg.bz2 in the scripts folder to make it work.

comment:7 Changed 3 years ago by hellais

I am unable to shrink the template file down to less than 2MB. So I can't upload it here. If you tell me where I should place it I will upload it there (it is around 30MB).

The template file is what is required to have the background image and the correct icon positioning etc.

comment:8 Changed 3 years ago by ioerror

Here's the screenshot for the DMG:
http://postimage.org/image/1xqc4q8kk/

comment:9 Changed 3 years ago by kaner

About the 25MB limit on GMail: I can only assume that its sort of a soft-limit and the real hard limit is somewhat above 25MB. Otherwise the current TBB package for Windows wouldn't make it through.

comment:10 Changed 3 years ago by mikeperry

I really like the dmg screenshot. I have also run into OS X users confused by the .zip file, fwiw. We should definitely be shipping a .dmg, esp if we can make it bzip2 and smaller than a .zip.

comment:11 Changed 3 years ago by ioerror

Well all around, I think this is an improvement; helix, do you need anything else?

comment:12 Changed 3 years ago by erinn

hellais, great work and thanks a lot for this patch! I am going to make some changes to it and the template instead of accepting as-is, which I will post here once I am done. (It is probably just going to be things like separating the whitespace patch and tweaking the workflow to fit with how TBB is built.) Do you have some web space where you can upload the template? If so and you do not want to share the link here publicly, can you send it to me privately?

comment:13 Changed 3 years ago by ioerror

Changed 3 years ago by ioerror

chrome apple script

comment:14 Changed 3 years ago by ioerror

Also check out the file (update_dsstore.applescript) that I just uploaded.

comment:15 Changed 3 years ago by hellais

Sorry for the delay in uploading and updating with the .dmg template, but the connectivity that I had in the past few days didn't allow me to upload the file :(.

Anyways, now you can find it here: http://server.globaleaks.org/TorBundleTemplate.dmg.bz2

sha512sum TorBundleTemplate.dmg.bz2
2351fcf18838e3b9da72c290d8962aa4c60ae89f1e40a2b4d7510902f372a5b39d4f7759f00c5377214b34f0661654445aae60e5897a57a9b88ef54ba9afeac1 TorBundleTemplate.dmg.bz2

shasum TorBundleTemplate.dmg.bz2
1303ee0de238101793c255b55cc8a4fb7e70e7f6

and to make also rransom happy:

md5sum TorBundleTemplate.dmg.bz2
d95e71f2cce824d61270885aa6648d22 TorBundleTemplate.dmg.bz2

:D

comment:16 Changed 3 years ago by erinn

hellais,

Could you please look at the way Chromium and Vidalia do it (https://gitweb.torproject.org/vidalia.git/tree/HEAD:/pkg/osx) and try to use a more programmatic method? I don't think the template solution works because in order to make it replicable for other people, the template would have to be in git, which is a definite no-go. Also how did you generate the .DS_Store file? Am I correct that the template mainly just has a .stuff/tor-browser-bundle.png and a .DS_Store? If so we ought to be generating the .dmg from the .app, rather than using the template solution.

comment:17 Changed 3 years ago by hellais

I have made a patch for making this work programmatically.
Creating DS_Store files is a PITA. What mozilla does is even more dirty than the solution that we are going to use. The basically take a .DS_Store file and modify it to their needs.

What we will do is instead use AppleScript to set the parameters of the .dmg image and have the OSX handle the DS_Store writing.

On a high level what it does is create a .dmg image with background image, icons and proper sizing.

The only doubt that we had with erinn is that this will have a slightly different behavior than what users are usually used to. Since the .dmg is mounted as read-only they will not be able to run TBB directly from the .dmg container.

We have solved this by writing in the background image of the .dmg the text "Copy to you applications folder or to an external drive".

This is how the mounted image looks like:
http://hellais.github.com/drop/Tor%20Browser%20Bundle%20OSX-screenshot.png

Changed 3 years ago by hellais

DMG writing for Tor Browser Bundle

comment:18 Changed 3 years ago by erinn

Oh this is very close! Thanks a lot. Here are some immediate (and relatively minor) suggestions:

  1. Can you make it so that it follows the naming convention for the Mac OS X browser bundles? These are determined in the makefile (torbrowser.git/build-scripts/osx.mk in the compressed-bundle-localized target, such as: zip -r $(DISTDIR)/$(DEFAULT_COMPRESSED_BASENAME)$(LANGCODE).zip $(NAME)_$(LANGCODE).app)

You can probably reconstruct the basic name format by looking at the standard bundle name right now (TorBrowser-2.2.35-4-dev-osx-i386-ar.zip ), or we can change it. The information that must be contained, as per the naming conventions mentioned in torbrowser.git/docs/HACKING, is the version, arch, and language. I think something like TorBrowser-2.2.35-4-osx-i386-ar.dmg should be okay. It occurs to me that I don't mention the langcode in the HACKING doc, so I'll have to update it...

Where is the applescript we talked about?

Regarding the image, I think it's a good start. Would you mind me emailing Jeremy to see if he has any suggestions?

Thanks again, we're really close now!

comment:19 Changed 3 years ago by gk

  • Cc g.koppen@… added

comment:20 Changed 3 years ago by mikeperry

  • Cc Sebastian ioerror added
  • Owner changed from erinn to hellias
  • Status changed from new to assigned

Every time I download the .zip on my Mac, I'm a little saddened this isn't merged yet. Is this just blocking because no one is aware who is supposed to make the tweaks? I'm going to assign this to hellias and put it in needs_revision in the hopes that that is all it takes :).

comment:21 Changed 3 years ago by mikeperry

  • Status changed from assigned to needs_revision

comment:22 follow-ups: Changed 3 years ago by Sebastian

No, this isn't merged because it doesn't make sense. .dmg files are for applications which get dropped into /Applications, which TBB should never be on any system with more than one user. People who separate an admin account from their normal account (like sane people should) are always surprised when their TBB doesn't work right for them if they put it int /Applications, or C:\Program Files on Windows, etc.

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary. I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

comment:23 in reply to: ↑ 22 ; follow-up: Changed 3 years ago by rransom

Replying to Sebastian:

I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

‘trams’ reports that TBB is not confined to one directory on MacOS. I don't see any hope of fixing the ooze on Mac, so it probably is time to give up there.

comment:24 in reply to: ↑ 22 Changed 3 years ago by rransom

Replying to Sebastian:

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary.

Can downloading another zip archive cause Safari to unpack other random files into the TBB directory?

comment:25 in reply to: ↑ 23 Changed 3 years ago by arma

Replying to rransom:

‘trams’ reports that TBB is not confined to one directory on MacOS. I don't see any hope of fixing the ooze on Mac, so it probably is time to give up there.

I think it's way too early to conclude that we should give up there, especially since trams is just starting to get results. See also #5791.

comment:26 in reply to: ↑ 22 Changed 3 years ago by arma

Replying to Sebastian:

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary. I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

I am unable to think like either an osx user or windows user wrt installers. So while I guess people look to me as good at simulating users in general, in this case I'm not useful.

Do we have among us the right people with the right skillsets to both figure out what the packages should be doing on each platform, and change them so they do it? Or should we be trying to find those people?

comment:27 Changed 3 years ago by mikeperry

Pretty much every other OSX app in the world has a .dmg that has the .app folder and a shortcut to Applications folder, and an arrow or some other instructions that indicate you need to drag one to the other.

So how about instead of having an Applications folder shortcut, we just have a shortcut to the Desktop folder? Wouldn't that solve the problem? Or does this drag operation itself update all sorts of plists, regardless of the destination folder?

I know for me, I hate the zip file because it naturally downloads to Downloads, and then I click on it (if you download TBB from TBB, it does *not* auto-unzip), and then the TBB app is in the wrong place (the Downloads folder still). Worse, when I download more than one update, the app gets renamed to things like "Tor Browser-en_US 2" by default after clicking on the .zip. This is a nightmare, because a non-zero number of novice users will just continue to run the old, exploitable Tor Browser-en_US from Downloads, if they even manage find the damn thing in Downloads in the first place.

comment:28 in reply to: ↑ 22 ; follow-up: Changed 3 years ago by hellais

Replying to Sebastian:

No, this isn't merged because it doesn't make sense. .dmg files are for applications which get dropped into /Applications, which TBB should never be on any system with more than one user. People who separate an admin account from their normal account (like sane people should) are always surprised when their TBB doesn't work right for them if they put it int /Applications, or C:\Program Files on Windows, etc.

If the issue is the fact that the application ends up installed for all the users of the system instead of putting an icon shortcut to /Applications we can put an icon shortcut to ~/Applications. This is done by a few applications on OSX and it's the equivalent of doing an installation of an app only for that user.

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary. I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

What is the difference with running TBB from ~/Applications compared with running it from your ~/Downloaded directory? In the second it just pollutes a directory that is uber polluted already and the user actually looses track of it.

It's also worth noting that the default behavior in Safari for unzipping files automatically makes you end up with the .zip file inside of your trashcan. The user may not be aware of this and could lead him into getting into trouble.

comment:29 Changed 3 years ago by mikeperry

  • Keywords tbb-usability added

FYI: http://petsymposium.org/2012/papers/hotpets12-1-usability.pdf lists "archive confusion" as the 4th most common "stop issue" that prevented people from using TBB. They tested only Windows 7, which uses the self-extracting exe, which I would have actually guessed as easier to figure out than a zip? Even in the best case, it's the same user activity though: Click on exe/zip, lose track of extraction folder, get confused.

comment:30 Changed 3 years ago by arma

  • Cc gnorcie@… added

comment:31 in reply to: ↑ 28 Changed 3 years ago by greg

Replying to hellais:

Replying to Sebastian:

No, this isn't merged because it doesn't make sense. .dmg files are for applications which get dropped into /Applications, which TBB should never be on any system with more than one user. People who separate an admin account from their normal account (like sane people should) are always surprised when their TBB doesn't work right for them if they put it int /Applications, or C:\Program Files on Windows, etc.

If the issue is the fact that the application ends up installed for all the users of the system instead of putting an icon shortcut to /Applications we can put an icon shortcut to ~/Applications. This is done by a few applications on OSX and it's the equivalent of doing an installation of an app only for that user.

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary. I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

What is the difference with running TBB from ~/Applications compared with running it from your ~/Downloaded directory? In the second it just pollutes a directory that is uber polluted already and the user actually looses track of it.

It's also worth noting that the default behavior in Safari for unzipping files automatically makes you end up with the .zip file inside of your trashcan. The user may not be aware of this and could lead him into getting into trouble.

A typical user expects TBB to act like an application. As long as they can navigate to their "Applications" folder, and click on something that launches the TBB, they will be happy.

I understand there are technical constraints here, but users have a preconceived notion about how software installation occurs. They may not be familiar with concepts like a user directory vs a system directory.

If having TBB installed systemwide is an issue, by all means, avoid it at all costs.

But we should have the installation process be as similar to any other OSX application.

comment:32 Changed 2 years ago by runa

What's the status here? Is it blocking on something?

comment:33 Changed 21 months ago by mikeperry

  • Keywords tbb-3.0 added
  • Priority changed from normal to major

I want this for TBB 3.0, but we now build MacOS binaries on Linux (using Gitian), so this patch won't work there. We need to do this instead:
http://stackoverflow.com/questions/286419/how-to-build-a-dmg-mac-os-x-file-on-a-non-mac-platform#3776463

That post doesn't mention how to make it use bzip2, though (which we want).

comment:34 Changed 13 months ago by mikeperry

  • Keywords tbb-easy added

It turns out that stackoverflow link does not successfully create UDIF/UDBZ compressed dmg images (they produce CRC errors when used on MacOS). However, it looks like Mozilla succeeded at this by forking libdmg-hfsplus to add UDBZ support. See https://bugzilla.mozilla.org/show_bug.cgi?id=935237 and https://github.com/vasi/libdmg-hfsplus/tree/bzip2. A LZ compressed DMG (UDZO format) can also be produced with https://github.com/erwint/libdmg-hfsplus, but it will likely be larger than UDBZ.

If their UDBZ fork actually works as they claim, this should be relatively easy, though perhaps a little time consuming to build, test, and get the shortcuts to work properly to provide an easy link to the Desktop and/or the Applications folder for maximum usability.

comment:35 Changed 12 months ago by mikeperry

The following seems to more or less work:

# First: Extract TBB + localization files in ~/install/tbb/
# Then do:
mkdir -p ~/install/tbb/.background/
cp ~/build/tor-browser/browser/branding/official/background.png ~/install/tbb/.background/
cp ~/build/tor-browser/browser/branding/official/disk.icns ~/install/tbb/.VolumeIcon.icns
cp ~/build/tor-browser/browser/branding/official/dsstore ~/install/tbb/.DS_Store
ln -s /Applications "~/install/tbb/ "
genisoimage -D -V "TorBrowserBundle_en-US" -no-pad -R -apple -o tbb-uncompressed.dmg ~/install/tbb
./dmg/dmg dmg tbb-uncompressed.dmg tbb-compressed.dmg

However, the background and the icon positioning is not properly picked up, possibly because the provided dsstore file still references Firefox instead of TorBrowserBundle_en-US. A simple sed -i to change those strings did not appear to work. I tried asking in https://bugzilla.mozilla.org/show_bug.cgi?id=935237#c32. We'll see what Mozilla says about how we can generate/edit dsstore.

Last edited 12 months ago by mikeperry (previous) (diff)

comment:36 Changed 11 months ago by mikeperry

  • Resolution set to fixed
  • Status changed from needs_revision to closed

This was fixed in 3.6-beta-1. It's also reproducible, with an additional patch to libdmghfs-plus.

Note: See TracTickets for help on using tickets.