Make Tor Browser on macOS compatible with Apple's notarization

Notarization is a technique by Apple to make apps on macOS more secure to run. There a numerous parts to this and one can find more details about that on:

https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution

Mozilla is tracking the work in:

https://bugzilla.mozilla.org/show_bug.cgi?id=1470607

and there are a bunch of large pieces that still need to get solved on their side, like enabling the Hardened Runtime and building with the 10.14 SDK.

However, at some point in the future apps won't run without that anymore and the potential changes we need to made are probably considerable. Thus, we should keep an eye on that and start thinking about which pieces of our signing infrastructure need to get adapted. Questions could be:

1. Is it still enough to sign the builds on a 10.9 machine?
2. How do we integrate sending the apps to Apple to get their blessing into our release process?
3. How does that system work with our plan to get rid of the Apple signing machine and do the signing on Linux? (see: #29815 (moved))

I don't see this being relevant for ESR 68 but it might become so during the transition to the ESR after that one (or for the regular release train in case we'll start following that one instead).

Trac:
Child Ticket(s): #31465 (moved)

added TorBrowserTeam201909 actualpoints::5.5 component::applications/tor browser owner::tbb-team points::2 priority::very high resolution::fixed severity::normal status::closed tbb-security type::task labels

Trac:
Description: Notarization is a technique by Apple to make running apps on macOS more secure to run. There a numerous parts to this and one can find more details about that on:

https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution

Mozilla is tracking the work in:

https://bugzilla.mozilla.org/show_bug.cgi?id=1470607

and there are a bunch of large pieces that still need to get solved on their side, like enabling the Hardened Runtime and building with the 10.14 SDK.

However, at some point in the future apps won't run without that anymore and the potential changes we need to made are probably considerable. Thus, we should keep an eye on that and start thinking about which pieces of our signing infrastructure need to get adapted. Questions could be:

1. Is it still enough to sign the builds on a 10.9 machine?
2. How do we integrate sending the apps to Apple to get their blessing into our release process?
3. How does that system work with our plan to get rid of the Apple signing machine and do the signing on Linux? (see: #29815 (moved))

I don't see this being relevant for ESR 68 but it might become so during the transition to the ESR after that one (or for the regular release train in case we'll start following that one instead).

to

Notarization is a technique by Apple to make apps on macOS more secure to run. There a numerous parts to this and one can find more details about that on:

https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution

Mozilla is tracking the work in:

https://bugzilla.mozilla.org/show_bug.cgi?id=1470607

and there are a bunch of large pieces that still need to get solved on their side, like enabling the Hardened Runtime and building with the 10.14 SDK.

However, at some point in the future apps won't run without that anymore and the potential changes we need to made are probably considerable. Thus, we should keep an eye on that and start thinking about which pieces of our signing infrastructure need to get adapted. Questions could be:

1. Is it still enough to sign the builds on a 10.9 machine?
2. How do we integrate sending the apps to Apple to get their blessing into our release process?
3. How does that system work with our plan to get rid of the Apple signing machine and do the signing on Linux? (see: #29815 (moved))

I don't see this being relevant for ESR 68 but it might become so during the transition to the ESR after that one (or for the regular release train in case we'll start following that one instead).

It seems the future is getting faster to us than we hoped: https://blog.torproject.org/comment/282597#comment-282597 :(

We might not have all the Firefox code pieces ready to have this properly working but we should look pretty soon into making our signing infrastructure ready.

Trac:
Keywords: N/A deleted, TorBrowserTeam201906 added
Cc: mcs, brade to mcs, brade, boklm
Priority: Medium to Very High

FWIW: It seems the updater is one of the first victims here: https://blog.torproject.org/comment/282619#comment-282619 and https://bugzilla.mozilla.org/show_bug.cgi?id=1556733.

Replying to gk:

FWIW: It seems the updater is one of the first victims here: https://blog.torproject.org/comment/282619#comment-282619 and https://bugzilla.mozilla.org/show_bug.cgi?id=1556733.

That particular issue got fixed on Mozilla's side and backported to all relevant branches including esr60. Thus, we are good for that issue at least.

Moving tickets to July

Trac:
Keywords: TorBrowserTeam201906 deleted, TorBrowserTeam201907 added

https://bugzilla.mozilla.org/show_bug.cgi?id=1563631#c5 seems relevant here:

According to the WWDC presentation[1] around 8:00, all software "signed on built after June 1st" must be Notarized.

The presentation itself is at https://developer.apple.com/videos/play/wwdc2019/701/.

Kathy and I need to do more research, but here are some things we learned so far.

Additional resources:

• https://stackoverflow.com/a/53121755/2517441 (assuming this answer is accurate, it provides detailed steps we will need to execute).
• https://blog.zeplin.io/dev-journal-automate-notarizing-macos-apps-94b0b144ba9d (provides a good overview of a command line approach to notarization).
• https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution/customizing_the_notarization_workflow

Some of the requirements, as specified by Apple's documentation:

• Link against the macOS 10.9 or later SDK (already done for Tor Browser).
• Notarization requires Xcode 10 or later (maybe simply because we need an xcrun that supports the altool, and that first appeared in Xcode 10.0).
• Building a new app for notarization requires macOS 10.13.6 or later & Xcode 10 (macOS 10.13.6 is required for Xcode 10.0).
• Stapling an app requires macOS 10.12 or later (but I guess we will have macOS 10.13.16 or newer anyway).
• Enable code-signing for all of the executables you distribute (hopefully we already do this).
• Use a Developer ID application, kernel extension, or installer certificate for your code-signing signature (a Mac Distribution or local development certificate will not work).
• Include a secure timestamp with your code-signing signature (which means we need to include the --timestamp option when running the codesign tool).
• Enable the Hardened Runtime capability for your app (how do we handle entitlements?)
• Don't include the com.apple.security.get-task-allow entitlement with the value set to any variation of true (again, how do we add entitlements during our build process — if at all?)

The following Firefox bug includes at least one patch related to entitlements, although the patches are for taskcluster and not core Firefox code: https://bugzilla.mozilla.org/show_bug.cgi?id=1471004

It was suggested that we look at how Bitcoin Core is handling notarization, but all we found so far is this open issue: https://github.com/bitcoin/bitcoin/issues/15774

If you have questions; :haik spent a considerable time investigating and deploying this for Mozilla.

Replying to tom:

If you have questions; :haik spent a considerable time investigating and deploying this for Mozilla.

Good idea! mcs/brade could you follow up on that with a particular focus on our signing requirements, so we get the remaining questions we have answered and have a clearer picture on our constraints?

Oh, I stumbled over https://searchfox.org/mozilla-esr68/source/security/mac/hardenedruntime/codesign.bash by chance which seems to have additional useful information.

Today, Kathy and I downloaded https://dist.torproject.org/torbrowser/9.0a4/TorBrowser-9.0a4-osx64_en-US.dmg, extracted Tor Browser.app, and experimented with Apple's notarization process. Using our own "Developer ID Application" signing key, we re-signed the app bundle like this: CERT="Developer ID Application: ..." ENTITLEMENTS=/path/to/production.entitlements.xml codesign -vvv --deep -o runtime --entitlements "

ENTITLEMENTS" --timestamp -f -s "

CERT" "Tor Browser.app/"

(compared to the usual Tor Browser Gatekeeper signing we added -o runtime to enable the hardened runtime, we added the Firefox 68 entitlements file, and we enabled timestamping).

Then we used Xcode 10.1's Application Loader tool to upload a .zip to Apple for notarization, like this: rm -f tb.zip zip -qr tb.zip "Tor Browser.app" export PW="secret" BUNDLEID="org.torproject.torbrowser" xcrun altool --notarize-app -t osx -f tb.zip --primary-bundle-id "$BUNDLEID" -u ID -p @env:PW --output-format xml

Finally, we used altool to poll for the status of Apple's server-side notarization process, e.g., xcrun altool --notarization-info REQUESTID -u ID -p @env:PW --output-format xml (REQUESTID is a GUID returned by the altool --notarize-app command).

The result was a "Package Invalid" error. Apple also provides an URL that points to a detailed log, and that shows a series of repeated issues (one for each of our binaries), e.g., "severity": "error", "code": null, "path": "tb.zip/Tor Browser.app/Contents/MacOS/firefox", "message": "The binary uses an SDK older than the 10.9 SDK.", "docUrl": null, "architecture": "x86_64" ...

That is strange. We then used otool -l on macOS to look at the firefox binary. The output indeed indicates that the minimum macOS version and SDK version are not what we would expect: cmd LC_VERSION_MIN_MACOSX cmdsize 16 version 10.7 sdk 10.6

Kathy and I do not know enough about the cross-compile process to know where to look for a solution to this problem, but I think we need to fix this SDK issue before we can go further with notarization.

https://bugzilla.mozilla.org/show_bug.cgi?id=1270217

Where sdk is leaked from is still unknown: https://github.com/NixOS/nixpkgs/issues/21629

Replying to mcs:

[snip]

That is strange. We then used otool -l on macOS to look at the firefox binary. The output indeed indicates that the minimum macOS version and SDK version are not what we would expect: cmd LC_VERSION_MIN_MACOSX cmdsize 16 version 10.7 sdk 10.6

Kathy and I do not know enough about the cross-compile process to know where to look for a solution to this problem, but I think we need to fix this SDK issue before we can go further with notarization.

Hm, I think we don't do something differently to what Mozilla is using. So, what is the Firefox ESR 60 output here? And does that change with the patch for bug 1270217 landed in Firefox (that is with Firefox 62)? What does the firefox binary for ESR 68 say in that regard?

Meanwhile I've started a test build with the patch for bug 1270217 applied, in case that buys us something. I'll post a link to a bundle later when the build has finished.

How about:

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126-osx64_en-US.dmg https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126-osx64_en-US.dmg.asc

?

Replying to gk:

Hm, I think we don't do something differently to what Mozilla is using. So, what is the Firefox ESR 60 output here? And does that change with the patch for bug 1270217 landed in Firefox (that is with Firefox 62)? What does the firefox binary for ESR 68 say in that regard?

Here is the LC_VERSION_MIN_MACOSX info for various builds:

Build	min OS version	SDK version
Firefox 60.8.0 ESR	10.7	10.11
Firefox 68.0.1 ESR	10.9	10.11
Firefox 62.0	10.9	10.11
Tor Browser 9.0a4	10.7	10.6
2019-07-28 Patched Tor Browser build	10.9	10.6

Meanwhile I've started a test build with the patch for bug 1270217 applied, in case that buys us something. I'll post a link to a bundle later when the build has finished.

See the last row in the table above. The patch affects the minimum supported OS version but not the SDK version.

Kathy and I looked at the cctools code, and maybe the problem is that the SDK version is not defaulting to the correct value. There is code to pick it up from the SDK path, but our SDK path is just SDK/. Search for if -sdk_version not on command line, infer from -syslibroot within https://github.com/tpoechtrager/cctools-port/blob/8e9c3f2506b51cf56725eaa60b6e90e240e249ca/cctools/ld64/src/ld/Options.cpp to see the relevant code.

One solution is to leave our SDK directory name as MacOSX10.11.sdk. An alternative is to add -sdk_version 10.11 to the ld command.

By the way, we could not find an open source tool that dumps mach-o header fields like the macOS otool and objdump commands can.

Replying to mcs:

[snip]

Kathy and I looked at the cctools code, and maybe the problem is that the SDK version is not defaulting to the correct value. There is code to pick it up from the SDK path, but our SDK path is just SDK/. Search for if -sdk_version not on command line, infer from -syslibroot within https://github.com/tpoechtrager/cctools-port/blob/8e9c3f2506b51cf56725eaa60b6e90e240e249ca/cctools/ld64/src/ld/Options.cpp to see the relevant code.

One solution is to leave our SDK directory name as MacOSX10.11.sdk. An alternative is to add -sdk_version 10.11 to the ld command.

Thanks for the investigation! I think I have a fix for that which follows Mozilla leaving the SDK directory name as MacOSX10.11.sdk:

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_2-osx64_en-US.dmg https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_2-osx64_en-US.dmg.asc

Is Apple happier with that one? (Note: that's without the patch for bug 1270217 which we might need as well) If we are good I'll open a child bug just for the SDK issue and get that one fixed there.

By the way, we could not find an open source tool that dumps mach-o header fields like the macOS otool and objdump commands can.

That would be unfortunate, so I looked a bit around. It turns out that you are already building such a tool while building the macOS bundles :) : x86_64-apple-darwin11-otool (and a bunch of similar tools) gets built when assembling the macosx-toolchain and works for me for the purposes at hand (you need to put the path to clang/lib into LD_LIBRARY_PATH to find libc++abi.so.1).

Replying to gk:

Is Apple happier with that one? Apple is happier with https://bugzilla.mozilla.org/show_bug.cgi?id=1471004#c41 (Note: that's without the patch for bug 1270217 which we might need as well) It's preferred, but only SDK is required (tested by other apps). x86_64-apple-darwin11-otool (and a bunch of similar tools) gets built when assembling the macosx-toolchain and works for me for the purposes at hand So, what sdk did it show? :)