Opened 6 months ago

Closed 5 weeks ago

Last modified 5 weeks ago

#30126 closed task (fixed)

Make Tor Browser on macOS compatible with Apple's notarization

Reported by: gk Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-security, TorBrowserTeam201909
Cc: mcs, brade, boklm, dcf Actual Points: 5.5
Parent ID: Points: 2
Reviewer: Sponsor:

Description (last modified by gk)

Notarization is a technique by Apple to make apps on macOS more secure to run. There a numerous parts to this and one can find more details about that on:

https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution

Mozilla is tracking the work in:

https://bugzilla.mozilla.org/show_bug.cgi?id=1470607

and there are a bunch of large pieces that still need to get solved on their side, like enabling the Hardened Runtime and building with the 10.14 SDK.

However, at some point in the future apps won't run without that anymore and the potential changes we need to made are probably considerable. Thus, we should keep an eye on that and start thinking about which pieces of our signing infrastructure need to get adapted. Questions could be:

1) Is it still enough to sign the builds on a 10.9 machine?
2) How do we integrate sending the apps to Apple to get their blessing into our release process?
3) How does that system work with our plan to get rid of the Apple signing machine and do the signing on Linux? (see: #29815)

I don't see this being relevant for ESR 68 but it might become so during the transition to the ESR after that one (or for the regular release train in case we'll start following that one instead).

Child Tickets

TicketStatusOwnerSummaryComponent
#31465closedtbb-teamAdapt tor-browser-build projects for macOS notarizationApplications/Tor Browser

Attachments (1)

GatekeeperError.png (95.8 KB) - added by mcs 5 weeks ago.
macOS 10.15 Gatekeeper error alert

Download all attachments as: .zip

Change History (61)

comment:1 Changed 6 months ago by gk

Description: modified (diff)

comment:2 Changed 4 months ago by gk

Cc: boklm added
Keywords: TorBrowserTeam201906 added
Priority: MediumVery High

It seems the future is getting faster to us than we hoped: https://blog.torproject.org/comment/282597#comment-282597 :(

We might not have all the Firefox code pieces ready to have this properly working but we should look pretty soon into making our signing infrastructure ready.

comment:3 Changed 4 months ago by gk

comment:4 in reply to:  3 Changed 4 months ago by gk

Replying to gk:

FWIW: It seems the updater is one of the first victims here: https://blog.torproject.org/comment/282619#comment-282619 and https://bugzilla.mozilla.org/show_bug.cgi?id=1556733.

That particular issue got fixed on Mozilla's side and backported to all relevant branches including esr60. Thus, we are good for that issue at least.

comment:5 Changed 3 months ago by gk

Keywords: TorBrowserTeam201907 added; TorBrowserTeam201906 removed

Moving tickets to July

comment:6 Changed 3 months ago by gk

https://bugzilla.mozilla.org/show_bug.cgi?id=1563631#c5 seems relevant here:

According to the WWDC presentation[1] around 8:00, all software "signed on built after June 1st" must be Notarized.

The presentation itself is at https://developer.apple.com/videos/play/wwdc2019/701/.

comment:7 Changed 3 months ago by mcs

Kathy and I need to do more research, but here are some things we learned so far.

Additional resources:

Some of the requirements, as specified by Apple's documentation:

  • Link against the macOS 10.9 or later SDK (already done for Tor Browser).
  • Notarization requires Xcode 10 or later (maybe simply because we need an xcrun that supports the altool, and that first appeared in Xcode 10.0).
  • Building a new app for notarization requires macOS 10.13.6 or later & Xcode 10 (macOS 10.13.6 is required for Xcode 10.0).
  • Stapling an app requires macOS 10.12 or later (but I guess we will have macOS 10.13.16 or newer anyway).
  • Enable code-signing for all of the executables you distribute (hopefully we already do this).
  • Use a Developer ID application, kernel extension, or installer certificate for your code-signing signature (a Mac Distribution or local development certificate will not work).
  • Include a secure timestamp with your code-signing signature (which means we need to include the --timestamp option when running the codesign tool).
  • Enable the Hardened Runtime capability for your app (how do we handle entitlements?)
  • Don't include the com.apple.security.get-task-allow entitlement with the value set to any variation of true (again, how do we add entitlements during our build process — if at all?)

The following Firefox bug includes at least one patch related to entitlements, although the patches are for taskcluster and not core Firefox code: https://bugzilla.mozilla.org/show_bug.cgi?id=1471004

It was suggested that we look at how Bitcoin Core is handling notarization, but all we found so far is this open issue: https://github.com/bitcoin/bitcoin/issues/15774

comment:8 Changed 3 months ago by tom

If you have questions; :haik spent a considerable time investigating and deploying this for Mozilla.

comment:9 in reply to:  8 Changed 3 months ago by gk

Replying to tom:

If you have questions; :haik spent a considerable time investigating and deploying this for Mozilla.

Good idea! mcs/brade could you follow up on that with a particular focus on our signing requirements, so we get the remaining questions we have answered and have a clearer picture on our constraints?

Last edited 3 months ago by gk (previous) (diff)

comment:10 Changed 3 months ago by gk

Oh, I stumbled over https://searchfox.org/mozilla-esr68/source/security/mac/hardenedruntime/codesign.bash by chance which seems to have additional useful information.

comment:11 Changed 3 months ago by mcs

Today, Kathy and I downloaded https://dist.torproject.org/torbrowser/9.0a4/TorBrowser-9.0a4-osx64_en-US.dmg, extracted Tor Browser.app, and experimented with Apple's notarization process. Using our own "Developer ID Application" signing key, we re-signed the app bundle like this:

CERT="Developer ID Application: ..."
ENTITLEMENTS=/path/to/production.entitlements.xml
codesign -vvv --deep -o runtime --entitlements "$ENTITLEMENTS" --timestamp -f -s "$CERT" "Tor Browser.app/"

(compared to the usual Tor Browser Gatekeeper signing we added -o runtime to enable the hardened runtime, we added the Firefox 68 entitlements file, and we enabled timestamping).

Then we used Xcode 10.1's Application Loader tool to upload a .zip to Apple for notarization, like this:

rm -f tb.zip
zip -qr tb.zip "Tor Browser.app"
export PW="secret"
BUNDLEID="org.torproject.torbrowser"
xcrun altool --notarize-app -t osx -f tb.zip --primary-bundle-id "$BUNDLEID" -u ID -p @env:PW --output-format xml

Finally, we used altool to poll for the status of Apple's server-side notarization process, e.g.,

xcrun altool --notarization-info REQUESTID -u ID -p @env:PW --output-format xml

(REQUESTID is a GUID returned by the altool --notarize-app command).

The result was a "Package Invalid" error. Apple also provides an URL that points to a detailed log, and that shows a series of repeated issues (one for each of our binaries), e.g.,

"severity": "error",
"code": null,
"path": "tb.zip/Tor Browser.app/Contents/MacOS/firefox",
"message": "The binary uses an SDK older than the 10.9 SDK.",
"docUrl": null,
"architecture": "x86_64"
...

That is strange. We then used otool -l on macOS to look at the firefox binary. The output indeed indicates that the minimum macOS version and SDK version are not what we would expect:

cmd LC_VERSION_MIN_MACOSX
cmdsize 16
version 10.7
sdk 10.6

Kathy and I do not know enough about the cross-compile process to know where to look for a solution to this problem, but I think we need to fix this SDK issue before we can go further with notarization.

comment:13 Changed 3 months ago by cypherpunks

Where sdk is leaked from is still unknown: https://github.com/NixOS/nixpkgs/issues/21629

comment:14 in reply to:  11 ; Changed 3 months ago by gk

Replying to mcs:

[snip]

That is strange. We then used otool -l on macOS to look at the firefox binary. The output indeed indicates that the minimum macOS version and SDK version are not what we would expect:

cmd LC_VERSION_MIN_MACOSX
cmdsize 16
version 10.7
sdk 10.6

Kathy and I do not know enough about the cross-compile process to know where to look for a solution to this problem, but I think we need to fix this SDK issue before we can go further with notarization.

Hm, I think we don't do something differently to what Mozilla is using. So, what is the Firefox ESR 60 output here? And does that change with the patch for bug 1270217 landed in Firefox (that is with Firefox 62)? What does the firefox binary for ESR 68 say in that regard?

Meanwhile I've started a test build with the patch for bug 1270217 applied, in case that buys us something. I'll post a link to a bundle later when the build has finished.

Last edited 3 months ago by gk (previous) (diff)

comment:16 in reply to:  14 ; Changed 3 months ago by mcs

Replying to gk:

Hm, I think we don't do something differently to what Mozilla is using. So, what is the Firefox ESR 60 output here? And does that change with the patch for bug 1270217 landed in Firefox (that is with Firefox 62)? What does the firefox binary for ESR 68 say in that regard?

Here is the LC_VERSION_MIN_MACOSX info for various builds:

Buildmin OS versionSDK version
Firefox 60.8.0 ESR10.710.11
Firefox 68.0.1 ESR10.910.11
Firefox 62.010.910.11
Tor Browser 9.0a410.710.6
2019-07-28 Patched Tor Browser build10.910.6

Meanwhile I've started a test build with the patch for bug 1270217 applied, in case that buys us something. I'll post a link to a bundle later when the build has finished.

See the last row in the table above. The patch affects the minimum supported OS version but not the SDK version.

Kathy and I looked at the cctools code, and maybe the problem is that the SDK version is not defaulting to the correct value. There is code to pick it up from the SDK path, but our SDK path is just SDK/. Search for if -sdk_version not on command line, infer from -syslibroot within https://github.com/tpoechtrager/cctools-port/blob/8e9c3f2506b51cf56725eaa60b6e90e240e249ca/cctools/ld64/src/ld/Options.cpp to see the relevant code.

One solution is to leave our SDK directory name as MacOSX10.11.sdk. An alternative is to add -sdk_version 10.11 to the ld command.

By the way, we could not find an open source tool that dumps mach-o header fields like the macOS otool and objdump commands can.

comment:17 in reply to:  16 ; Changed 3 months ago by gk

Replying to mcs:

[snip]

Kathy and I looked at the cctools code, and maybe the problem is that the SDK version is not defaulting to the correct value. There is code to pick it up from the SDK path, but our SDK path is just SDK/. Search for if -sdk_version not on command line, infer from -syslibroot within https://github.com/tpoechtrager/cctools-port/blob/8e9c3f2506b51cf56725eaa60b6e90e240e249ca/cctools/ld64/src/ld/Options.cpp to see the relevant code.

One solution is to leave our SDK directory name as MacOSX10.11.sdk. An alternative is to add -sdk_version 10.11 to the ld command.

Thanks for the investigation! I think I have a fix for that which follows Mozilla leaving the SDK directory name as MacOSX10.11.sdk:

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_2-osx64_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_2-osx64_en-US.dmg.asc

Is Apple happier with that one? (Note: that's without the patch for bug 1270217 which we might need as well) If we are good I'll open a child bug just for the SDK issue and get that one fixed there.

By the way, we could not find an open source tool that dumps mach-o header fields like the macOS otool and objdump commands can.

That would be unfortunate, so I looked a bit around. It turns out that you are already building such a tool while building the macOS bundles :) : x86_64-apple-darwin11-otool (and a bunch of similar tools) gets built when assembling the macosx-toolchain and works for me for the purposes at hand (you need to put the path to clang/lib into LD_LIBRARY_PATH to find libc++abi.so.1).

comment:18 in reply to:  17 ; Changed 3 months ago by cypherpunks

Replying to gk:

Is Apple happier with that one?

Apple is happier with https://bugzilla.mozilla.org/show_bug.cgi?id=1471004#c41

(Note: that's without the patch for bug 1270217 which we might need as well)

It's preferred, but only SDK is required (tested by other apps).

x86_64-apple-darwin11-otool (and a bunch of similar tools) gets built when assembling the macosx-toolchain and works for me for the purposes at hand

So, what sdk did it show? :)

comment:19 in reply to:  18 Changed 3 months ago by gk

Replying to cypherpunks:

Replying to gk:

Is Apple happier with that one?

Apple is happier with https://bugzilla.mozilla.org/show_bug.cgi?id=1471004#c41

(Note: that's without the patch for bug 1270217 which we might need as well)

It's preferred, but only SDK is required (tested by other apps).

Maybe preferred, but I think we should stick to what Mozilla is doing with esr60 and so far it does not look like they are backporting the fix for bug 1270217 to it.

x86_64-apple-darwin11-otool (and a bunch of similar tools) gets built when assembling the macosx-toolchain and works for me for the purposes at hand

So, what sdk did it show? :)

10.11.

comment:20 in reply to:  17 ; Changed 3 months ago by mcs

Replying to gk:

Thanks for the investigation! I think I have a fix for that which follows Mozilla leaving the SDK directory name as MacOSX10.11.sdk:

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_2-osx64_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_2-osx64_en-US.dmg.asc

Is Apple happier with that one? (Note: that's without the patch for bug 1270217 which we might need as well) If we are good I'll open a child bug just for the SDK issue and get that one fixed there.

It is almost perfect. Apple complains about the following three files which have sdk 10.7 in the mach-o header:

Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client-torbrowser
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/obfs4proxy

Is the build process different for those binaries?

After finding those anomolies, Kathy and I did some more checking and found that all of our other binaries have (min) version 10.7 and sdk 10.11 (as expected) with the exception of two files. The following have sdk 10.11 (good) but for some reason have (min) version 10.11 (possibly bad):

Tor Browser.app/Contents/MacOS/Tor/libevent-2.1.6.dylib
Tor Browser.app/Contents/MacOS/Tor/tor.real

That won't break notarization, but I wonder if it will cause problems when trying to run on older macOS systems.

In any case, after Kathy and I removed meek-client, meek-client-torbrowser, and obfs4proxy we followed the steps from comment:11 again and notarization (finally) succeeded. There is one more required step to avoid macOS having to contact Apple to check notarization status every time the app is opened: stapling. This also requires Internet access but at least it did its job quickly:

xcrun stapler staple Tor\ Browser.app

The above command adds one new file to the app bundle (Tor Browser.app/Contents/CodeResources) and makes no other changes. Near the end of the following article there is some info about network access requirements:

https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution/customizing_the_notarization_workflow

Now we can check the app status and note that the source is a "Notarized Developer ID":

spctl -av Tor\ Browser.app
Tor Browser.app: accepted
source=Notarized Developer ID

Finally, the prompt displayed when a user tries to open a downloaded app ("Tor Browser.app is an app downloaded from the Internet. Are you sure you want to open it?") now includes the "Apple checked it for malicious software and none was detected" text as expected for a notarized app.

By the way, we could not find an open source tool that dumps mach-o header fields like the macOS otool and objdump commands can.

That would be unfortunate, so I looked a bit around. It turns out that you are already building such a tool while building the macOS bundles :) : x86_64-apple-darwin11-otool (and a bunch of similar tools) gets built when assembling the macosx-toolchain and works for me for the purposes at hand (you need to put the path to clang/lib into LD_LIBRARY_PATH to find libc++abi.so.1).

Nice! (and good to know).

comment:21 in reply to:  20 ; Changed 3 months ago by gk

Replying to mcs:

[snip]

It is almost perfect. Apple complains about the following three files which have sdk 10.7 in the mach-o header:

Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client-torbrowser
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/obfs4proxy

Is the build process different for those binaries?

Those are pure go builds. Thus, I suspect we need to find some magic way to set the required flags when CGO is not involved. Hrm.

After finding those anomolies, Kathy and I did some more checking and found that all of our other binaries have (min) version 10.7 and sdk 10.11 (as expected) with the exception of two files. The following have sdk 10.11 (good) but for some reason have (min) version 10.11 (possibly bad):

Tor Browser.app/Contents/MacOS/Tor/libevent-2.1.6.dylib
Tor Browser.app/Contents/MacOS/Tor/tor.real

That won't break notarization, but I wonder if it will cause problems when trying to run on older macOS systems.

Nice catch! That can get solved by setting the proper MACOSX_DEPLOYMENT_TARGET version. To answer my IRC question: we were not affected by that previously as we did not get the proper SDK version but fixing that and not setting MACOSX_DEPLOYMENT_TARGET just takes the SDK version as min OS version it seems.

(Oh and I finally realized that my concerns about snowflake not running on 10.9 were a non-issue as we set the corresponding -mmacosx-version-min=10.7 flag.)

comment:22 in reply to:  21 Changed 3 months ago by gk

Replying to gk:

Replying to mcs:

[snip]

It is almost perfect. Apple complains about the following three files which have sdk 10.7 in the mach-o header:

Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client-torbrowser
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/obfs4proxy

Is the build process different for those binaries?

Those are pure go builds. Thus, I suspect we need to find some magic way to set the required flags when CGO is not involved. Hrm.

I suspect that's because we use internal linking mode here which hardcodes 10.7: https://go-review.googlesource.com/c/go/+/18588/

comment:24 in reply to:  23 Changed 3 months ago by gk

Replying to cypherpunks:

https://github.com/golang/go/issues/30488

Yeah, I guess we go here with bumping the go version instead of trying to get external linker mode linking working for obfs4 and meek.

comment:25 Changed 3 months ago by cypherpunks

The security updates for Go 1.12 turn out to be as important as for Firefox. Go > 1.12.5 is required to fix the issue and 1.12.7 is available.

comment:26 Changed 3 months ago by gk

Alright: mcs/brade: could you give

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_3-osx64_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_3-osx64_en-US.dmg.asc

a round of testing? I think I've fixed all the blockers for notarization in our build system now. making sure the whole bundle is still running (even on a system < 10.11) would be neat as well. :)

comment:27 in reply to:  26 ; Changed 3 months ago by mcs

Replying to gk:

Alright: mcs/brade: could you give

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_3-osx64_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_3-osx64_en-US.dmg.asc

a round of testing? I think I've fixed all the blockers for notarization in our build system now. making sure the whole bundle is still running (even on a system < 10.11) would be neat as well. :)

The good news is that the notarization process works with that build.

I am not sure it matters, but the following three files have different min OS version and SDK values in their mach-o headers:

Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client-torbrowser
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/obfs4proxy

While all other binaries have min OS version 10.7 and SDK 10.11, the above three files have 10.9 and 10.9.

We did find one blocker though: when we tested on a macOS 10.9.x system we experienced #26876 again. Since Tor Browser 9.0a4 does not have this problem, there must besome difference in how tor.real is built for your test builds.

comment:28 in reply to:  27 ; Changed 3 months ago by gk

Replying to mcs:

Replying to gk:

Alright: mcs/brade: could you give

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_3-osx64_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_3-osx64_en-US.dmg.asc

a round of testing? I think I've fixed all the blockers for notarization in our build system now. making sure the whole bundle is still running (even on a system < 10.11) would be neat as well. :)

The good news is that the notarization process works with that build.

\o/

I am not sure it matters, but the following three files have different min OS version and SDK values in their mach-o headers:

Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/meek-client-torbrowser
Tor Browser.app/Contents/MacOS/Tor/PluggableTransports/obfs4proxy

While all other binaries have min OS version 10.7 and SDK 10.11, the above three files have 10.9 and 10.9.

That's expected with the switch to Go >= 1.12.6 and is okay I think

We did find one blocker though: when we tested on a macOS 10.9.x system we experienced #26876 again. Since Tor Browser 9.0a4 does not have this problem, there must besome difference in how tor.real is built for your test builds.

If you look at bug_30126_v2 in my tor-browser-build repo you see the changes I am doing. For tor.real in particular I set export MACOSX_DEPLOYMENT_TARGET=10.7. _That_ alone should not cause this issue. A couple of possible options come to mind here: 1) We might need to set it earlier, that is before the configure step. 2) It's somehow caused by us setting the SDK option to 10.11 now.

I suspect you get the same problem with the build 2 in comment:17?

Last edited 3 months ago by gk (previous) (diff)

comment:29 in reply to:  28 Changed 3 months ago by teor

Replying to gk:

Replying to mcs:

We did find one blocker though: when we tested on a macOS 10.9.x system we experienced #26876 again. Since Tor Browser 9.0a4 does not have this problem, there must besome difference in how tor.real is built for your test builds.

If you look at bug_30126_v2 in my tor-browser-build repo you see the changes I am doing. For tor.real in particular I set export MACOSX_DEPLOYMENT_TARGET=10.7. _That_ alone should not cause this issue. A couple of possible options come to mind here: 1) We might need to set it earlier, that is before the configure step. 2) It's somehow caused by us setting the SDK option to 10.11 now.

I suspect you get the same problem with the build 2 in comment:17?

Tor's workaround for Apple's header declaration bug happens in the configure step, so you'll need to export MACOSX_DEPLOYMENT_TARGET=10.7 before tor's configure runs,

comment:30 in reply to:  28 Changed 3 months ago by mcs

Replying to gk:

I suspect you get the same problem with the build 2 in comment:17?

Yes.

comment:32 in reply to:  31 ; Changed 2 months ago by mcs

Replying to gk:

Alright: mcs/brade how does:

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_4-osx64_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_4-osx64_en-US.dmg.asc

That build fixes the problem with tor.real. With tor working, we could test further on macOS 10.9. Unfortunately, meek is not working. Trying to exec meek-client-torbrowser shows:

 dyld: Symbol not found: _unlinkat
  Referenced from: .../Tor Browser.app/Contents/MacOS/./Tor/PluggableTransports/meek-client-torbrowser
  Expected in: flat namespace

According to the unlink/unlinkat man page on macOS 10.14, "The unlinkat() system call appeared in OS X 10.10."

comment:33 in reply to:  32 ; Changed 2 months ago by gk

Cc: dcf added
Keywords: TorBrowserTeam201908 added; TorBrowserTeam201907 removed

Replying to mcs:

Replying to gk:

Alright: mcs/brade how does:

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_4-osx64_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_4-osx64_en-US.dmg.asc

That build fixes the problem with tor.real. With tor working, we could test further on macOS 10.9. Unfortunately, meek is not working. Trying to exec meek-client-torbrowser shows:

 dyld: Symbol not found: _unlinkat
  Referenced from: .../Tor Browser.app/Contents/MacOS/./Tor/PluggableTransports/meek-client-torbrowser
  Expected in: flat namespace

According to the unlink/unlinkat man page on macOS 10.14, "The unlinkat() system call appeared in OS X 10.10."

So, we have min OS version 10.9 and SDK version 10.9 but a syscall got used that appeared in 10.10? Weird.

comment:34 Changed 2 months ago by gk

mcs/brade: could you assemble a list of minimal requirements we have to have to get the notarization going, with the focus on what we'd need for our signing machine (plus a script or something you used).

Another thought I had: can we buy us some time if we pretend we have signed our stuff _before_ June 2019? IIRC the notarization requirement is only a requirement for binaries signed _after_ that threshold.

comment:35 in reply to:  33 Changed 2 months ago by teor

Replying to gk:

Replying to mcs:

That build fixes the problem with tor.real. With tor working, we could test further on macOS 10.9. Unfortunately, meek is not working. Trying to exec meek-client-torbrowser shows:

 dyld: Symbol not found: _unlinkat
  Referenced from: .../Tor Browser.app/Contents/MacOS/./Tor/PluggableTransports/meek-client-torbrowser
  Expected in: flat namespace

According to the unlink/unlinkat man page on macOS 10.14, "The unlinkat() system call appeared in OS X 10.10."

So, we have min OS version 10.9 and SDK version 10.9 but a syscall got used that appeared in 10.10? Weird.

In 10.10.0, Apple correctly declared unlinkat() as only available in 10.10 and later:

int	unlinkat(int, const char *, int) __OSX_AVAILABLE_STARTING(__MAC_10_10, __IPHONE_8_0);

https://opensource.apple.com/source/xnu/xnu-2782.1.97/bsd/sys/unistd.h.auto.html

In 10.9.5 and earlier, unlinkat() does not exist, and is not declared by Apple.

Maybe Go's bindings are wrong?

Last edited 2 months ago by teor (previous) (diff)

comment:36 in reply to:  32 ; Changed 2 months ago by gk

Replying to mcs:

Replying to gk:

Alright: mcs/brade how does:

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_4-osx64_en-US.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-30126_4-osx64_en-US.dmg.asc

That build fixes the problem with tor.real. With tor working, we could test further on macOS 10.9. Unfortunately, meek is not working. Trying to exec meek-client-torbrowser shows:

 dyld: Symbol not found: _unlinkat
  Referenced from: .../Tor Browser.app/Contents/MacOS/./Tor/PluggableTransports/meek-client-torbrowser
  Expected in: flat namespace

According to the unlink/unlinkat man page on macOS 10.14, "The unlinkat() system call appeared in OS X 10.10."

Okay, I am not convinced yet this is caused by the potential changes making notarization possible. mcs/brade: If I am reading your comments right, you did _not_ test whether meek as we ship it in 9.0a4 works on your 10.9 system? If so, could you do that just to be sure whether the above problem is actually a new bug caused by my patch?

I suspect the underlying problem is a change in Go 1.12 where Go is starting to use libSystem.dylib etc. for syscalls: https://github.com/golang/go/issues/17490. But in that case we should hit the issue in our already existing bundles. (The mac-ports folks had to work around that e.g. in https://trac.macports.org/ticket/58138).

comment:37 in reply to:  36 ; Changed 2 months ago by mcs

Replying to gk:

Okay, I am not convinced yet this is caused by the potential changes making notarization possible. mcs/brade: If I am reading your comments right, you did _not_ test whether meek as we ship it in 9.0a4 works on your 10.9 system? If so, could you do that just to be sure whether the above problem is actually a new bug caused by my patch?

I suspect the underlying problem is a change in Go 1.12 where Go is starting to use libSystem.dylib etc. for syscalls: https://github.com/golang/go/issues/17490. But in that case we should hit the issue in our already existing bundles. (The mac-ports folks had to work around that e.g. in https://trac.macports.org/ticket/58138).

I don't think we tested 9.0a4 + meek on macOS 10.9.x but we can do so.

comment:38 in reply to:  37 ; Changed 2 months ago by mcs

Replying to mcs:

I don't think we tested 9.0a4 + meek on macOS 10.9.x but we can do so.

As you predicted, TB 9.0a4 has the same issue: meek-client-torbrowser expects to find_unlinkat, but that entry point does not exist on macOS 10.9.x.

comment:39 in reply to:  38 Changed 2 months ago by gk

Replying to mcs:

Replying to mcs:

I don't think we tested 9.0a4 + meek on macOS 10.9.x but we can do so.

As you predicted, TB 9.0a4 has the same issue: meek-client-torbrowser expects to find_unlinkat, but that entry point does not exist on macOS 10.9.x.

That's #31464 now.

comment:40 Changed 8 weeks ago by ha

Are the entitlement files Tor plans to use available online somewhere to look at?

If you're using the Firefox production entitlements as a starting point, you might be able to change some rules to be more restrictive.

Assuming Tor only loads shared libraries signed by Tor or Apple, you should be able to set the disable library validation entitlement[1] to false. Firefox needs to load libraries signed by Adobe and Google for Flash and Widevine video decoding respectively.

com.apple.security.cs.disable-library-validation=false

In Firefox, we had to recently set this[2] to true because some WebExtensions using the native message API relied on helper applications that use Apple Events. I suspect Tor wouldn't need this and could set the entitlement to false.

com.apple.security.automation.apple-events=false

  1. https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_disable-library-validation
  2. https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_automation_apple-events
Last edited 8 weeks ago by ha (previous) (diff)

comment:41 in reply to:  40 Changed 8 weeks ago by gk

Replying to ha:

Are the entitlement files Tor plans to use available online somewhere to look at?

Not yet.

If you're using the Firefox production entitlements as a starting point, you might be able to change some rules to be more restrictive.

Yes, I think starting with the Firefox production ones was our plan.

Assuming Tor only loads shared libraries signed by Tor or Apple, you should be able to set the disable library validation entitlement[1] to false. Firefox needs to load libraries signed by Adobe and Google for Flash and Widevine video decoding respectively.

com.apple.security.cs.disable-library-validation=false

In Firefox, we had to recently set this[2] to true because some WebExtensions using the native message API relied on helper applications that use Apple Events. I suspect Tor wouldn't need this and could set the entitlement to false.

com.apple.security.automation.apple-events=false

  1. https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_disable-library-validation
  2. https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_automation_apple-events

Thanks for those hints, really appreciated. We might start with the Firefox ones first, though, to get a feeling for the whole process but are looking forward to tighten the entitlements down as far as we can. And, yes, setting the entitlements above as you suggested makes a lot of sense to me from what I've read and you said.

comment:42 in reply to:  34 ; Changed 7 weeks ago by mcs

Replying to gk:

mcs/brade: could you assemble a list of minimal requirements we have to have to get the notarization going, with the focus on what we'd need for our signing machine (plus a script or something you used).

The only scripts we used were very simple ones that only saved us typing the commands I mentioned in comment:11 and comment:20 (codesign, xcrun altool, and xcrun stapler commands). I don't know exactly how the notarization steps will fit into the overall Tor Browser build process, but ideally someone would write a script to automate things and especially to allow the submission/wait for a reply from Apple part to be done in parallel for our .dmg files. Maybe that is something boklm could do?

Now that we have solved (most of) the build-related requirements, here are the remaining things we need:

  • An Apple Developer ID key and certificate (I think we already have this for the existing Gatekeeper signing).
  • An entitlements file. So far we have always used the one from Firefox, e.g., https://searchfox.org/mozilla-esr68/source/security/mac/hardenedruntime/production.entitlements.xml
  • A macOS computer running 10.13.6 or later (required for the xcrun notarization commands that are part of Xcode 10.1 and later). I do not know enough about the Tor Browser signing and release process to know what kind of computer to recommend. If we plan to buy a new computer and portability is needed, maybe a MacBook Air. If portability is less of a concern, maybe a Mac Mini (still somewhat portable but you need to add a keyboard, mouse, and display).
  • A copy of Xcode 10.1 or later (note that 10.3 is the highest stable release, but 10.2 and up require macOS 10.14.3 or later).
  • Connectivity to the Internet (at least to reach Apple's timestamping and notarization servers).
  • A script or set of scripts to automated things some, especially for the part where we have to wait for Apple to respond to the a notarization request. This and the network connectivity requirement are the most annoying parts of the entire process.

Another thought I had: can we buy us some time if we pretend we have signed our stuff _before_ June 2019? IIRC the notarization requirement is only a requirement for binaries signed _after_ that threshold.

This is an interesting idea, but it seems like a loophole that Apple would have closed by now. But maybe it would work. I don't have any experience with running a timestamping server; can we easily set one up that uses a time prior to June 2019?

Kathy and I would like to install the macOS 10.15 beta and see what the behavior is if someone tries to run an app that has not been notarized (and also to see how difficult it is for people to work around a lack of notarization). But other ESR68 work seems more important given the fact that items such as the updater and meek affect all platforms/all OS versions.

comment:43 in reply to:  42 Changed 7 weeks ago by teor

Replying to mcs:

Replying to gk:

  • A macOS computer running 10.13.6 or later (required for the xcrun notarization commands that are part of Xcode 10.1 and later). I do not know enough about the Tor Browser signing and release process to know what kind of computer to recommend. If we plan to buy a new computer and portability is needed, maybe a MacBook Air. If portability is less of a concern, maybe a Mac Mini (still somewhat portable but you need to add a keyboard, mouse, and display).

New macs will come with the latest macOS.

  • A copy of Xcode 10.1 or later (note that 10.3 is the highest stable release, but 10.2 and up require macOS 10.14.3 or later).

Downloadable from the App Store, requires an App Store account for every download and update.

  • Connectivity to the Internet (at least to reach Apple's timestamping and notarization servers).

Another thought I had: can we buy us some time if we pretend we have signed our stuff _before_ June 2019? IIRC the notarization requirement is only a requirement for binaries signed _after_ that threshold.

This is an interesting idea, but it seems like a loophole that Apple would have closed by now. But maybe it would work. I don't have any experience with running a timestamping server; can we easily set one up that uses a time prior to June 2019?

Apple has banned apps for evading rules like this. Might not be the best idea.

comment:44 Changed 7 weeks ago by gk

Keywords: TorBrowserTeam201909 added; TorBrowserTeam201908 removed

Moving tickets to September

comment:45 Changed 7 weeks ago by pili

Points: 2

comment:46 Changed 6 weeks ago by mcs

Apple just announced that they have relaxed the requirements for notarization (until January 2020):
https://developer.apple.com/news/?id=09032019a

At this point, I think the only item on their list that might help us is that a secure timestamp is not required during code signing.

comment:47 Changed 6 weeks ago by gk

Just as a status update. I think I am done setting the Mac up properly to begin with codesigning experimentation.

comment:48 Changed 5 weeks ago by gk

Okay, mcs/brade: What about:

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-osx64_en-US_30126_signed.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-osx64_en-US_30126_signed.dmg.asc

It works on the 10.14 system I have and Gatekepper is telling me that Apple checked our package for malware (and did not find any !11!).

comment:49 in reply to:  48 Changed 5 weeks ago by mcs

Replying to gk:

Okay, mcs/brade: What about:

https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-osx64_en-US_30126_signed.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-tbb-nightly-osx64_en-US_30126_signed.dmg.asc

It works on the 10.14 system I have and Gatekepper is telling me that Apple checked our package for malware (and did not find any !11!).

As I mentioned on IRC, when we use the above build we are blocked by Gatekeeper on macOS 10.15 beta 7 (but everything is OK on 10.14.6). I will attach a screenshot of the error we see.

Another data point is the nightly build that Kathy and I notarized a few weeks ago using our own Apple developer identity works fine on both OS versions,and so does Firefox 68.1.0 ESR.

I wonder if there is some difference in the process you used that is breaking things? For example, Kathy and I did not create a DMG after notarizing a zipped up copy of Tor Browser.app.

Nearly all of the command line checks we have tried indicate that everything is OK, e.g.,

% codesign -vvv --deep --strict ./Tor\ Browser.app
...
./Tor Browser.app: valid on disk
./Tor Browser.app: satisfies its Designated Requirement

% spctl -vvv --assess --type exec ./Tor\ Browser.app/
./Tor Browser.app/: accepted
source=Notarized Developer ID
origin=Developer ID Application: The Tor Project, Inc (MADPSAYN6T)

% codesign -dvv ./Tor\ Browser.app
Executable=/Applications/Tor Browser.app/Contents/MacOS/firefox
Identifier=org.torproject.torbrowser
Format=app bundle with Mach-O thin (x86_64)
CodeDirectory v=20500 size=421 flags=0x10000(runtime) hashes=4+5 location=embedded
Signature size=9022
Authority=Developer ID Application: The Tor Project, Inc (MADPSAYN6T)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Sep 10, 2019 at 7:07:40 AM
Info.plist entries=27
TeamIdentifier=MADPSAYN6T
Runtime Version=10.11.0
Sealed Resources version=2 rules=13 files=130
Internal requirements count=1 size=188

% xcrun stapler validate ./Tor\ Browser.app
Processing: /Applications/Tor Browser.app
The validate action worked!

There is one command variant which fails; compare these two (--type exec vs. --type open):

% spctl -vvvv --assess --type exec --context context:primary-signature Tor\ Browser.app
Tor Browser.app: accepted
source=Notarized Developer ID
origin=Developer ID Application: The Tor Project, Inc (MADPSAYN6T)

% spctl -vvvv --assess --type open --context context:primary-signature Tor\ Browser.app
Tor Browser.app: rejected
source=Unnotarized Developer ID
origin=Developer ID Application: The Tor Project, Inc (MADPSAYN6T)

With the .app that Kathy and I notarized, both of these commands succeed. I am not sure if this is an important difference, but it is the only one we have found so far.

There must be some step that we missed. I assume you included the entitlements file? Can you give us the zipped up Tor Browser.app to try (i.e., no .dmg processing)?

Other ideas?

Changed 5 weeks ago by mcs

Attachment: GatekeeperError.png added

macOS 10.15 Gatekeeper error alert

comment:50 Changed 5 weeks ago by gk

Okay, here comes the zipped up .app dir:

https://people.torproject.org/~gk/testbuilds/tbb-30126.zip
https://people.torproject.org/~gk/testbuilds/tbb-30126.zip.asc

When I unzip the archive after doing all the codesigning things I just end up with a Contents folder. I need to (re-)create Tor Browser.app and move that one into it. Not sure whether that's expected. Another thing I probably did differently: I looked at the codesign.bash file in security/mac/hardenedruntime and used an adapted
ditto -c -k "${BUNDLE}" "${OUTPUT_ZIP_FILE}" for zipping the bundle up after signing but before notarization.

Other than that I don't know.

comment:51 in reply to:  50 ; Changed 5 weeks ago by mcs

Replying to gk:

Okay, here comes the zipped up .app dir:

https://people.torproject.org/~gk/testbuilds/tbb-30126.zip
https://people.torproject.org/~gk/testbuilds/tbb-30126.zip.asc

Using this results in the same behavior (works fine on macOS 10.14.6, Gatekeeper error on 10.15 beta).

When I unzip the archive after doing all the codesigning things I just end up with a Contents folder. I need to (re-)create Tor Browser.app and move that one into it. Not sure whether that's expected. Another thing I probably did differently: I looked at the codesign.bash file in security/mac/hardenedruntime and used an adapted
ditto -c -k "${BUNDLE}" "${OUTPUT_ZIP_FILE}" for zipping the bundle up after signing but before notarization.

What did you submit to Apple? As described in comment:11, Kathy and I ran the codesign command on Tor Browser.app and then we created a .zip that contained Tor Browser.app, which we then submitted via the xcrun altool --notarize-app ... command.

But I just realized there is a much bigger difference between what you are doing and our earlier experiments: because we did not have ESR68 macOS builds at that time, Kathy and I used an ESR60-based nightly build. We will try to re-create our experiment using a current nightly build.

comment:52 in reply to:  51 Changed 5 weeks ago by mcs

Replying to mcs:

But I just realized there is a much bigger difference between what you are doing and our earlier experiments: because we did not have ESR68 macOS builds at that time, Kathy and I used an ESR60-based nightly build. We will try to re-create our experiment using a current nightly build.

I did the ESR68-based experiment using browser bits that I extracted from your comment:48 build. My notarized and stapled Tor Browser.app opens correctly on macOS 10.15. I used the entitlements file from https://gitweb.torproject.org/tor-browser.git/plain/security/mac/hardenedruntime/production.entitlements.xml?h=tor-browser-68.1.0esr-9.0-2-build2

In detail, here are the steps I followed (all on a macOS 10.14.6 computer):

Opened your .dmg in Finder and copied Tor Browser.app to a new folder.

Removed your signatures:

rm -rf Tor\ Browser.app/Contents/CodeResources Tor\ Browser.app/Contents/_CodeSignature

Signed it and created tb.zip which contains Tor Browser.app at the top level:

CERT="Developer ID Application: XXX"
ENTITLEMENTS=entitlements/production.entitlements.xml
codesign -vvv --deep -o runtime --entitlements "$ENTITLEMENTS" \
    --timestamp -f -s "$CERT" "Tor Browser.app/"
zip -qr tb.zip "Tor Browser.app"

Submitted the zip file for notarization:

BUNDLEID="org.torproject.torbrowser"
xcrun altool --notarize-app -t osx -f tb.zip --primary-bundle-id "$BUNDLEID" \
    -u REDACTED -p @env:PW --output-format xml

Checked status until it was done:

xcrun altool --notarization-info GUID \
    -u REDACTED -p @env:PW --output-format xml

Stapled the notarization ticket to the app bundle and created a new zip file:

xcrun stapler staple Tor\ Browser.app
zip -r tb-stapled.zip Tor\ Browser.app

Then I put tb-stapled.zip on an HTTP server and downloaded it to macOS for testing.

There were three things that surprised me on macOS 10.15:

  1. The "Tor Browser is an app downloaded from the Internet. Are you sure you want to open it?" prompt did not mention that the app had been checked by Apple for malicious software. But that message does not appear for Firefox 68.1.0 ESR either
  2. Even though I had the app on the desktop, wjen I clicked Open and allowed Tor Browser to start up, it placed its TorBrowser-Data folder under ~/Library/Application Support/TorBrowser-Data/ instead of next to the app. Apparently notarized applications do not have access to the desktop by default, because this problem occurs on macOS 10.14.6 as well.
  3. A more serious problem is that on macOS 10.15 but not on 10.14.6, all tabs seem to crash (content process crash). This problem and 2. both disappear if I run ./Tor Browser.app/Contents/MacOS/firefox from bash.
Last edited 5 weeks ago by gk (previous) (diff)

comment:53 Changed 5 weeks ago by gk

Hrm, okay. Let's first test whether my signing attempts now succeed by following your instructions closer (I think I messed up the stapling at least yesterday) and appease our Gatekeeper:

https://people.torproject.org/~gk/testbuilds/TorBrowser-9.0a6-osx64_en-US_30126.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-9.0a6-osx64_en-US_30126.dmg.asc

I then thought, testing with 8.5.5 as well because maybe the crash you saw is esr68-based. Here is the result:

https://people.torproject.org/~gk/testbuilds/TorBrowser-8.5.5-osx64_en-US_30126.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-8.5.5-osx64_en-US_30126.dmg.asc

Note: the latter is very surprising. macOS shows both for alpha and stable that Apple checked for malware and the notarization succeeded, however, we did not backport the build changes made in #31465 yet. Will be interesting to see what Catalina says.

Regarding the crash there is https://bugzilla.mozilla.org/show_bug.cgi?id=1578075 which we might somehow hit? Although it would be kind of weird why starting from bash would avoid that.

comment:54 Changed 5 weeks ago by gk

Checking the signed 8.5.5 with x86_64-apple-darwin11-otool -l Tor\ Browser.app/Contents/MacOS/firefox shows still

cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.7
      sdk 10.6

So, maybe Apple silently relaxed some requirements here (too)?

Last edited 5 weeks ago by gk (previous) (diff)

comment:55 in reply to:  54 Changed 5 weeks ago by mcs

Replying to gk:

So, maybe Apple silently relaxed some requirements here (too)?

Not silently :) From the Apple announcement mentioned in comment:46:

You can now notarize Mac software that: ... Was built with an older SDK.

comment:56 in reply to:  53 Changed 5 weeks ago by mcs

Replying to gk:

Hrm, okay. Let's first test whether my signing attempts now succeed by following your instructions closer (I think I messed up the stapling at least yesterday) and appease our Gatekeeper:

https://people.torproject.org/~gk/testbuilds/TorBrowser-9.0a6-osx64_en-US_30126.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-9.0a6-osx64_en-US_30126.dmg.asc

I then thought, testing with 8.5.5 as well because maybe the crash you saw is esr68-based. Here is the result:

https://people.torproject.org/~gk/testbuilds/TorBrowser-8.5.5-osx64_en-US_30126.dmg
https://people.torproject.org/~gk/testbuilds/TorBrowser-8.5.5-osx64_en-US_30126.dmg.asc

Both of these builds work fine on both 10.15 beta 7 and 10.14.6. Even the TorBrowser-Data weirdness and content process crashes have disappeared.

Kathy and I also tested the 9.0a6 build on macOS 10.9.x and confirmed that the #31464 problem is gone too. Good news all the way around!

comment:57 Changed 5 weeks ago by gk

Actual Points: 2.5
Resolution: fixed
Status: newclosed

\o/ It seems we are done here. I opened #31702 for backporting the crash bug fix and then we should roll out updates asap I think, ideally starting with our alpha series.

mcs/brade please add to the points actually spent. I am just adding mine.

comment:58 Changed 5 weeks ago by gk

Resolution: fixed
Status: closedreopened

comment:59 Changed 5 weeks ago by gk

Resolution: fixed
Status: reopenedclosed

comment:60 Changed 5 weeks ago by mcs

Actual Points: 2.55.5
Note: See TracTickets for help on using tickets.