Adapt macOS snowflake compilation to new toolchain
We need to adapt our macOS toolchain due to new Firefox requirements when switching to ESR 60 (the new one will use a different cctools, clang 3.9.1 and the SDK 10.11). This means the snowflake and related projects need to get updated, too.
I'll disable building and shipping snowflake until this bug is fixed (alas, it's not a top prio right now for us) in order to get macOS nightlies based on ESR 60 going as fast as possible.
Activity
bug_26195(https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_26195) has three commits that build the new toolchain and could be a good start for this bug. I am not done preparing the patches for all the projects yet, but I can build the whole mac bundle with it (modulo the snowflake parts) if I did not miss some project (I did not attempt to build the whole bundle straight yet).Please review:
https://github.com/azadi/tor-browser-build-1/tree/bug-26251
I have not properly patched
build/config/mac/BUILD.gninwebrtc-mac.patchas you may notice and just added the-Wno-unknown-warning-optionmanually. I can do that inrev2or before finalizing the patch, but I wanted to put it out for review before I deal withgclient.I tested it on macOS and by modifying
torrc(thanks to arlolra for the tip) as PT selection is missing pending #26039 (moved).https://github.com/keroserene/snowflake/blob/master/client/torrc
Tor NOTICE: Bootstrapped 5%: Connecting to directory server Tor NOTICE: Bootstrapped 10%: Finishing handshake with directory server Tor NOTICE: Learned fingerprint <> for bridge 0.0.3.0:1 (with transport 'snowflake'). Tor NOTICE: Bootstrapped 15%: Establishing an encrypted directory connection Tor NOTICE: Bootstrapped 20%: Asking for networkstatus consensus Tor NOTICE: new bridge descriptor <> Tor NOTICE: Bootstrapped 25%: Loading networkstatus consensus Tor NOTICE: I learned some more directory information, but not enough to build a circuit: We have no usable consensus. Tor NOTICE: Bootstrapped 40%: Loading authority key certs Tor NOTICE: Bootstrapped 45%: Asking for relay descriptors ... Tor NOTICE: Bootstrapped 50%: Loading relay descriptors Tor NOTICE: Bootstrapped 57%: Loading relay descriptors Tor NOTICE: Bootstrapped 66%: Loading relay descriptors Tor NOTICE: Bootstrapped 73%: Loading relay descriptors Tor NOTICE: Bootstrapped 80%: Connecting to the Tor network Tor NOTICE: Bootstrapped 90%: Establishing a Tor circuit Tor NOTICE: Tor has successfully opened a circuit. Looks like client functionality is working.Trac:
Status: assigned to needs_review-
Before I forget it: we should make sure snowflake is working on macOS < 10.11. IIRC there was a note somewhere in the snowflake build patch indicating that the SDK version used is at the same time the minimum macOS version supported. But that's not the case for us: we use the 10.11 SDK but want to have Tor Browser running on, say, 10.9 as well.
Replying to gk:
Before I forget it: we should make sure snowflake is working on macOS < 10.11. IIRC there was a note somewhere in the snowflake build patch indicating that the SDK version used is at the same time the minimum macOS version supported. But that's not the case for us: we use the 10.11 SDK but want to have Tor Browser running on, say, 10.9 as well.
Good point and I think dcf or arlolra can comment better, but two things I did not change for this reason:
print("machine_os_build=\"10.7\"")and
GN_ARGS+=" target_os=\"mac\" target_cpu=\"x64\" mac_deployment_target=\"10.7\""But I tested it on 10.13.4 High Sierra so I am not sure.
EDIT: Version on which I tested.
-
Replying to sukhbir:
Please review:
https://github.com/azadi/tor-browser-build-1/tree/bug-26251
I have not properly patched
build/config/mac/BUILD.gninwebrtc-mac.patchas you may notice and just added the-Wno-unknown-warning-optionmanually. I can do that inrev2or before finalizing the patch, but I wanted to put it out for review before I deal withgclient.Does not look unreasonable
-mlinker-version=136and probably other flags are not needed anymore. You could compare the macOS mozconfig file used on themaint-7.5and on themasterbranch for the diff. Then I am looking forward to the full patch withgclientetc.Trac:
Status: needs_review to needs_revision For review:
https://github.com/azadi/tor-browser-build-1/tree/bug-26251-rev1
I removed
-mlinker-version=136but the other stuff seems to be relevant.$ git diff tor/maint-7.5...tor/master -- projects/firefox/mozconfig-osx-x86_64 -FLAGS="-target x86_64-apple-darwin10 -mlinker-version=136 -B $CROSS_CCTOOLS_PATH/bin -isysroot $CROSS_SYSROOT $HARDENING_FLAGS" +FLAGS="-target x86_64-apple-darwin11 -B $CROSS_CCTOOLS_PATH/bin -isysroot $CROSS_SYSROOT $HARDENING_FLAGS" -export TOOLCHAIN_PREFIX=$CROSS_CCTOOLS_PATH/bin/x86_64-apple-darwin10- -#TODO: bug 1184202 - would be nice if these could be detected with TOOLCHAIN_PREFIX automatically -export AR=${TOOLCHAIN_PREFIX}ar -export RANLIB=${TOOLCHAIN_PREFIX}ranlib -export STRIP=${TOOLCHAIN_PREFIX}strip -export OTOOL=${TOOLCHAIN_PREFIX}otool +export BINDGEN_CFLAGS="$FLAGS" +export TOOLCHAIN_PREFIX=$CROSS_CCTOOLS_PATH/bin/x86_64-apple-darwin11-I tested with the same setup as in #comment:4. As for as
gclient, I didn't need to do that since I did an in-place of the patch. (We will end up with the same thing since I replaced the linker line with the clang flag, so I didn't bother.)Trac:
Status: needs_revision to needs_reviewReplying to sukhbir:
dcf: This commit makes a change to your patch. Can you please go through it once and see if the change is fine with you before it's merged? Since I modified the patch in-place, I don't want it to attribute to you without your permission.
I'll take a look but generally I don't worry about the attribution--set it to yourself or to me, it doesn't matter much. Thanks for checking.
After some trial and error and a lot of time waiting for macOS to install and boot, I succeeded in testing the 2018-07-18 Tor Browser nightly build on macOS 10.9 (via a bootable USB flash drive). Unfortunately, tor.real does not start due to an undefined mach_XXX symbol. I will post the details tomorrow, but it is possible that I just need to install some macOS 10.9.x patches to fix that problem. Or maybe the network team needs to fix that problem before we can get to Snowflake.
Replying to mcs:
After some trial and error and a lot of time waiting for macOS to install and boot, I succeeded in testing the 2018-07-18 Tor Browser nightly build on macOS 10.9 (via a bootable USB flash drive). Unfortunately, tor.real does not start due to an undefined mach_XXX symbol. I will post the details tomorrow, but it is possible that I just need to install some macOS 10.9.x patches to fix that problem. Or maybe the network team needs to fix that problem before we can get to Snowflake.
If you tell us what the mach_XXX symbol is, we'll have a better chance of debugging the issue.
But I'm wondering if it's something like #20241 (moved) or #20235 (moved) - Apple doesn't do weak linking for low-level Darwin functions (like mach_XXX), because their weak linking macros depend on high-level macOS headers.
-
Replying to teor:
If you tell us what the mach_XXX symbol is, we'll have a better chance of debugging the issue.
Indeed. I am sorry for not posting the details last night; it was getting late for me and the log file was on a Flash drive that I didn't have with me when I posted comment:15.
But I'm wondering if it's something like #20241 (moved) or #20235 (moved) - Apple doesn't do weak linking for low-level Darwin functions (like mach_XXX), because their weak linking macros depend on high-level macOS headers.
I filed #26876 (moved) to track this issue. Help from you or other Network Team people would be greatly appreciated!
mentioned in issue #26876 (moved)
mentioned in issue tpo/core/tor#26876 (closed)
