Opened 5 years ago

Closed 6 months ago

#9711 closed task (fixed)

Build our own cctools for macOS cross-compilation

Reported by: mikeperry Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, GeorgKoppen201805, TorBrowserTeam201805R
Cc: gk, arlolra, boklm Actual Points:
Parent ID: #24632 Points:
Reviewer: Sponsor:

Description

Ray Donnelly has been working on merging his OSX cross compiler patches into the crosstools-ng project, which should allow us to more easily build his compilers from source instead of relying on a binary blob.

Jotting down Ray's instructions in case anyone wants to try this:

On Linux it is possible now to build the equivalent of what toolchain4 provided (for OSX only - iOS still has some build issues)

You should be able to clone:
https://github.com/diorcety/crosstool-ng.git

.. checkout the cctools-llvm branch, and enter:

./configure --prefix=<SOMEWHERE>/ct-ng-build
make
make install
export PATH="${PATH}":<SOMEWHERE>/ct-ng-build
mkdir <SOMEWHEREELSE>/ct-ng-final
cd <SOMEWHEREELSE>/ct-ng-final
ct-ng list-samples
ct-ng i686-apple-darwin11 (or i686-apple-darwin10)

Check the differences between samples/i686-apple-darwin11 and samples/i686-apple-darwin10 too, the main one being that 10 copies the sysroot into the final build whereas 11 doesn't. This means you need to pass --sysroot to the compilers built with 11 but in theory don't need to for the 10 ones (of course 10 is set up to use MacOSX10.6.sdk while 11 is set up to use MacOSX10.7.sdk).

We've been adding clang to the project most recently which has caused us to disabled llvm-gcc for now and the addition of clang isn't ready yet (it is nearly there but I am going on holiday for 2 weeks after this week so I'm not sure if we will get it finished before then - likely not).

So, all the above instructions are still valid; go with commit 7d555f284b6977e64640a30bcec77597580d3049 if you can. Any problems give me a shout. FWIW, llvm/clang is only now coming on-line for Darwin software anyway, and vanilla gcc-4.2 (i.e. not even llvm-gcc-4.2) is much more reliable for building OSX software. Seems Apple focussed more on iOS than OSX for clang.

Let me know how it goes. We've started to submit some patches upstream to crosstool-ng this last week, but I have a feeling it could be a drawn out process. We will regularly merge the other way the meantime though.

Child Tickets

Attachments (15)

cross_mac_gmp1 (82.8 KB) - added by gk 5 years ago.
cross_mac_build1 (49.8 KB) - added by gk 5 years ago.
cross_mac_build2 (86.0 KB) - added by gk 5 years ago.
cross_mac_vm (10.1 KB) - added by gk 5 years ago.
build.log.tar.xz (338.4 KB) - added by mingwandroid 5 years ago.
Ray's build.log (tar.xz'ed due to size limits)
build-i686.log.tar.xz (336.5 KB) - added by mingwandroid 5 years ago.
Ray's build.log (tar.xz'ed due to size limits - targeting OS X, not iPhone!)
cross_mac_vm_full.tar.bz2 (180.1 KB) - added by gk 5 years ago.
cross_mac_build3.tar.bz2 (567.9 KB) - added by gk 5 years ago.
cross_mac_build4.tar.bz2 (516.7 KB) - added by gk 5 years ago.
cross_mac_gcc_errors.tar.bz2 (771.2 KB) - added by gk 5 years ago.
0001-crosstool-ng-patch.patch (4.2 KB) - added by gk 5 years ago.
cross_mac_gcc_errors2.tar.bz2 (758.5 KB) - added by gk 5 years ago.
0002-use-ssh-t-for-commands-run-on-target-via-sudo.patch (1.2 KB) - added by mingwandroid 5 years ago.
0001-Use-my-faketime_0.9.6-test-package.patch (3.6 KB) - added by mingwandroid 5 years ago.
Patch for tor-browser-bundle to use my faketime 0.9.6 test package
gcc_build_error.log (14.7 KB) - added by gk 5 years ago.

Change History (101)

comment:1 Changed 5 years ago by gk

Cc: g.koppen@… added

comment:2 Changed 5 years ago by gk

Status: newneeds_information

Mike, one question:
Is there a reason for using i686-apple-darwin11 in the current setup when compiling and not i686-apple-darwin10 given that we want to support OSX10.6 as the minVersion (and are therefore using the 10.6 SDK anyway)?

comment:3 Changed 5 years ago by gk

Oh, and while I am at it: Could you send me Ray's email address? I already encountered bugs while trying to compile the toolchain that seem to be fun bugs :)

comment:4 Changed 5 years ago by mingwandroid

Mike, are you using llvmgcc or normal gcc from toolchain4 at present? If normal gcc, then I'd recommend that you guys build from the latest of the cctools-llvm branch to get the latest fixes (llvmgcc is disabled at present, IMHO it was never very reliable, even the official Apple binaries). If you update to the latest then clang can be built now, however this is very untested so far and if you look in the TODO file in the root folder you'll get a better picture of some things that may still be problematic with clang.

Georg, there's no reason to use darwin11 instead of darwin10. When I started this work it was with a darwin11 SDK so it's purely a matter of habit for me. The merge to crosstool-ng is being done by Yann Diorcet and myself; I handle the darwin11 samples and he does the same for the darwin10 ones.

A compiler built with the darwin11 SDK (MacOSX10.7.sdk) can be used fine to build software for darwin10 using -mmacosx-version-min=10.5, However feel free to base your version on i686-apple-darwin10 (e.g. flosoft's MacOSX10.6.sdk) and the i686-apple-darwin10 sample instead.

I recommend studying the two crosstool.config files:

samples/i686-apple-darwin10/crosstool.config and samples/i686-apple-darwin11/crosstool.config

.. these are just samples of course. You could make your own, e.g. darwin-gitian. The final compiler prefixes don't depend on the sample folder name but rather the options specified in the crosstool.config file itself.

Some options that should be noted are:

CT_DARWIN_SDK_PATH="${HOME}/MacOSX10.7.sdk"
.. this is where you placed your OSX SDK.

CT_DARWIN_COPY_SDK_TO_SYSROOT=n
.. setting this to y means that some headers and dylibs from the OSX SDK are copied into the installation prefix. Doing this means you do not need to pass -sysroot to the compiler on the commandline, with the down-side being that you'd not be able to distribute the built toolchain due to copyrights and the OSX SDK license.

CT_LLVM_V_3_3=y
.. specifies the version of LLVM (and clang) to use. LLVM is compiled into cctools for LTO support and also, obviously for clang. Toolchain4 used LLVM 2.7 which is quite old at this stage. 3.3 is the latest release. I'm more interested in modernising things than using old versions (though reliability trumps all other concerns).

CT_DEBUGGABLE_TOOLCHAIN=n
.. expect to use 20GB if you set this to y. The built tools (and also those installed) are built at -O0 -ggdb and left un-stripped.

I think it would be sensible for me to setup exactly the same environment you are using. Is this Ubuntu? If so, can you point me to the exact ISO and any scripts you use to prepare it for development? I will then make a new VM.

If you've got some logs (build.log) detailing the "fun bugs" please post them.

comment:5 Changed 5 years ago by mingwandroid

Something else I should mention is that the old GCC in the crosstool-ng port doesn't yet support the -arch and -Xarch flags (whereas toolchain4 does).

This is because those flags are actually handled by driverdriver.c which is an Apple-specific program that's compiled outside of the normal GCC/llvmgcc compilation and when run calls the real GCC once for each arch specified (using the standard -m32 or -m64 flag). It then calls lipo to merge the objects for each arch together into a fat object/binary.

I'm not sure how the tbb build system for Mac invokes gcc; I would hope it is easy to change it from using "-arch i386" to "-m32" and/or "-arch x86_64" to "-m64", of course, if you build fat binaries in a single invocation then this is a bigger problem.

Having said that, adding the driverdriver program is high priority and something I expect to do tomorrow.

Finally, clang 'natively' supports these flags but I'd be (pleasantly) surprised if switching tbb to build with clang was trivial.

comment:6 in reply to:  4 ; Changed 5 years ago by gk

Replying to mingwandroid:

Mike, are you using llvmgcc or normal gcc from toolchain4 at present? If normal gcc, then I'd recommend that you guys build from the latest of the cctools-llvm branch to get the latest fixes (llvmgcc is disabled at present, IMHO it was never very reliable, even the official Apple binaries).

Normal gcc and yes, I am using the latest cctools-llvm branch.

Georg, there's no reason to use darwin11 instead of darwin10. When I started this work it was with a darwin11 SDK so it's purely a matter of habit for me. The merge to crosstool-ng is being done by Yann Diorcet and myself; I handle the darwin11 samples and he does the same for the darwin10 ones.

A compiler built with the darwin11 SDK (MacOSX10.7.sdk) can be used fine to build software for darwin10 using -mmacosx-version-min=10.5, However feel free to base your version on i686-apple-darwin10 (e.g. flosoft's MacOSX10.6.sdk) and the i686-apple-darwin10 sample instead.

That was actually the reason why I asked. We use already flosoft's MacOSX10.6.sdk and I hoped to avoid using the 10.7 SDK to get a compiler to use that one with the 10.6 SDK... Good.

I recommend studying the two crosstool.config files:

samples/i686-apple-darwin10/crosstool.config and samples/i686-apple-darwin11/crosstool.config

.. these are just samples of course. You could make your own, e.g. darwin-gitian. The final compiler prefixes don't depend on the sample folder name but rather the options specified in the crosstool.config file itself.

Thanks I'll think about doing an own while I digest that whole system.

I think it would be sensible for me to setup exactly the same environment you are using. Is this Ubuntu? If so, can you point me to the exact ISO and any scripts you use to prepare it for development? I will then make a new VM.

The gitian build system is working with (at least) the current LTS (12.04). I order to build the Mac TBB you do the following

git clone https://git.torproject.org/builders/tor-browser-bundle.git
cd tor-browser-bundle/gitian
make prep

Install all the stuff you are asked to and run

make prep

again until you have all the necessary sources (i.e. until |make prep| does not give you any errors anymore)
Then do

./mkbundle-mac.sh

and it should give you (after a while) Mac TBBs in a new directory (currently 3.0-alpha-4). Anyway, if you have issues with setting it up. Just write me a mail and we'll sort these things out.

If you've got some logs (build.log) detailing the "fun bugs" please post them.

So I have found so far three issues (disclaimer: I am not sure yet if there is anything you can do about as this might actually be things belonging to the source packages used in building the compiler):

1) I hit intermittent GMP configure failures due to newly created files being older than distributed ones. (log: cross_mac_gmp1) That is likely due to the gitian build environment but it would be good to somehow avoid that. I am wondering whether the reason here is some old version of GMP? I have been building mingw-w64 from source for a while with a similar gitian environment but never hit this problem.

2) 'groff' was missing (the check in the configure script is there but although 'groff' is needed for building the script is not returning with an error if it is not available) (cross_mac_build1). Installing the necessary package probably solves the issue.

3) a Makefile.config(?) is missing? (cross_mac_build2).

Now, what bothers me most, though, is that Firefox 24 is not working with a gcc4.2 anymore (at least Mozilla is saying that in the configure script and I don't have a reason to doubt that). I am under the impression that your cross-compilers are (by default) gcc4.2, no? If so, what are our options here? Are there non-interactive ways to change the gcc version?

Changed 5 years ago by gk

Attachment: cross_mac_gmp1 added

Changed 5 years ago by gk

Attachment: cross_mac_build1 added

Changed 5 years ago by gk

Attachment: cross_mac_build2 added

comment:7 in reply to:  6 Changed 5 years ago by gk

Replying to gk:

The gitian build system is working with (at least) the current LTS (12.04). I order to build the Mac TBB you do the following

git clone https://git.torproject.org/builders/tor-browser-bundle.git
cd tor-browser-bundle/gitian
make prep

Install all the stuff you are asked to and run

make prep

again until you have all the necessary sources (i.e. until |make prep| does not give you any errors anymore)
Then do

./mkbundle-mac.sh

and it should give you (after a while) Mac TBBs in a new directory (currently 3.0-alpha-4). Anyway, if you have issues with setting it up. Just write me a mail and we'll sort these things out.

I forgot to add that the relevant files for cross-compiling the TBB code are gitian-tor.yml and gitian-firefox.yml in tor-browser-bundle/gitian/descriptors/mac. This should give you a hint on how your cross-compilers are currently used in the gitian build setup. If you have questions about that as well, drop me a mail.

comment:8 in reply to:  6 ; Changed 5 years ago by mingwandroid

Replying to gk:

Replying to mingwandroid:

Mike, are you using llvmgcc or normal gcc from toolchain4 at present? If normal gcc, then I'd recommend that you guys build from the latest of the cctools-llvm branch to get the latest fixes (llvmgcc is disabled at present, IMHO it was never very reliable, even the official Apple binaries).

Normal gcc and yes, I am using the latest cctools-llvm branch.

Georg, there's no reason to use darwin11 instead of darwin10. When I started this work it was with a darwin11 SDK so it's purely a matter of habit for me. The merge to crosstool-ng is being done by Yann Diorcet and myself; I handle the darwin11 samples and he does the same for the darwin10 ones.

A compiler built with the darwin11 SDK (MacOSX10.7.sdk) can be used fine to build software for darwin10 using -mmacosx-version-min=10.5, However feel free to base your version on i686-apple-darwin10 (e.g. flosoft's MacOSX10.6.sdk) and the i686-apple-darwin10 sample instead.

That was actually the reason why I asked. We use already flosoft's MacOSX10.6.sdk and I hoped to avoid using the 10.7 SDK to get a compiler to use that one with the 10.6 SDK... Good.

I recommend studying the two crosstool.config files:

samples/i686-apple-darwin10/crosstool.config and samples/i686-apple-darwin11/crosstool.config

.. these are just samples of course. You could make your own, e.g. darwin-gitian. The final compiler prefixes don't depend on the sample folder name but rather the options specified in the crosstool.config file itself.

Thanks I'll think about doing an own while I digest that whole system.

I think it would be sensible for me to setup exactly the same environment you are using. Is this Ubuntu? If so, can you point me to the exact ISO and any scripts you use to prepare it for development? I will then make a new VM.

The gitian build system is working with (at least) the current LTS (12.04). I order to build the Mac TBB you do the following

Should I use the 32bit version of the LTS?

git clone https://git.torproject.org/builders/tor-browser-bundle.git
cd tor-browser-bundle/gitian
make prep

Install all the stuff you are asked to and run

make prep

again until you have all the necessary sources (i.e. until |make prep| does not give you any errors anymore)
Then do

./mkbundle-mac.sh

and it should give you (after a while) Mac TBBs in a new directory (currently 3.0-alpha-4). Anyway, if you have issues with setting it up. Just write me a mail and we'll sort these things out.

If you've got some logs (build.log) detailing the "fun bugs" please post them.

So I have found so far three issues (disclaimer: I am not sure yet if there is anything you can do about as this might actually be things belonging to the source packages used in building the compiler):

1) I hit intermittent GMP configure failures due to newly created files being older than distributed ones. (log: cross_mac_gmp1) That is likely due to the gitian build environment but it would be good to somehow avoid that. I am wondering whether the reason here is some old version of GMP? I have been building mingw-w64 from source for a while with a similar gitian environment but never hit this problem.

In general, how does gitian handle filestamp dependencies? I'm under the impression that it uses .so injection to force specific time functions to return the same thing, given that, I'm a little confused. Is there a mechanism to force some fixed negative delta onto files created by specific tools (tar and patch come to mind)?

2) 'groff' was missing (the check in the configure script is there but although 'groff' is needed for building the script is not returning with an error if it is not available) (cross_mac_build1). Installing the necessary package probably solves the issue.

Ok this sounds like an upstream crosstool-ng problem. I will put it on the TODO list for now.

3) a Makefile.config(?) is missing? (cross_mac_build2).

Now, what bothers me most, though, is that Firefox 24 is not working with a gcc4.2 anymore (at least Mozilla is saying that in the configure script and I don't have a reason to doubt that). I am under the impression that your cross-compilers are (by default) gcc4.2, no? If so, what are our options here? Are there non-interactive ways to change the gcc version?

4.2.1 was as the last GCC that Apple provided. They heavily patched this version (and made llvmgcc from it) and they never updated beyond that (understandable given the level and nature of patching, of course they could've opted to make their changes cleanly and worked with GCC core developers to upstream but I suspect GCC was something they worked on grudgingly). They switched from GCC to clang, so when you say that Firefox 24 is not working with gcc4.2 anymore, I can only assume that for Mac, they switched to building with clang - or maybe llvmgcc, if you can determine the exact details of this it would be helpful, also can you find out it requires libc++? When do you plan to switch over to Firefox 24?

I will download your git repo, give it a test and study your logs at lunchtime. You didn't say whether you use the -arch flag or not, but no problem I can take a look.

comment:9 Changed 5 years ago by gk

Just for reference: Even though the build (i686-apple-darwin10) is much smoother outside the gitian setup it still does not go through on a clean Ubuntu VM, see output in cross_mac_vm.

Changed 5 years ago by gk

Attachment: cross_mac_vm added

Changed 5 years ago by mingwandroid

Attachment: build.log.tar.xz added

Ray's build.log (tar.xz'ed due to size limits)

Changed 5 years ago by mingwandroid

Attachment: build-i686.log.tar.xz added

Ray's build.log (tar.xz'ed due to size limits - targeting OS X, not iPhone!)

comment:10 Changed 5 years ago by mingwandroid

I added two logs, the first is actually targeting ARM (iPhone), so you probably should ignore that one for now.

Was your log complete? In mine, "[INFO ] Installing cctools for host" doesn't happen until line 42686 whereas it's the first line in your log. I'm guessing you snipped your log to avoid size limits and so that it can be referred to via URLs? I think having your complete log would be helpful if this is the case.

From your log:
"[CFG ] configure: WARNING: llvm-config (/home/firefox/x-tools/i686-apple-darwin10/bin/i686-apple-darwin10-llvm-config) not found, will use LLVM 2.7 defaults" is interesting. It indicates that LLVM wasn't built or installed correctly.

comment:11 in reply to:  10 Changed 5 years ago by gk

Replying to mingwandroid:

I added two logs, the first is actually targeting ARM (iPhone), so you probably should ignore that one for now.

Was your log complete? In mine, "[INFO ] Installing cctools for host" doesn't happen until line 42686 whereas it's the first line in your log. I'm guessing you snipped your log to avoid size limits and so that it can be referred to via URLs? I think having your complete log would be helpful if this is the case.

From your log:
"[CFG ] configure: WARNING: llvm-config (/home/firefox/x-tools/i686-apple-darwin10/bin/i686-apple-darwin10-llvm-config) not found, will use LLVM 2.7 defaults" is interesting. It indicates that LLVM wasn't built or installed correctly.

Attached is the full build log (cross_mac_vm_full.tar.bz2).

Changed 5 years ago by gk

Attachment: cross_mac_vm_full.tar.bz2 added

comment:12 in reply to:  8 Changed 5 years ago by gk

Replying to mingwandroid:
[snip]

Should I use the 32bit version of the LTS?

That does not matter as gitian is creating own VMs for building.

[snip]

In general, how does gitian handle filestamp dependencies? I'm under the impression that it uses .so injection to force specific time functions to return the same thing, given that, I'm a little confused. Is there a mechanism to force some fixed negative delta onto files created by specific tools (tar and patch come to mind)?

Mike, could you comment on that. I am not sure about the details...

Now, what bothers me most, though, is that Firefox 24 is not working with a gcc4.2 anymore (at least Mozilla is saying that in the configure script and I don't have a reason to doubt that). I am under the impression that your cross-compilers are (by default) gcc4.2, no? If so, what are our options here? Are there non-interactive ways to change the gcc version?

4.2.1 was as the last GCC that Apple provided. They heavily patched this version (and made llvmgcc from it) and they never updated beyond that (understandable given the level and nature of patching, of course they could've opted to make their changes cleanly and worked with GCC core developers to upstream but I suspect GCC was something they worked on grudgingly). They switched from GCC to clang, so when you say that Firefox 24 is not working with gcc4.2 anymore, I can only assume that for Mac, they switched to building with clang

Yes, they did: https://bugzilla.mozilla.org/show_bug.cgi?id=733905

  • or maybe llvmgcc, if you can determine the exact details of this it would be helpful, also can you find out it requires libc++?

Will see what I can find.

When do you plan to switch over to Firefox 24?

As soon as possible I think. TBB with Firefox 24 should be ready if the ESR 17 reaches its EOL (which is IIRC in about 7-8 weeks).

I will download your git repo, give it a test and study your logs at lunchtime. You didn't say whether you use the -arch flag or not, but no problem I can take a look.

We use no -arch flags if I see it correctly.

comment:13 Changed 5 years ago by mingwandroid

I've had an initial look and here's what I've gleamed:

There's currently an issue which forces us to need to clean out both the builddir and any existing installation before doing subsequent builds in the same directories. For you this would mean doing:

rm -rf /home/firefox/crosstool-ng/ct-ng-final /home/firefox/x-tools

You're currently basing your build on the darwin10 sample, but actually you'd be better off starting from my darwin11 one (see note [1]) so I'd replace samples/i686-apple-darwin10/crosstool.config with samples/i686-apple-darwin11/crosstool.config and then, to turn it back into a darwin10 config, change the following entries:

CT_DARWIN_MAC_OSX_V_10_7=y
to:
CT_DARWIN_MAC_OSX_V_10_6=y

CT_DARWIN_VERSION="11"
to:
CT_DARWIN_VERSION="10"

CT_DARWIN_SDK_PATH="${HOME}/MacOSX10.7.sdk"
to:
CT_DARWIN_SDK_PATH="${HOME}/MacOSX10.6.sdk"

.. if you want to be able to use the generated toolchains without passing --sysroot $MacOSXSDKDir then change (I don't recommend this - see note [2]) change

CT_DARWIN_COPY_SDK_TO_SYSROOT=n
to:
CT_DARWIN_COPY_SDK_TO_SYSROOT=y

.. depending on whether you would want to debug issues if/when we run into them change (likely worthwhile for now, provided you've got ~20GB to spare)

CT_DEBUGGABLE_TOOLCHAIN=n
to:
CT_DEBUGGABLE_TOOLCHAIN=y

Notes:

[1] I suspect I maintain my darwin11 samples a bit more regularly, whereas Yann generates new ones each time via make menuconfig. Also my samples will always use the latest versions of things that are in a reasonably working state (LLVM and clang 3.3 in this case)

[2] I don't recommend this as it's not a use-case I have verified as working, specifically, AFAIK the frameworks are not copied over so building anything that requires more components than a C and C++ library is likely to fail. Further, toolchains built like this cannot be legally distributed.

Changed 5 years ago by gk

Attachment: cross_mac_build3.tar.bz2 added

comment:14 in reply to:  13 Changed 5 years ago by gk

Replying to mingwandroid:

You're currently basing your build on the darwin10 sample, but actually you'd be better off starting from my darwin11 one (see note [1]) so I'd replace samples/i686-apple-darwin10/crosstool.config with samples/i686-apple-darwin11/crosstool.config and then, to turn it back into a darwin10 config, change the following entries:

I did that but am running in the same error as in cross_mac_build2 (the Makefile.config issue mentioned above). I attached the whole log (cross_mac_build3.tar.bz2) in case it helps.

comment:15 Changed 5 years ago by mingwandroid

Are you running it via gitian now? The reason I ask is that the unpacking stage of the log is full of:

[FILE ] gmp-5.1.1/gen-fac.c
[FILE ] tar: gen-fac.c: time stamp 2013-02-11 15:29:14 is 413911753.757075149 s in the future

.. clearly 2013-02-11 is in the past (given a sane, non-US date ordering at least)

Then the LLVM build itself has lots of:

[ALL ] make[4]: Warning: File `/home/ubuntu/build/crosstool-ng/ct-ng-final/.build/i686-apple-darwin10/build/build-LLVM-host-i686-build_pc-linux-gnu/Makefile.llvmbuild' has modification time 0.81 s in the future
[ALL ] llvm[4]: Regenerating /home/ubuntu/build/crosstool-ng/ct-ng-final/.build/i686-apple-darwin10/build/build-LLVM-host-i686-build_pc-linux-gnu/Makefile.config

.. so it's constantly re-generating the Makefiles and doing other "maintainer mode" build steps that shouldn't need to be done with a released tarball.

comment:16 in reply to:  15 ; Changed 5 years ago by gk

Replying to mingwandroid:

Are you running it via gitian now?

Yes. This bug is about getting everything running in gitian which is currently my main concern (or better my main concern is getting Fx 24 ESR cross-compiled for Mac which is actually #9829). I am happy to test with a clean VM without gitian, though, (to debug other/related issues) but that needs to be done in a different bug (not sure where crosstools-ng has its bugtracker).

comment:17 in reply to:  16 Changed 5 years ago by mingwandroid

Replying to gk:

Replying to mingwandroid:

Are you running it via gitian now?

Yes. This bug is about getting everything running in gitian which is currently my main concern (or better my main concern is getting Fx 24 ESR cross-compiled for Mac which is actually #9829). I am happy to test with a clean VM without gitian, though, (to debug other/related issues) but that needs to be done in a different bug (not sure where crosstools-ng has its bugtracker).

I agree that gitian is the end result here, however having crosstool-ng working outside of gitian is a necessary first step toward that result. If you feel this step merits a ticket of its own then that's fine, please go ahead and create one.

comment:18 Changed 5 years ago by mingwandroid

I am happy if the bug report lives on this tracker, but https://github.com/diorcety/crosstool-ng/issues would arguably be a better place for it. I can try to create one there if you prefer?

comment:19 Changed 5 years ago by mingwandroid

Finally, before creating this bug report it's definitely worth knowing first if there's a failure outside of gitian ;-)

comment:20 in reply to:  18 Changed 5 years ago by gk

Replying to mingwandroid:

I am happy if the bug report lives on this tracker, but https://github.com/diorcety/crosstool-ng/issues would arguably be a better place for it. I can try to create one there if you prefer?

As soon as my build machine is ready again I'll try your suggestions in a clean VM outside of gitian and report an issue in the crosstool-ng bugtracker (if there are any, that is).

comment:21 Changed 5 years ago by mingwandroid

I used:

http://www.google.co.uk/ig/directory?type=gadgets&url=echo3.net/dcalc/gadget.xml

.. and entered "now - 413911773 seconds" and it said:

"Monday, 28-Aug-2000"

gnutar will extract files with the date stamp as embedded in the archives; having these dates 13 years in the future is likely to cause the build failure you are seeing. I think you need to fix your date at "the-date-of-the-most-recent-tarball-that-you-use + 1 second" or something like that. Mike's input on this would be very useful.

Of course if you've got solutions to any of these issues that require options being added to crosstool-ng then I can try to make those changes and get them merged when everything is working.

comment:22 Changed 5 years ago by gk

Okay, in case it helps:

Gitian relies on libfaketime to set the clock to a fixed value to deal with embedded timestamps in archives and in the build process.

See: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details and see: https://github.com/wolfcw/libfaketime/blob/master/README for how libfaketime works (basically via LD_PRELOAD; and we are setting FAKETIME to "2000-01-01 00:00:00")

That said, even if there is a lot of regenerating going on I currently don't see why Makefile.config should be _missing_ because of this. Thus, I am not sure whether dealing with the faketime issue is sufficient to solve this particular problem.

comment:23 in reply to:  22 Changed 5 years ago by gk

Replying to gk:

Okay, in case it helps:

Gitian relies on libfaketime to set the clock to a fixed value to deal with embedded timestamps in archives and in the build process.

See: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details and see: https://github.com/wolfcw/libfaketime/blob/master/README for how libfaketime works (basically via LD_PRELOAD; and we are setting FAKETIME to "2000-01-01 00:00:00")

Forgot to add that we do

find -type f | xargs touch --date="$REFERENCE_DATETIME"

in source directories where "$REFERENCE_DATETIME" is currently "2000-01-01 00:00:00". That might be a thing worth trying regarding the crosstools-ng sources...

comment:24 in reply to:  22 Changed 5 years ago by mingwandroid

Replying to gk:

Okay, in case it helps:

Gitian relies on libfaketime to set the clock to a fixed value to deal with embedded timestamps in archives and in the build process.

See: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details and see: https://github.com/wolfcw/libfaketime/blob/master/README for how libfaketime works (basically via LD_PRELOAD; and we are setting FAKETIME to "2000-01-01 00:00:00")

That said, even if there is a lot of regenerating going on I currently don't see why Makefile.config should be _missing_ because of this. Thus, I am not sure whether dealing with the faketime issue is sufficient to solve this particular problem.

There's continual regeneration of Makefile.config in the presence of recursive make (-j5). So I can easily imagine that at various points Makefile.config does not exist. Removing and re-writing a file is not a multithread-safe atomic operation.

comment:25 Changed 5 years ago by mingwandroid

I added an option for this feature:
From menuconfig's perspective it is called "Date and time to set on all source files"
From config file perspective it's called CT_SRC_REFERENCE_DATETIME.
I also added it to samples/i686-apple-darwin11/crosstool.config as: CT_SRC_REFERENCE_DATETIME="1999-12-31 23:59:59"
It would be good if you could pull the latest, try this out and see where things stand.

comment:26 Changed 5 years ago by mingwandroid

Oh, I set the timestamp on directories not just files in-case any Makefile uses a directory as a dependency and also because it looks more tidy.

comment:27 Changed 5 years ago by gk

Okay. The build failed. See cross_mac_build4.tar.bzip for the full log. Gitian gave me (additionally it seems) the following messages:

[00:00] / 
[INFO ]    Forcing date and time of patched sources to 1999-12-31 23:59:59
[00:00] / touch: cannot touch `space/test.h': Not a directory
touch: cannot touch `space/test.h.result': Not a directory
touch: cannot touch `space/test2.m.in': Not a directory
touch: cannot touch `space/test1.m.in.result': Not a directory
touch: cannot touch `space/test1.m.in': Not a directory
touch: cannot touch `space/test2.m.in.result': Not a directory
[ERROR]  
[00:00] / 
[ERROR]  >>
[00:00] / 
[ERROR]  >>  Build failed in step 'Forcing date and time of patched sources to 1999-12-31 23:59:59'
[00:00] / 
[ERROR]  >>        called in step 'Extracting and patching toolchain components'
[00:00] / 
[ERROR]  >>        called in step '(top-level)'
[00:00] / 
[ERROR]  >>
[00:00] / 
[ERROR]  >>  Error happened in: main[scripts/crosstool-NG.sh@651]

Changed 5 years ago by gk

Attachment: cross_mac_build4.tar.bz2 added

comment:28 Changed 5 years ago by mingwandroid

Ah, I had locally disabled clang from my config, and it the "find | xargs touch" command doesn't correctly handle spaces in paths (which one of the clang test-suite files contains).

I pushed a fix for this now using:

find .. -exec touch --date= .. "{}" \;

.. instead.

comment:29 Changed 5 years ago by gk

Looks good. Now, the gitian build process is stuck with https://github.com/diorcety/crosstool-ng/issues/1 as well.

comment:30 Changed 5 years ago by mingwandroid

This was down to a bash-ism that crept into the cctools patch-set. It should be fixed now. Please update and re-test.

comment:31 Changed 5 years ago by mingwandroid

I made a test set of binaries (outside of gitian though) on a clean Ubuntu 12.04.3 VM:
https://www.dropbox.com/s/k5s09xhqv3mv2ti/i686-apple-darwin10.tar.xz

.. the exact config I used was:
https://www.dropbox.com/s/q9z66w119fvhwd2/crosstool.config

I'm now giving gitian a go. First thing I noticed is that:
sudo torsocks apt-get install

.. is very noisy:

07:54:57 libtorsocks(933): The symbol res_query() was not found in any shared library. The error reported was: not found!
07:54:57 libtorsocks(933): The symbol res_search() was not found in any shared library. The error reported was: not found!
07:54:57 libtorsocks(933): The symbol res_send() was not found in any shared library. The error reported was: not found!
07:54:57 libtorsocks(933): The symbol res_querydomain() was not found in any shared library. The error reported was: not found!

.. should I be concerned about this?

Also, you've been trying to build crosstool-ng under gitian, is that just a question of prepending "sudo torsocks" in front of all the crosstool-ng build commands? If you have a new script that you can commit then that would be good.

comment:32 in reply to:  31 Changed 5 years ago by gk

Replying to mingwandroid:

I made a test set of binaries (outside of gitian though) on a clean Ubuntu 12.04.3 VM:
https://www.dropbox.com/s/k5s09xhqv3mv2ti/i686-apple-darwin10.tar.xz

.. the exact config I used was:
https://www.dropbox.com/s/q9z66w119fvhwd2/crosstool.config

I'm now giving gitian a go. First thing I noticed is that:
sudo torsocks apt-get install

.. is very noisy:

07:54:57 libtorsocks(933): The symbol res_query() was not found in any shared library. The error reported was: not found!
07:54:57 libtorsocks(933): The symbol res_search() was not found in any shared library. The error reported was: not found!
07:54:57 libtorsocks(933): The symbol res_send() was not found in any shared library. The error reported was: not found!
07:54:57 libtorsocks(933): The symbol res_querydomain() was not found in any shared library. The error reported was: not found!

.. should I be concerned about this?

No.

Also, you've been trying to build crosstool-ng under gitian, is that just a question of prepending "sudo torsocks" in front of all the crosstool-ng build commands?

No. 'torsocks' is just used to download the dependencies via Tor.

If you have a new script that you can commit then that would be good.

Let me know when you are ready (i.e. the VMs are up and you built a Mac bundle successfully). I'll send you a patch which you can apply on top of your local branch and which should give you the environment I am currently using.

That said, I've been running now into three different issues which are likely to due to the gitian build process: 1) the linker got killed 2) not enough space on the VM 3) another build failure for yet unknown reasons. I fixed 1) and 2) and am currently debugging 3).

But your latest two commits fixing bashisms seemed to work. I'll test that on my Ubuntu VM again as soon as my build machine is not occupied with gitian build stuff and will comment on the github bugtracker then.

comment:33 Changed 5 years ago by mingwandroid

Can I run the VM from within a VM?

It was last night when I last tried but I remember getting something about needing to do:

export USE_LXC=1

.. because AFAIR it said VM-in-a-VM wouldn't work so to do this instead? Is that the case? Are there some settings I can apply to make VM-in-a-VM work?

comment:34 Changed 5 years ago by gk

Good questions. But, honestly, I don't know the answers to them. I never tried that setup as building without nested VMs is already slow enough for me :)

comment:35 Changed 5 years ago by gk

You probably hit the KVM check telling you something like

Most likely, this means you will need to use LXC.

Please run this in your shell before each build:
 export USE_LXC=1

LXC mode is currently kind of highly experimental and presumably broken on trunk. Your better bet is getting KVM running inside your VM.

Last edited 5 years ago by gk (previous) (diff)

comment:36 Changed 5 years ago by gk

It seems the VM gitian creates by default is not big enough, *sigh*. Trying a larger one...

comment:37 Changed 5 years ago by mingwandroid

CT_DEBUGGABLE_TOOLCHAIN=n results in a much reduced space requirement.

comment:38 in reply to:  37 Changed 5 years ago by gk

Replying to mingwandroid:

CT_DEBUGGABLE_TOOLCHAIN=n results in a much reduced space requirement.

Yes, that is how I fixed 2) in comment 32 but it was not enough to solve 3)...

comment:39 Changed 5 years ago by mingwandroid

Trying with a native Ubuntu 12.04.3 now, I'm getting:

Formatting 'target-lucid-i386.qcow2', fmt=qcow2 size=11811160064 backing_file='base-lucid-i386.qcow2' encryption=off cluster_size=65536
ssh_exchange_identification: read: Connection reset by peer
ssh: connect to host localhost port 2223: Connection refused
ssh: connect to host localhost port 2223: Connection refused

Lucid i386 VM build failed... Trying again

I will let it try again then I'll try "make vmclean && make build".

Let me know when you are ready (i.e. the VMs are up and you built a Mac bundle successfully). I'll send you a patch which you can apply on top of your local branch and which should give you the environment I am currently using.

Although I've not been able to make a build yet I think having your patch to hand would be good.

comment:40 Changed 5 years ago by gk

The build is still failing although after 4 hours of compiling I feel I am close. Seems to be some files are missing (although the first error is probably not the culprit here) (see cross_mac_gcc_errors.tar.bz2).

Some things to note:
1) I still hit one error of type 3) in comment 4. Thus, your FAKETIME fix probably needs to be tuned (somehow) a bit more.
2) I got another presumably related thing tonight: all 4 CPUs run with 100% for hours but nothing got compiled (I had to abort this one)
3) Not sure if

CT_DEBUGGABLE_TOOLCHAIN=n

is really working for me as I am already running my builds with it but I had to bump the size of the VM already twice. I am now at 32GB and the latest build was already at 26GB when it was failing.
4) The gitian build log gives me hard to read output of the whole compiling process :)

[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] - 
[00:00] - 
[00:00] - 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[00:00] / 
[EXTRA]    Retrieving 'llvm-3.3.src'

That happens only when using crosstools-ng (and the log output of the actual compiling process looks similar). Not sure if there is anything you can do about, though. But it would be nice if this could be fixed.

That said, I'll attach a patch set with instructions on how to use it with your local gitian environment later today.

Changed 5 years ago by gk

comment:41 Changed 5 years ago by mingwandroid

I imagine I can turn off the /-\ animation for you easily enough; investigating ..

Please do attach the patch. Also, I emailed you & Mike 3 patches I had to apply to get the toolchain4-based tbb-mac to build. If you can consider them I'd appreciate it.

comment:42 Changed 5 years ago by mingwandroid

If you add:
CT_LOG_PROGRESS_BAR=n
.. to your crosstool.config file then the logs should be cleaner.

comment:43 in reply to:  42 Changed 5 years ago by gk

Replying to mingwandroid:

If you add:
CT_LOG_PROGRESS_BAR=n
.. to your crosstool.config file then the logs should be cleaner.

Thanks, I'll try that the next time.

Attached is the patch. It is against fc14747c24c87085f6ca502d1643dd2e7a0ea7a0. Additional things you need to to:
1) put your crosstool.config file into gitian-builder/inputs
2) run fetch-inputs.sh again
3) in gitian-builder/inputs/crosstoll-ng you need to checkout the revision you want and create a tag (this is currently 'test5' which is mirrored in gitian/versions
4) You need to raise the max_size of the precise VM. I am currently trying 32GB (32768 as value to --rootsize). Maybe this is enough (I have debug output in the crosstool.config disabled). You do that with gitian-builder/bin/make-base-vm and the --rootsize paramter in line 76.
5) Now, remove both *precise*qcow2 files in gitian-builder
6) Remove the tor-browser-mac32-gbuilt.zip in gitian-builder/inputs
7) Start the build with

./mkbundle-mac.sh

again.

Hope I haven't forgotten something...

Thanks for the patches I'll take a look at them.

Changed 5 years ago by gk

comment:44 Changed 5 years ago by gk

Attached a new log that actually fails at the same place but slightly differently. Might shed some light on the issue at hand.

Changed 5 years ago by gk

comment:45 Changed 5 years ago by mingwandroid

From ../../gitian-builder/var/build.log :

+ sudo dpkg -i apple-uni-sdk-10.6_20110407-0.flosoft1_i386.deb
sudo: no tty present and no askpass program specified
Sorry, try again.

(I tried entering my login password but I'm guessing there's some other password that we need.. does the gitian VM not just do everything as root anyway?)

comment:46 Changed 5 years ago by mingwandroid

Ok I managed to work around this, but the things I had to do were horrible:

I decided that I needed my (inside the VM) user to have sudoer access without passwords because the actual build commands are run with redirection for logging so the can't use the tty to ask for passwords anyway.

In order to be able to edit the /etc/sudoers in the VM I had to:

  1. Patch gitian-builder with 0002-use-ssh-t-for-commands-run-on-target-via-sudo.patch (attached - please read the comments on this it is likely to break some other things, basically, if sudo appears in $* then I add -t to the ssh command).
  1. Then with this done, run on-target sudo nano /etc/sudoers to add "Defaults:ubuntu !authenticate" to it.

However, instead of trying to install the flosoft MacOSX SDK .deb file at build time, I think the following options are better:

  1. Install the .deb at target VM creation time (as I'm guessing other packages are installed?)
  2. If we must unpack the SDK at build time then don't use the .deb, instead use https://launchpad.net/~flosoft/+archive/cross-apple/+files/apple-uni-sdk-10.6_20110407.orig.tar.gz and untar it.

Also, I've seen various references to /usr/lib/i686-apple-darwin*, I think these are leftovers from before Mike switch to using toolchain4. These toolchains are fully relocatable so none of this should be needed.

comment:47 Changed 5 years ago by mingwandroid

Well, ignore all of that, somehow I'd messed up my sshd config locally so while I though I was connecting to the VM, I was actually connecting to my host machine!

It is building away now.

comment:48 Changed 5 years ago by mingwandroid

Good news! Building at -j1 allowed me to complete a gitian build of crosstool-ng.

To force this, in crosstool.config you need to add:
CT_PARALLEL_JOBS=1

It seems a shame to forgo parallelism since everything built ok (to the best of my knowledge at least) up to the last thing, which is actually the legacy GCC 4.2.1, not clang. I will compare build artefacts to try to isolate what exactly causes this. FWIW, gnumake outputs warnings:

make[5]: warning: Clock skew detected. Your build may be incomplete.

.. forcing a specific date then generating tools (gengtype) that generate build artefacts that (Makefile-wise) depend on the build tool that created them is in the realm of unspecified behaviour I expect.

comment:49 Changed 5 years ago by mingwandroid

While researching this a more, I read:

https://github.com/wolfcw/libfaketime/blob/master/README

Could you check the bits relating to libfaketimeMT.so.1? It seems to me that this may be safer to use, but given our use pattern (basically always return a constant value), likely it is a red herring, also, I don't think gnumake is multithread (sure pthreads and fork use sys_clone at the lowest level, but when forking no MT protection would be required. I'm not sure why gnumake links to pthreads to be honest). It's just a thought and I will do a test build later to see if it makes any difference.

I've also started to examine the behaviour of gnumake more closely as I've noticed that during the build, a file that should be built only once is getting built 8 times (and overall, the log from a gitian build is about 2* the length of one from a non-gitian build) From my initial prodding, it seems that somehow make has an un-faked timestamp for things it has just created, but this could be down to a problem with my test-case setup and that wouldn't cause the 8 times rebuild either.

comment:50 in reply to:  48 Changed 5 years ago by gk

Replying to mingwandroid:

Good news! Building at -j1 allowed me to complete a gitian build of crosstool-ng.

To force this, in crosstool.config you need to add:
CT_PARALLEL_JOBS=1

Nice! Yes, avoiding parallel jobs might make sense given the log output. I'll test that idea later on my gitian setup.

comment:51 Changed 5 years ago by mingwandroid

I've concluded that libfaketime is somewhat broken for the use case that gitian build requires.

Some problems I've seen include:

  1. The fake time values are not setup correctly for early calls (at least in the MT version any that happen between semaphore creation and the getenv("FAKETIME")/getenv("FAKETIME_FMT") calls).
  2. Caching functionality is broken; it doesn't respect the "FAKETIME" env. var, merrily replacing it with "+0" instead as soon as it decides that the cached values are too old. This means that long running processes stop faking time properly early in their execution.
  3. nano-second values are never touched for the stat family of functions - or likely any that call fake_time - and gnumake uses nanoseconds for dependency checking. Without fixing this, the dependency timestamp check could be replaced with "rand()&1" (Mike's blog post mentions this in point 4 of "Remaining Build Reproducibility Issues [Millisecond and below timestamps are not fixed by libfaketime]")

I've got a messy patch for these issues that I can share?

Also, libfaketime doesn't actually effect the real filestamps as written to any files, it only intercepts attempts to query these values.

Last edited 5 years ago by mingwandroid (previous) (diff)

comment:52 in reply to:  51 Changed 5 years ago by gk

Replying to mingwandroid:

I've concluded that libfaketime is somewhat broken for the use case that gitian build requires.

Some problems I've seen include:

  1. The fake time values are not setup correctly for early calls (at least in the MT version any that happen between semaphore creation and the getenv("FAKETIME")/getenv("FAKETIME_FMT") calls).
  2. Caching functionality is broken; it doesn't respect the "FAKETIME" env. var, merrily replacing it with "+0" instead as soon as it decides that the cached values are too old. This means that long running processes stop faking time properly early in their execution.
  3. nano-second values are never touched for the stat family of functions - or likely any that call fake_time - and gnumake uses nanoseconds for dependency checking. Without fixing this, the dependency timestamp check could be replaced with "rand()&1" (Mike's blog post mentions this in point 4 of "Remaining Build Reproducibility Issues [Millisecond and below timestamps are not fixed by libfaketime]")

I've got a messy patch for these issues that I can share?

Sure. Best would probably be to file these issues upstream (or check whether they are already there). I can do both. Now, what are our options here? Accept building with

CT_PARALLEL_JOBS=1

until these issues are fixed? Trying to get the clang compiler working (which we probably need to to anyway, see #9829) and hoping we are getting away with that?

comment:53 Changed 5 years ago by mingwandroid

I forked and committed 4 patches.

https://github.com/mingwandroid/libfaketime

.. and made a pull request to upstream:

https://github.com/wolfcw/libfaketime/pull/34

Yes for now I recommend sticking with -j1, and in-fact even though I think these are 'good' fixes within the remit of libfaketime, I suspect that they may turn occasional failure in gitian into deterministic failure!

.. this is just speculation at this point (it could actually make things work with -jN). To know for sure, and so that I can apply fixes needed elsewhere in the build system I need to build a .deb for libfaketime with my patches and roll that into the gitian descriptors. My .deb building skills are non-existent so I'd appreciate help with that.

comment:54 Changed 5 years ago by mingwandroid

Here's a new faketime debian package I built with my fixes:

https://www.dropbox.com/s/tm9n9zimmhsx474/faketime_0.9.6_i386.deb

sha1sum faketime_0.9.6_i386.deb
59e66e537c6770aed0862fbe2e479d60f28d5e1d faketime_0.9.6_i386.deb

.. I will try to make the changes needed to make use of it. I am a bit confused about that though, I could make fetch-inputs.sh download it from my dropbox and then install it, but I wondered if there was a way to place it in the base VM images? Also, is Vagrant used?

Changed 5 years ago by mingwandroid

Patch for tor-browser-bundle to use my faketime 0.9.6 test package

comment:55 Changed 5 years ago by mingwandroid

If you want to try my faketime then apply 0001-Use-my-faketime_0.9.6-test-package.patch​

It should be applied after your patch, but there's some intermediate patches I've not included (just debugging info) so apply this with "patch -p1 < 0001-Use-my-faketime_0.9.6-test-package.patch".

comment:56 in reply to:  54 Changed 5 years ago by gk

Replying to mingwandroid:

Here's a new faketime debian package I built with my fixes:

https://www.dropbox.com/s/tm9n9zimmhsx474/faketime_0.9.6_i386.deb

sha1sum faketime_0.9.6_i386.deb
59e66e537c6770aed0862fbe2e479d60f28d5e1d faketime_0.9.6_i386.deb

.. I will try to make the changes needed to make use of it. I am a bit confused about that though, I could make fetch-inputs.sh download it from my dropbox and then install it, but I wondered if there was a way to place it in the base VM images?

Your .deb is introducing the GMP failures for me again (see comment 6 case 1)) while this is not the case with the unpatched libfaketime anymore. I don't know why this is happening yet. What I did was downloading, verifying and copying your version to gitian-builder/inputs. Then I changed gitian-firefox.yml: 1) I removed faketime in the packages section and added your .deb to the files section 2) I put

sudo dpkg -i faketime_0.9.6_i386.deb

at the first place in the script section. Then I restarted the build. Not sure why you want to have your patched faketime package in the base VM. That shouldn't buy you anything.

Also, is Vagrant used?

Building with it should be possible again with the latest trunk if that is what you mean.

comment:57 Changed 5 years ago by mingwandroid

I was more wondering out loud what Vagrant was about, I guess it's a VirtualBox based replacement for the KVM based system? Is that roughly correct?

Your .deb is introducing the GMP failures for me again (see comment 6 case 1)) while this is not the case with the unpatched libfaketime anymore. I don't know why this is happening yet.

I had this happen too. So this comes down to the fact that configure checks to see if "the build system is sane" where by sane it means that a file it just created is newer than a file from the $srcdir (it does this by using ls -t and checking that the first line is the new file). When using a libfaketime that correctly freezes time (including nanoseconds), this is not the case. This is what I meant when I said:

even though I think these are 'good' fixes within the remit of libfaketime, I suspect that they may turn occasional failure in gitian into deterministic failure!

.. I need to figure out a good way to fix this; replacing all instances of "ls -t" in configure* before performing the crosstool-ng build is an option here. I'd like to get Mike's take on this. FWIW, I had comments back from the maintainer of libfaketime and he's going to implement nanosecond freezing (as opposed to my nanosecond zero'ing).

Having said all this, we need to get ESR 24 to build so I'm proposing to move onto concentrating on that instead.

comment:58 Changed 5 years ago by mingwandroid

.. or rather, getting clang to build either ESR 17 or ESR 24.

comment:59 in reply to:  57 Changed 5 years ago by gk

Replying to mingwandroid:

I was more wondering out loud what Vagrant was about, I guess it's a VirtualBox based replacement for the KVM based system? Is that roughly correct?

An alternative, yes.

Having said all this, we need to get ESR 24 to build so I'm proposing to move onto concentrating on that instead.

Yeah, that's pretty important. See my last mail and #9829 as the new playground ;)

BTW: Building with

CT_PARALLEL_JOBS=1

worked for me as well. Thanks! At least one option to build the compiler for ESR 17 from source...

comment:60 Changed 5 years ago by gk

I can't build the toolchain anymore it seems. At least not with gitian. See the attached log (gcc_build_error.log). Can you verify that? I am basically using crosstool.config_real3 (found in #9829) and rev 9a711e316a9c374f815c3d018dd2614fea2382d5.

Last edited 5 years ago by gk (previous) (diff)

Changed 5 years ago by gk

Attachment: gcc_build_error.log added

comment:61 Changed 5 years ago by mingwandroid

I've built that revision with gitian.

I suspect that the problem is to do with libfaketime and the random nano-second values, meaning that the determinism we're looking for is not there yet.

I think that the libfaketime guys are going to fix the nano-second values bug but that will make builds fail deterministically ( I speculated about a potential fix in https://trac.torproject.org/projects/tor/ticket/9711#comment:59 )

The other option is to not build GCC at all; we should not need it as we're using clang.

comment:62 in reply to:  61 ; Changed 5 years ago by gk

Replying to mingwandroid:

The other option is to not build GCC at all; we should not need it as we're using clang.

That's a good idea for a couple of reasons. How do I do that?

comment:63 in reply to:  62 ; Changed 5 years ago by gk

Replying to gk:

Replying to mingwandroid:

The other option is to not build GCC at all; we should not need it as we're using clang.

That's a good idea for a couple of reasons. How do I do that?

Hmm.. I'll probably start with

CT_CC_GCC_V_apple_5666_3=n

comment:64 in reply to:  63 Changed 5 years ago by gk

Replying to gk:

Replying to gk:

Replying to mingwandroid:

The other option is to not build GCC at all; we should not need it as we're using clang.

That's a good idea for a couple of reasons. How do I do that?

Hmm.. I'll probably start with

CT_CC_GCC_V_apple_5666_3=n

But that is still downloading GCC and patching it...

Last edited 5 years ago by gk (previous) (diff)

comment:65 Changed 5 years ago by mingwandroid

You may also need

CT_CC=n
CT_CC_gcc=n
CT_CC_GCC_V_apple_5666_3=y

.. in my test builds, I post-process the .config file with sed before calling "ct-ng build" as follows:

# Wishing to iterate quickly on clang/llvm!
$SED -i "s/CT_CC=\"gcc\"/#CT_CC=\"gcc\"/g" ./.config
$SED -i "s#CT_CC_gcc=y#CT_CC_gcc=n#g" ./.config
$SED -i "s#CT_CC_GCC_V_apple_5666_3=y#CT_CC_GCC_V_apple_5666_3=n#g" ./.config

.. but the first option is definitely nicer if it works!

comment:66 in reply to:  65 Changed 5 years ago by gk

Replying to mingwandroid:

You may also need

CT_CC=n
CT_CC_gcc=n
CT_CC_GCC_V_apple_5666_3=y

That did not work (even with CT_CC_GCC_V_apple_5666_3=n ;))

.. in my test builds, I post-process the .config file with sed before calling "ct-ng build" as follows:

# Wishing to iterate quickly on clang/llvm!
$SED -i "s/CT_CC=\"gcc\"/#CT_CC=\"gcc\"/g" ./.config
$SED -i "s#CT_CC_gcc=y#CT_CC_gcc=n#g" ./.config
$SED -i "s#CT_CC_GCC_V_apple_5666_3=y#CT_CC_GCC_V_apple_5666_3=n#g" ./.config

Will try that one but probably not until #9829 is fixed.

comment:67 Changed 5 years ago by gk

Status: needs_informationnew

comment:68 Changed 5 years ago by gk

Note: work on this bug depends on fixing #11459.

comment:69 Changed 5 years ago by gk

Cc: gk added; g.koppen@… removed
Keywords: tbb-3.0 removed

comment:70 Changed 4 years ago by erinn

Component: Tor bundles/installationTor Browser
Keywords: tbb-gitian added
Owner: changed from erinn to tbb-team

comment:71 Changed 4 years ago by gk

mingwandroid: which branch is the one we should choose nowadays if we want to move this forward?

comment:72 Changed 4 years ago by mingwandroid

I'll check where I left things at and get back to you tonight.

comment:73 Changed 4 years ago by mingwandroid

This is the build procedure I had success with just now:

untar your Mac OSX 10.6 SDK to $HOME (you should have $HOME/MacOSX10.6.sdk - if it's one with broken absolute links like some are you'll need to fix those links, but the one you've been using in the past should be fine).
# clone (master branch is fine).
git clone https://github.com/diorcety/crosstool-ng.git
cd crosstool-ng
# Default prefix is /usr/local, pass --prefix=<elsewhere> if you want to install it elsewhere.
./bootstrap && ./configure && make
sudo make install
mkdir build
cd build
/usr/local/bin/ct-ng x86_64-apple-darwin10
/usr/local/bin/ct-ng build
# Wait an hour or two. Libfaketime can make this several hours!
# The cross compilers should be installed to $HOME/x-tools/x86_64-apple-darwin10/

Regarding libfaketime, I ran into an infinite loop bug after I corrected the bug where it didn't change the nanosecond field of the time structure (which I'm guessing upstream have since fixed too), and also an autotools issue where it reports "checking whether build environment is sane .. no". This mail from Eric Blake to the GNU make bugs mailing list touches on both these subjects from the Austin Group's perspective (though not directly) http://lists.gnu.org/archive/html/bug-make/2014-08/msg00044.html. I don't think fixing these issues would be too much effort, but unless they are fixed I'd recommend a different approach to deterministic builds involving post processing object files instead.

comment:74 Changed 3 years ago by gk

Keywords: GeorgKoppen201506 added

comment:75 Changed 3 years ago by gk

Status: newneeds_information

mingwandroid: Is there a way to avoid downloading all the dependencies every time we build the cross-compiler? Can I somehow point crosstools-ng to already downloaded sources and convince it to use those instead? Ideally, there should be no need for a network connection at all in the build phase (after fetching all the sources + installing the necessary packages).

comment:76 Changed 3 years ago by mingwandroid

gk: Yeah there is. By default it's: CT_LOCAL_TARBALLS_DIR="${HOME}/src". If using the menuconfig then it's called "Local tarballs directory".

If you set CT_SAVE_TARBALLS=y then new ones needed are downloaded and cached there too. If you know ahead of time what tarballs are needed then just place them in CT_LOCAL_TARBALLS_DIR.

I've recently started working on crosstool-ng again. My goal is to upstream all of my work. The new maintainer of crosstool-ng is very interested in this work which helps tremendously.

comment:77 Changed 3 years ago by gk

Severity: Normal

hi mingwandroid: it might be time to fix this bug once and for all. We need a new toolchain due to #18331. Is it possible with your current setup to get a new cross-compiler using clang >= 3.7? Or do we have to tweak things in a non-trivial way? It seems we can workaround the breakage visible in #18331 by backing out the responsible ICU commit but still needing to do things like that makes me pretty nervous.

comment:78 Changed 3 years ago by arlolra

Cc: arlolra added

comment:79 Changed 3 years ago by boklm

Cc: boklm added

comment:80 Changed 15 months ago by gk

Keywords: gitian removed
Summary: Test out crosstools-ng for Gitian OSX builds (instead of toolchain4 binaries)Test out crosstools-ng for rbm/tor-browser-build OSX builds (instead of toolchain4 binaries)

comment:81 Changed 15 months ago by cypherpunks

Keywords: tbb-rbm added; tbb-gitian removed

comment:82 Changed 7 months ago by gk

Mozilla switched away from crosstools-ng to cctools-port (https://github.com/tpoechtrager/cctools-port) after ESR52. Seems worthwhile to follow that step just to keep the toolchains in sync as good as possible. (See: https://bugzilla.mozilla.org/show_bug.cgi?id=1331957#c11 for the reasoning).

comment:83 Changed 6 months ago by gk

Keywords: GeorgKoppen201805 added; GeorgKoppen201506 removed
Parent ID: #24631
Status: needs_informationassigned

comment:84 Changed 6 months ago by gk

Keywords: TorBrowserTeam201805 added
Parent ID: #24631#24632
Priority: MediumVery High

comment:85 Changed 6 months ago by gk

Keywords: TorBrowserTeam201805R added; TorBrowserTeam201805 removed
Status: assignedneeds_review

See: bug_9711_v2 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_9711_v2&id=94b31b1822961b25fec66078bcf48a279d6e9aa6) for a patch for review. (This can land as well as the Rust macOS compiler patch did before we switch the whole toolchain over).

Last edited 6 months ago by gk (previous) (diff)

comment:86 Changed 6 months ago by boklm

Resolution: fixed
Status: needs_reviewclosed
Summary: Test out crosstools-ng for rbm/tor-browser-build OSX builds (instead of toolchain4 binaries)Build our own cctools for macOS cross-compilation

This looks good to me. I merged commit 94b31b1822961b25fec66078bcf48a279d6e9aa6 to master.

Note: See TracTickets for help on using tickets.