Ship tor in Tor Browser nightly builds on Linux with Rust enabled
We should test tor with Rust enabled in our Linux nightly builds.
Activity
bug_25481_v2(https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_25481_v2&id=1f7c6f2ae24c9e79382bd1fdf380bbceac4e4d1c) has a patch for this bug up for review. Some notes might be in order, though:- Rust needs CMake 3.4.3 at least + LLVM 3.9, and some compiler that supports C++-11. It turns out we want the first two anyway during the update of our macOS toolchain and g++4.7 is the minimum requirement outlined on the rust-lang website. So, we should be good, right? Alas, it turns our that's not the case. For one, not all parts we need to build are happy with g++ 4.7.2 which Wheezy ships, so we need our own compiler. But, two, even salvaging some of our LLVM build efforts are failing: using LLVM 3.9 crashes the compiler during the cargo build step. See e.g.:
process didn't exit successfully: `/var/tmp/build/rustc-1.25.0-src/build/build/bootstrap/debug/rustc --crate-name lazycell vendor/lazycell/src/lib.rs --crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=1c971c2102b3dfa3 -C extra-filename=-1c971c2102b3dfa3 --out-dir /var/tmp/build/rustc-1.25.0-src/build/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/var/tmp/build/rustc-1.25.0-src/build/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps -L dependency=/var/tmp/build/rustc-1.25.0-src/build/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)So, I worked around that by building LLVM during the rust compilation (which make the whole process taking significantly more time) using that version in turn.
-
The current setup is building basically all the tools which come with the
rustcsource tarball. We might only needrustcandcargo, and one way to achieve that might be to split both and build them in separate projects. I am fine if we want to pursue this option later on after we got the basics working. -
There is a bunch of documentation that is getting built during normal compilation. I guess we can optimize that away later on and find other knobs to produce a smaller toolchain faster.
Trac:
Keywords: TorBrowserTeam201804 deleted, TorBrowserTeam201804R added
Replying to gk:
So, I worked around that by building LLVM during the rust compilation (which make the whole process taking significantly more time) using that version in turn.
I don't see LLVM being built in your patch. Is it done internally by the rust build system?
The extracting of gcc and update of the PATH/LD_LIBRARY_PATH could be replaced by:
[% pc('gcc', 'var/setup', { compiler_tarfile => c('input_files_by_name/gcc') }) %]However it will also set hardened-cc links. Is it because the build doesn't work with hardened-cc that you didn't use it?
For the
ln -s gcc cclink, I am wondering if we should add it directly inprojects/gcc/build, or if there are cases where we might not want this link.In
projects/tor/build, I think the lines after theIF c("var/linux") && c("var/nightly")can be indented.Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201804R deleted, TorBrowserTeam201804 added-
Replying to boklm:
For the
ln -s gcc cclink, I am wondering if we should add it directly inprojects/gcc/build, or if there are cases where we might not want this link.If we add the
cc -> gcclink directly inprojects/gcc/build, we can remove the same link created in the windows build inprojects/firefox/build. -
Replying to boklm:
Replying to boklm:
For the
ln -s gcc cclink, I am wondering if we should add it directly inprojects/gcc/build, or if there are cases where we might not want this link.If we add the
cc -> gcclink directly inprojects/gcc/build, we can remove the same link created in the windows build inprojects/firefox/build.I have not looked at the Windows part for the Firefox build but I have a better fix for the Rust build and think we should use that one.
-
Replying to boklm:
Replying to gk:
So, I worked around that by building LLVM during the rust compilation (which make the whole process taking significantly more time) using that version in turn.
I don't see LLVM being built in your patch. Is it done internally by the rust build system?
Yes, it uses the LLVM code that is shipped with the Rust source.
The extracting of gcc and update of the PATH/LD_LIBRARY_PATH could be replaced by: {{{ [% pc('gcc', 'var/setup', { compiler_tarfile => c('input_files_by_name/gcc') }) %] }}} However it will also set hardened-cc links. Is it because the build doesn't work with hardened-cc that you didn't use it?
No, I originally wanted to avoid using our own GCC as this adds complexity but piece by piece I realized that this is not realistic and added workarounds which resulted in the final patch, forgetting about this macro. This is fixed now.
For the
ln -s gcc cclink, I am wondering if we should add it directly inprojects/gcc/build, or if there are cases where we might not want this link.I can avoid this clumsy symlink by using a configure option which seems neat and which we probably need to use anyway when actually cross-compiling. So, I think we should go with that.
In
projects/tor/build, I think the lines after theIF c("var/linux") && c("var/nightly")can be indented.Fixed.
The updated patch is at
bug_25481_v4(https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_25481_v4&id=fde93d07ca425ee884930cf01fd435b31ace48b8) in my publictor-browser-buildrepo. Note: I tested the x86_64 tor and it is running fine for me. I even checked whether tor built from the same revision twice is identical, and it is! Moreover, I verified that the hardening properties on the tor binary are still the ones we want to have (with the checksec script). So, we are good here. I guess the 32bit tor could get tested, though, which I did not do yet.Trac:
Status: needs_revision to needs_review
Keywords: TorBrowserTeam201804 deleted, TorBrowserTeam201804R added -
One small thing: the
prev_versionvariable inprojects/rust/configshould be namedvar/prev_versioninstead, as it is not a variable that has a specific meaning for rbm.Other than this the patch looks good to me.
I tried a 32bit build of Tor, ran checksec on it, and have been able to bootstrap with it.
Trac:
Status: needs_review to needs_revision -
Alright! How does
bug_25481_v5(https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_25481_v5) look like?Trac:
Status: needs_revision to needs_review mentioned in issue #25779 (moved)
mentioned in issue #25849 (moved)