Opened 8 months ago

Closed 7 months ago

#25481 closed task (fixed)

Ship tor in Tor Browser nightly builds on Linux with Rust enabled

Reported by: gk Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, GeorgKoppen201804, boklm201804, TorBrowserTeam201804R
Cc: Actual Points:
Parent ID: #25220 Points:
Reviewer: Sponsor:

Description

We should test tor with Rust enabled in our Linux nightly builds.

Child Tickets

Change History (15)

comment:1 Changed 8 months ago by gk

Keywords: TorBrowserTeam201803 added; TorBrowserTeam201803. removed

comment:2 Changed 8 months ago by gk

Keywords: GeorgKoppen201804 added; GeorgKoppen201803 removed

Moving my tickets to April 2018

comment:3 Changed 8 months ago by boklm

Keywords: boklm201804 added; boklm201803 removed

boklm201803 -> boklm201804

comment:4 Changed 7 months ago by gk

Keywords: TorBrowserTeam201804 added; TorBrowserTeam201803 removed

Moving our tickets to April.

comment:5 Changed 7 months ago by gk

Priority: MediumHigh

comment:6 Changed 7 months ago by gk

Keywords: TorBrowserTeam201804R added; TorBrowserTeam201804 removed

bug_25481_v2 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_25481_v2&id=1f7c6f2ae24c9e79382bd1fdf380bbceac4e4d1c) has a patch for this bug up for review. Some notes might be in order, though:

1) Rust needs CMake 3.4.3 at least + LLVM 3.9, and some compiler that supports C++-11. It turns out we want the first two anyway during the update of our macOS toolchain and g++4.7 is the minimum requirement outlined on the rust-lang website. So, we should be good, right? Alas, it turns our that's not the case. For one, not all parts we need to build are happy with g++ 4.7.2 which Wheezy ships, so we need our own compiler. But, two, even salvaging some of our LLVM build efforts are failing: using LLVM 3.9 crashes the compiler during the cargo build step. See e.g.:

  process didn't exit successfully: `/var/tmp/build/rustc-1.25.0-src/build/build/bootstrap/debug/rustc --crate-name lazycell vendor/lazycell/src/lib.rs --crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=1c971c2102b3dfa3 -C extra-filename=-1c971c2102b3dfa3 --out-dir /var/tmp/build/rustc-1.25.0-src/build/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/var/tmp/build/rustc-1.25.0-src/build/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/deps -L dependency=/var/tmp/build/rustc-1.25.0-src/build/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)

So, I worked around that by building LLVM during the rust compilation (which make the whole process taking significantly more time) using that version in turn.

2) The current setup is building basically all the tools which come with the rustc source tarball. We might only need rustc and cargo, and one way to achieve that might be to split both and build them in separate projects. I am fine if we want to pursue this option later on after we got the basics working.

3) There is a bunch of documentation that is getting built during normal compilation. I guess we can optimize that away later on and find other knobs to produce a smaller toolchain faster.

comment:7 Changed 7 months ago by gk

Status: newneeds_review

comment:8 Changed 7 months ago by cypherpunks

Maybe, it's time to get rid of Wheezy as its EOL is May 2018.

comment:9 in reply to:  6 ; Changed 7 months ago by boklm

Keywords: TorBrowserTeam201804 added; TorBrowserTeam201804R removed
Status: needs_reviewneeds_revision

Replying to gk:

So, I worked around that by building LLVM during the rust compilation (which make the whole process taking significantly more time) using that version in turn.

I don't see LLVM being built in your patch. Is it done internally by the rust build system?

The extracting of gcc and update of the PATH/LD_LIBRARY_PATH could be replaced by:

[% pc('gcc', 'var/setup', { compiler_tarfile => c('input_files_by_name/gcc') }) %]

However it will also set hardened-cc links. Is it because the build doesn't work with hardened-cc that you didn't use it?

For the ln -s gcc cc link, I am wondering if we should add it directly in projects/gcc/build, or if there are cases where we might not want this link.

In projects/tor/build, I think the lines after the IF c("var/linux") && c("var/nightly") can be indented.

comment:10 in reply to:  9 ; Changed 7 months ago by boklm

Replying to boklm:

For the ln -s gcc cc link, I am wondering if we should add it directly in projects/gcc/build, or if there are cases where we might not want this link.

If we add the cc -> gcc link directly in projects/gcc/build, we can remove the same link created in the windows build in projects/firefox/build.

comment:11 in reply to:  10 Changed 7 months ago by gk

Replying to boklm:

Replying to boklm:

For the ln -s gcc cc link, I am wondering if we should add it directly in projects/gcc/build, or if there are cases where we might not want this link.

If we add the cc -> gcc link directly in projects/gcc/build, we can remove the same link created in the windows build in projects/firefox/build.

I have not looked at the Windows part for the Firefox build but I have a better fix for the Rust build and think we should use that one.

comment:12 in reply to:  9 Changed 7 months ago by gk

Keywords: TorBrowserTeam201804R added; TorBrowserTeam201804 removed
Status: needs_revisionneeds_review

Replying to boklm:

Replying to gk:

So, I worked around that by building LLVM during the rust compilation (which make the whole process taking significantly more time) using that version in turn.

I don't see LLVM being built in your patch. Is it done internally by the rust build system?

Yes, it uses the LLVM code that is shipped with the Rust source.

The extracting of gcc and update of the PATH/LD_LIBRARY_PATH could be replaced by:

[% pc('gcc', 'var/setup', { compiler_tarfile => c('input_files_by_name/gcc') }) %]

However it will also set hardened-cc links. Is it because the build doesn't work with hardened-cc that you didn't use it?

No, I originally wanted to avoid using our own GCC as this adds complexity but piece by piece I realized that this is not realistic and added workarounds which resulted in the final patch, forgetting about this macro. This is fixed now.

For the ln -s gcc cc link, I am wondering if we should add it directly in projects/gcc/build, or if there are cases where we might not want this link.

I can avoid this clumsy symlink by using a configure option which seems neat and which we probably need to use anyway when actually cross-compiling. So, I think we should go with that.

In projects/tor/build, I think the lines after the IF c("var/linux") && c("var/nightly") can be indented.

Fixed.

The updated patch is at bug_25481_v4 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_25481_v4&id=fde93d07ca425ee884930cf01fd435b31ace48b8) in my public tor-browser-build repo. Note: I tested the x86_64 tor and it is running fine for me. I even checked whether tor built from the same revision twice is identical, and it is! Moreover, I verified that the hardening properties on the tor binary are still the ones we want to have (with the checksec script). So, we are good here. I guess the 32bit tor could get tested, though, which I did not do yet.

Last edited 7 months ago by gk (previous) (diff)

comment:13 Changed 7 months ago by boklm

Status: needs_reviewneeds_revision

One small thing: the prev_version variable in projects/rust/config should be named var/prev_version instead, as it is not a variable that has a specific meaning for rbm.

Other than this the patch looks good to me.

I tried a 32bit build of Tor, ran checksec on it, and have been able to bootstrap with it.

comment:14 Changed 7 months ago by gk

Status: needs_revisionneeds_review

comment:15 Changed 7 months ago by boklm

Resolution: fixed
Status: needs_reviewclosed

This looks good. This is now commit 5be41831b52795ee87c57cdbd18da68bf73fe1cc in master.

Note: See TracTickets for help on using tickets.