Opened 6 months ago

Closed 3 weeks ago

#30321 closed task (fixed)

Adapt Linux toolchain for Firefox 68 ESR

Reported by: gk Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, ff68-esr, GeorgKoppen201908, tbb-9.0-must-alpha, TorBrowserTeam201909
Cc: boklm Actual Points:
Parent ID: #30320 Points:
Reviewer: Sponsor: Sponsor44-can

Description

This is the (parent) ticket for adapting our toolchain for Linux bundles built from Firefox 68 ESR.

Child Tickets

TicketStatusOwnerSummaryComponent
#25930closedtbb-teamUpdate gcc to 8.XApplications/Tor Browser
#30736closedtbb-teamMake sure we have Yasm >= 1.2.0 available on Linux 64bitApplications/Tor Browser
#31448closedtbb-teamgold and lld break linking 32bit Linux bundles we need to resort to bfdApplications/Tor Browser
#31449closedtbb-teamSigning tools for 32bit Linux are 64bit nowApplications/Tor Browser
#31618closedtbb-teamlinux32 builds of Tor Browser 9.0a6 are not matchingApplications/Tor Browser

Change History (27)

comment:1 Changed 5 months ago by gk

Okay, I think I have all prerequisites in place. Now on to get the whole thing actually compiled. Right now it is at least failing during Stylo compilation with the following backtrace:

27:59.16 thread 'stack backtrace:
27:59.32    0:     0x557146665c03 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h28991c1d2d7fb0bc
27:59.35                                at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
27:59.35    1:     0x5571466608ab - std::sys_common::backtrace::_print::hdb8f9a6bc6550c73
27:59.35                                at src/libstd/sys_common/backtrace.rs:70
27:59.35    2:     0x557146663986 - std::panicking::default_hook::{{closure}}::hf5bd0e166d7070c2
27:59.35                                at src/libstd/sys_common/backtrace.rs:58
27:59.35                                at src/libstd/panicking.rs:200
27:59.35    3:     0x557146663704 - std::panicking::default_hook::h551580c4307ee3ab
27:59.35                                at src/libstd/panicking.rs:215
27:59.35    4:     0x55714666408f - std::panicking::rust_panic_with_hook::h3b41ee611369f0b1
27:59.35                                at src/libstd/panicking.rs:478
27:59.35    5:     0x557146663c11 - std::panicking::continue_panic_fmt::h86542a6b075513dd
27:59.35                                at src/libstd/panicking.rs:385
27:59.35    6:     0x557146663b5e - std::panicking::begin_panic_fmt::hd511eaf3357d07a6
27:59.35                                at src/libstd/panicking.rs:340
27:59.35    7:     0x557146658103 - std::io::stdio::_print::h9035f66296b9efd1
27:59.35                                at src/libstd/io/stdio.rs:735
27:59.35                                at src/libstd/io/stdio.rs:744
27:59.35    8:     0x557146251c08 - cargo::core::compiler::job_queue::JobQueue::drain_the_queue::hd94276b883027b45
27:59.35    9:     0x55714611bc94 - std::panicking::try::do_call::h5f91376082cf7b1e
27:59.35   10:     0x55714666fb79 - __rust_maybe_catch_panic
27:59.35                                at src/libpanic_unwind/lib.rs:87
27:59.35   11:     0x5571462196c1 - cargo::core::compiler::context::Context::compile::hd97932f2c980d56c
27:59.35   12:     0x5571462c5bb7 - cargo::ops::cargo_compile::compile_ws::h63581a7f0cd1f2ed
27:59.35   13:     0x5571462c3a9e - cargo::ops::cargo_compile::compile::h632a606c9260ae50
27:59.35   14:     0x5571460bdceb - cargo::commands::rustc::exec::h614f2a41cf15f27d
27:59.35   15:     0x5571460c27d1 - cargo::main::hca1a1dbe3739f08f
27:59.35   16:     0x557146085fc2 - std::rt::lang_start::{{closure}}::he6e482ffcf9ae8a6
27:59.35   17:     0x557146663a92 - std::panicking::try::do_call::hdb24dd18b35dbd0e
27:59.35                                at src/libstd/rt.rs:49
27:59.35                                at src/libstd/panicking.rs:297
27:59.36   18:     0x55714666fb79 - __rust_maybe_catch_panic
27:59.36                                at src/libpanic_unwind/lib.rs:87
27:59.36   19:     0x55714666465c - std::rt::lang_start_internal::hc9e10b9bd186777b
27:59.36                                at src/libstd/panicking.rs:276
27:59.36                                at src/libstd/panic.rs:388
27:59.36                                at src/libstd/rt.rs:48
27:59.36   20:     0x5571460c5dc1 - main
27:59.36   21:     0x2b2fb08e4eac - __libc_start_main
27:59.36   22:     0x557146082d68 - <unknown>

There are a bunch of warnings but no errors shown besides

28:03.67 make[4]: *** [force-cargo-library-build] Error 101

comment:2 Changed 5 months ago by cypherpunks

If you depend on software rasterization, image decoding, or color space conversion and compile Skia with GCC, MSVC or another compiler, you will see dramatically worse performance than if you use Clang.

comment:3 Changed 5 months ago by tom

Yeah; we switched our linux builds to clang. Sorry I didn't see this earlier. One of the factors to that was Skia becoming much slower (I never got an idea of the numbers) with gcc.

comment:4 in reply to:  1 Changed 5 months ago by gk

Replying to gk:

Okay, I think I have all prerequisites in place. Now on to get the whole thing actually compiled. Right now it is at least failing during Stylo compilation with the following backtrace:

27:59.16 thread 'stack backtrace:
27:59.32    0:     0x557146665c03 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h28991c1d2d7fb0bc
27:59.35                                at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
27:59.35    1:     0x5571466608ab - std::sys_common::backtrace::_print::hdb8f9a6bc6550c73
27:59.35                                at src/libstd/sys_common/backtrace.rs:70
27:59.35    2:     0x557146663986 - std::panicking::default_hook::{{closure}}::hf5bd0e166d7070c2
27:59.35                                at src/libstd/sys_common/backtrace.rs:58
27:59.35                                at src/libstd/panicking.rs:200
27:59.35    3:     0x557146663704 - std::panicking::default_hook::h551580c4307ee3ab
27:59.35                                at src/libstd/panicking.rs:215
27:59.35    4:     0x55714666408f - std::panicking::rust_panic_with_hook::h3b41ee611369f0b1
27:59.35                                at src/libstd/panicking.rs:478
27:59.35    5:     0x557146663c11 - std::panicking::continue_panic_fmt::h86542a6b075513dd
27:59.35                                at src/libstd/panicking.rs:385
27:59.35    6:     0x557146663b5e - std::panicking::begin_panic_fmt::hd511eaf3357d07a6
27:59.35                                at src/libstd/panicking.rs:340
27:59.35    7:     0x557146658103 - std::io::stdio::_print::h9035f66296b9efd1
27:59.35                                at src/libstd/io/stdio.rs:735
27:59.35                                at src/libstd/io/stdio.rs:744
27:59.35    8:     0x557146251c08 - cargo::core::compiler::job_queue::JobQueue::drain_the_queue::hd94276b883027b45
27:59.35    9:     0x55714611bc94 - std::panicking::try::do_call::h5f91376082cf7b1e
27:59.35   10:     0x55714666fb79 - __rust_maybe_catch_panic
27:59.35                                at src/libpanic_unwind/lib.rs:87
27:59.35   11:     0x5571462196c1 - cargo::core::compiler::context::Context::compile::hd97932f2c980d56c
27:59.35   12:     0x5571462c5bb7 - cargo::ops::cargo_compile::compile_ws::h63581a7f0cd1f2ed
27:59.35   13:     0x5571462c3a9e - cargo::ops::cargo_compile::compile::h632a606c9260ae50
27:59.35   14:     0x5571460bdceb - cargo::commands::rustc::exec::h614f2a41cf15f27d
27:59.35   15:     0x5571460c27d1 - cargo::main::hca1a1dbe3739f08f
27:59.35   16:     0x557146085fc2 - std::rt::lang_start::{{closure}}::he6e482ffcf9ae8a6
27:59.35   17:     0x557146663a92 - std::panicking::try::do_call::hdb24dd18b35dbd0e
27:59.35                                at src/libstd/rt.rs:49
27:59.35                                at src/libstd/panicking.rs:297
27:59.36   18:     0x55714666fb79 - __rust_maybe_catch_panic
27:59.36                                at src/libpanic_unwind/lib.rs:87
27:59.36   19:     0x55714666465c - std::rt::lang_start_internal::hc9e10b9bd186777b
27:59.36                                at src/libstd/panicking.rs:276
27:59.36                                at src/libstd/panic.rs:388
27:59.36                                at src/libstd/rt.rs:48
27:59.36   20:     0x5571460c5dc1 - main
27:59.36   21:     0x2b2fb08e4eac - __libc_start_main
27:59.36   22:     0x557146082d68 - <unknown>

There are a bunch of warnings but no errors shown besides

28:03.67 make[4]: *** [force-cargo-library-build] Error 101

Looks like https://bugs.gentoo.org/680934, fun.

comment:5 Changed 5 months ago by gk

Alright, using 1.34.1 provided by the Rust folks does not change the situation and, as in the Gentoo bug, if I kick the build again via the debug shell I get past the problem and the Rust code compiles successfully.

comment:7 in reply to:  6 Changed 4 months ago by gk

Replying to gk:

https://github.com/rust-lang/rust/issues/61468
https://bugs.gentoo.org/687028

seem to be relevant as well.

Alright, some follow-ups: I tried a bunch of random things to no avail (using precompiled Rust versions 1.34.1, 1.35.0, just compiling Firefox with -j1, trying to make sure cargo is compiling only with -j1, resuming the build from within the VM, which surprisingly worked) and ended up with omitting --verbose to our mach build incantation which "worked". Trying to figure out the underlying problem Alex Crichton pointed me on the right path by linking to https://github.com/travis-ci/travis-ci/issues/4704. In particular https://github.com/travis-ci/travis-ci/issues/4704#issuecomment-348435959 has been useful:

Pretty much every commandline tool expects stdout to be in blocking mode, and does not properly retry when in nonblocking mode.

Testing the Python snippets from that bug both on the host machine and in the VM did not show any issues, thus something must change the blocking mode during compilation.

And, indeed, after more digging I stumbled over

https://github.com/nodejs/node/issues/14752
https://github.com/bnoordhuis/io.js/commit/256614ad5335419b75c798bf0d136bf1a0395e81

and I could reproduce the issue with the Python snippets from the Travis bug report and work around it.

comment:8 Changed 4 months ago by gk

Compiling our 32bits were failing, too. The first show-stopper was a failure related to us setting host and target to i686-linux-gnu:

 0:08.82 ERROR: The rust compiler host (x86_64-unknown-linux-gnu) is not suitable for the configure host (i686-pc-linux-gnu/i686-unknown-linux-gnu).

That gets solved by just keeping ac_add_options --target=i686-linux-gnu.

Then gold is breaking when linking libxul.so

41:17.86 /var/tmp/dist/binutils/bin/ld.gold.real: internal error in relocate_section, at i386.cc:3684
41:18.04 clang-8: error: linker command failed with exit code 1 (use -v to see invocation)

It seems we hit gold_assert(sh_type == elfcpp::SHT_REL); and others might have encountered the same, see: https://patchwork.openembedded.org/patch/155471/

I tried to figure out whether a GCC/binutils toolchain fares better but that breaks already early with an ICE:

25:09.35 /var/tmp/build/firefox-a194f586c005/gfx/skia/skia/third_party/skcms/src/Transform_inl.h:640:13: internal compiler error: in convert_move, at expr.c:218
25:09.35  static void exec_ops(const Op* ops, const void** args,
25:09.35              ^~~~~~~~
25:10.21 0x596b7a convert_move(rtx_def*, rtx_def*, int)
25:10.21 	../.././gcc/expr.c:218
25:10.21 0x8a06c2 convert_modes(machine_mode, machine_mode, rtx_def*, int)
25:10.22 	../.././gcc/expr.c:709
25:10.22 0xbc32f2 emit_partition_copy
25:10.22 	../.././gcc/tree-outof-ssa.c:220
25:10.22 0xbc32f2 insert_part_to_rtx_on_edge
25:10.22 	../.././gcc/tree-outof-ssa.c:389
25:10.22 0xbc32f2 elim_create
25:10.22 	../.././gcc/tree-outof-ssa.c:675
25:10.22 0xbc32f2 eliminate_phi
25:10.22 	../.././gcc/tree-outof-ssa.c:733
25:10.22 0xbc32f2 expand_phi_nodes(ssaexpand*)
25:10.22 	../.././gcc/tree-outof-ssa.c:909
25:10.23 0x7b23be execute
25:10.23 	../.././gcc/cfgexpand.c:6416

Using bfd with ac_add_options --enable-linker=bff works and I can successfully compile and link the browser. Using lld fails, too:

41:04.50 ld.lld: error: dwarf.c:(.debug_info+0x20DE295A): has non-ABS relocation R_386_GOTOFF against symbol '.LC25'
41:08.83 clang-8: error: linker command failed with exit code 1 (use -v to see invocation)

Might be worth figuring out at least why gold is not happy here.

Version 0, edited 4 months ago by gk (next)

comment:9 Changed 4 months ago by gk

Keywords: TorBrowserTeam201906 GeorgKoppen201906 added
Priority: MediumVery High

comment:10 Changed 4 months ago by gk

One big drawback we still have for the 32bit builds is that mar and mbsdiff are still 64bit binaries and thus, signing on 32bit systems is not working out of the box.

comment:11 Changed 4 months ago by gk

Keywords: GeorgKoppen201907 added; GeorgKoppen201906 removed

Moving my tickets to July.

comment:12 Changed 3 months ago by gk

Keywords: TorBrowserTeam201907 added; TorBrowserTeam201906 removed

Moving tickets to July

comment:13 Changed 3 months ago by gk

Keywords: tbb-9.0-must-nightly added

Starting with 9.0 keywords

comment:14 Changed 2 months ago by boklm

My branch linux_esr68_v6 contains the latest version of the patches for the Linux changes:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/log/?h=linux_esr68_v6

The top commit from this branch is only for testing and should not be merged. It should be replaced by a commit updating the firefox version (after we pushed an esr68 branch to tor-browser.git).

comment:15 Changed 2 months ago by cypherpunks

+ac_add_options --enable-proxy-bypass-protection for 64-bit only?

comment:16 Changed 2 months ago by pili

Sponsor: Sponsor44-can

Adding Sponsor 44 to ESR68 tickets

comment:17 in reply to:  14 ; Changed 2 months ago by gk

Replying to boklm:

My branch linux_esr68_v6 contains the latest version of the patches for the Linux changes:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/log/?h=linux_esr68_v6

The top commit from this branch is only for testing and should not be merged. It should be replaced by a commit updating the firefox version (after we pushed an esr68 branch to tor-browser.git).

Thanks. The language pack related commit looks good to me, nice find! I cherry-picked that one to master as commit 4a632f8ac10f5d01ac9ef075fc698bd7454ab6cb. I looked over the ESR 68 changes again and tweaked the commit slightly, e.g. by passing --enabled-proxy-bypass-protection to i686 as well. It's now commit c26b8196c90e908a83104c639865a2aca6598efe. Please have a look once you are back.

There are a number of XXX's and issues we still should try to address as well. For verbosity they are:

1) We are installing python just for mach. I guess we should use the one we built ourselves instead?
2) gold and lld break for 32bit and we need to resort to bfd. This needs investigation (I bet Mozilla is not affected here)
3) We don't get 32bit signing related binaries anymore but we need those. We need to fix that.
4) We need to enable the updater again once the rebases patches are available
5) I think we still should build the debug build with GCC enabled to catch potential issues Mozilla is not catching anymore after they switched to clang.

comment:18 Changed 2 months ago by gk

Keywords: TorBrowserTeam201908 added; TorBrowserTeam201907 removed

Moving tickets to August, part 1

comment:19 Changed 2 months ago by gk

Keywords: GeorgKoppen201908 added; GeorgKoppen201907 removed

Move my tickets.

comment:20 in reply to:  17 Changed 2 months ago by gk

Replying to gk:

Replying to boklm:

My branch linux_esr68_v6 contains the latest version of the patches for the Linux changes:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/log/?h=linux_esr68_v6

The top commit from this branch is only for testing and should not be merged. It should be replaced by a commit updating the firefox version (after we pushed an esr68 branch to tor-browser.git).

Thanks. The language pack related commit looks good to me, nice find! I cherry-picked that one to master as commit 4a632f8ac10f5d01ac9ef075fc698bd7454ab6cb. I looked over the ESR 68 changes again and tweaked the commit slightly, e.g. by passing --enabled-proxy-bypass-protection to i686 as well. It's now commit c26b8196c90e908a83104c639865a2aca6598efe. Please have a look once you are back.

There are a number of XXX's and issues we still should try to address as well. For verbosity they are:

1) We are installing python just for mach. I guess we should use the one we built ourselves instead?

#31447

2) gold and lld break for 32bit and we need to resort to bfd. This needs investigation (I bet Mozilla is not affected here)

#31448

3) We don't get 32bit signing related binaries anymore but we need those. We need to fix that.

#31449 and the only real blocker for this ticket.

4) We need to enable the updater again once the rebases patches are available

We'll do that on the fly once the updater patches landed.

5) I think we still should build the debug build with GCC enabled to catch potential issues Mozilla is not catching anymore after they switched to clang.

#31450

comment:21 Changed 7 weeks ago by gk

Keywords: tbb-9.0-must-alpha added; tbb-9.0-must-nightly removed

Move must-nightly items to must-alpha ones.

comment:22 Changed 7 weeks ago by gk

Keywords: TorBrowserTeam201909 added; TorBrowserTeam201908 removed

Moving must-alpha tickets to September.

comment:23 Changed 6 weeks ago by pili

Points: 2.25

comment:24 Changed 6 weeks ago by pili

Points: 2.25

comment:25 Changed 4 weeks ago by gk

Resolution: fixed
Status: newclosed

I think we are done here.

comment:26 Changed 3 weeks ago by boklm

Resolution: fixed
Status: closedreopened

Reopening temporarily to set Actual Points on #31448 (as trac does not allow any change on a ticket if the parent is closed).

comment:27 Changed 3 weeks ago by boklm

Resolution: fixed
Status: reopenedclosed
Note: See TracTickets for help on using tickets.