Opened 3 months ago

Last modified 4 days ago

#30321 new task

Adapt Linux toolchain for Firefox 68 ESR

Reported by: gk Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, ff68-esr, GeorgKoppen201907, TorBrowserTeam201907, tbb-9.0-must-nightly
Cc: boklm Actual Points:
Parent ID: #30320 Points:
Reviewer: Sponsor:

Description

This is the (parent) ticket for adapting our toolchain for Linux bundles built from Firefox 68 ESR.

Child Tickets

TicketStatusOwnerSummaryComponent
#25930closedtbb-teamUpdate gcc to 8.XApplications/Tor Browser
#30736newtbb-teamMake sure we have Yasm >= 1.2.0 available on Linux 64bitApplications/Tor Browser

Change History (13)

comment:1 Changed 7 weeks ago by gk

Okay, I think I have all prerequisites in place. Now on to get the whole thing actually compiled. Right now it is at least failing during Stylo compilation with the following backtrace:

27:59.16 thread 'stack backtrace:
27:59.32    0:     0x557146665c03 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h28991c1d2d7fb0bc
27:59.35                                at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
27:59.35    1:     0x5571466608ab - std::sys_common::backtrace::_print::hdb8f9a6bc6550c73
27:59.35                                at src/libstd/sys_common/backtrace.rs:70
27:59.35    2:     0x557146663986 - std::panicking::default_hook::{{closure}}::hf5bd0e166d7070c2
27:59.35                                at src/libstd/sys_common/backtrace.rs:58
27:59.35                                at src/libstd/panicking.rs:200
27:59.35    3:     0x557146663704 - std::panicking::default_hook::h551580c4307ee3ab
27:59.35                                at src/libstd/panicking.rs:215
27:59.35    4:     0x55714666408f - std::panicking::rust_panic_with_hook::h3b41ee611369f0b1
27:59.35                                at src/libstd/panicking.rs:478
27:59.35    5:     0x557146663c11 - std::panicking::continue_panic_fmt::h86542a6b075513dd
27:59.35                                at src/libstd/panicking.rs:385
27:59.35    6:     0x557146663b5e - std::panicking::begin_panic_fmt::hd511eaf3357d07a6
27:59.35                                at src/libstd/panicking.rs:340
27:59.35    7:     0x557146658103 - std::io::stdio::_print::h9035f66296b9efd1
27:59.35                                at src/libstd/io/stdio.rs:735
27:59.35                                at src/libstd/io/stdio.rs:744
27:59.35    8:     0x557146251c08 - cargo::core::compiler::job_queue::JobQueue::drain_the_queue::hd94276b883027b45
27:59.35    9:     0x55714611bc94 - std::panicking::try::do_call::h5f91376082cf7b1e
27:59.35   10:     0x55714666fb79 - __rust_maybe_catch_panic
27:59.35                                at src/libpanic_unwind/lib.rs:87
27:59.35   11:     0x5571462196c1 - cargo::core::compiler::context::Context::compile::hd97932f2c980d56c
27:59.35   12:     0x5571462c5bb7 - cargo::ops::cargo_compile::compile_ws::h63581a7f0cd1f2ed
27:59.35   13:     0x5571462c3a9e - cargo::ops::cargo_compile::compile::h632a606c9260ae50
27:59.35   14:     0x5571460bdceb - cargo::commands::rustc::exec::h614f2a41cf15f27d
27:59.35   15:     0x5571460c27d1 - cargo::main::hca1a1dbe3739f08f
27:59.35   16:     0x557146085fc2 - std::rt::lang_start::{{closure}}::he6e482ffcf9ae8a6
27:59.35   17:     0x557146663a92 - std::panicking::try::do_call::hdb24dd18b35dbd0e
27:59.35                                at src/libstd/rt.rs:49
27:59.35                                at src/libstd/panicking.rs:297
27:59.36   18:     0x55714666fb79 - __rust_maybe_catch_panic
27:59.36                                at src/libpanic_unwind/lib.rs:87
27:59.36   19:     0x55714666465c - std::rt::lang_start_internal::hc9e10b9bd186777b
27:59.36                                at src/libstd/panicking.rs:276
27:59.36                                at src/libstd/panic.rs:388
27:59.36                                at src/libstd/rt.rs:48
27:59.36   20:     0x5571460c5dc1 - main
27:59.36   21:     0x2b2fb08e4eac - __libc_start_main
27:59.36   22:     0x557146082d68 - <unknown>

There are a bunch of warnings but no errors shown besides

28:03.67 make[4]: *** [force-cargo-library-build] Error 101

comment:2 Changed 7 weeks ago by cypherpunks

If you depend on software rasterization, image decoding, or color space conversion and compile Skia with GCC, MSVC or another compiler, you will see dramatically worse performance than if you use Clang.

comment:3 Changed 7 weeks ago by tom

Yeah; we switched our linux builds to clang. Sorry I didn't see this earlier. One of the factors to that was Skia becoming much slower (I never got an idea of the numbers) with gcc.

comment:4 in reply to:  1 Changed 7 weeks ago by gk

Replying to gk:

Okay, I think I have all prerequisites in place. Now on to get the whole thing actually compiled. Right now it is at least failing during Stylo compilation with the following backtrace:

27:59.16 thread 'stack backtrace:
27:59.32    0:     0x557146665c03 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h28991c1d2d7fb0bc
27:59.35                                at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
27:59.35    1:     0x5571466608ab - std::sys_common::backtrace::_print::hdb8f9a6bc6550c73
27:59.35                                at src/libstd/sys_common/backtrace.rs:70
27:59.35    2:     0x557146663986 - std::panicking::default_hook::{{closure}}::hf5bd0e166d7070c2
27:59.35                                at src/libstd/sys_common/backtrace.rs:58
27:59.35                                at src/libstd/panicking.rs:200
27:59.35    3:     0x557146663704 - std::panicking::default_hook::h551580c4307ee3ab
27:59.35                                at src/libstd/panicking.rs:215
27:59.35    4:     0x55714666408f - std::panicking::rust_panic_with_hook::h3b41ee611369f0b1
27:59.35                                at src/libstd/panicking.rs:478
27:59.35    5:     0x557146663c11 - std::panicking::continue_panic_fmt::h86542a6b075513dd
27:59.35                                at src/libstd/panicking.rs:385
27:59.35    6:     0x557146663b5e - std::panicking::begin_panic_fmt::hd511eaf3357d07a6
27:59.35                                at src/libstd/panicking.rs:340
27:59.35    7:     0x557146658103 - std::io::stdio::_print::h9035f66296b9efd1
27:59.35                                at src/libstd/io/stdio.rs:735
27:59.35                                at src/libstd/io/stdio.rs:744
27:59.35    8:     0x557146251c08 - cargo::core::compiler::job_queue::JobQueue::drain_the_queue::hd94276b883027b45
27:59.35    9:     0x55714611bc94 - std::panicking::try::do_call::h5f91376082cf7b1e
27:59.35   10:     0x55714666fb79 - __rust_maybe_catch_panic
27:59.35                                at src/libpanic_unwind/lib.rs:87
27:59.35   11:     0x5571462196c1 - cargo::core::compiler::context::Context::compile::hd97932f2c980d56c
27:59.35   12:     0x5571462c5bb7 - cargo::ops::cargo_compile::compile_ws::h63581a7f0cd1f2ed
27:59.35   13:     0x5571462c3a9e - cargo::ops::cargo_compile::compile::h632a606c9260ae50
27:59.35   14:     0x5571460bdceb - cargo::commands::rustc::exec::h614f2a41cf15f27d
27:59.35   15:     0x5571460c27d1 - cargo::main::hca1a1dbe3739f08f
27:59.35   16:     0x557146085fc2 - std::rt::lang_start::{{closure}}::he6e482ffcf9ae8a6
27:59.35   17:     0x557146663a92 - std::panicking::try::do_call::hdb24dd18b35dbd0e
27:59.35                                at src/libstd/rt.rs:49
27:59.35                                at src/libstd/panicking.rs:297
27:59.36   18:     0x55714666fb79 - __rust_maybe_catch_panic
27:59.36                                at src/libpanic_unwind/lib.rs:87
27:59.36   19:     0x55714666465c - std::rt::lang_start_internal::hc9e10b9bd186777b
27:59.36                                at src/libstd/panicking.rs:276
27:59.36                                at src/libstd/panic.rs:388
27:59.36                                at src/libstd/rt.rs:48
27:59.36   20:     0x5571460c5dc1 - main
27:59.36   21:     0x2b2fb08e4eac - __libc_start_main
27:59.36   22:     0x557146082d68 - <unknown>

There are a bunch of warnings but no errors shown besides

28:03.67 make[4]: *** [force-cargo-library-build] Error 101

Looks like https://bugs.gentoo.org/680934, fun.

comment:5 Changed 7 weeks ago by gk

Alright, using 1.34.1 provided by the Rust folks does not change the situation and, as in the Gentoo bug, if I kick the build again via the debug shell I get past the problem and the Rust code compiles successfully.

comment:7 in reply to:  6 Changed 5 weeks ago by gk

Replying to gk:

https://github.com/rust-lang/rust/issues/61468
https://bugs.gentoo.org/687028

seem to be relevant as well.

Alright, some follow-ups: I tried a bunch of random things to no avail (using precompiled Rust versions 1.34.1, 1.35.0, just compiling Firefox with -j1, trying to make sure cargo is compiling only with -j1, resuming the build from within the VM, which surprisingly worked) and ended up with omitting --verbose to our mach build incantation which "worked". Trying to figure out the underlying problem Alex Crichton pointed me on the right path by linking to https://github.com/travis-ci/travis-ci/issues/4704. In particular https://github.com/travis-ci/travis-ci/issues/4704#issuecomment-348435959 has been useful:

Pretty much every commandline tool expects stdout to be in blocking mode, and does not properly retry when in nonblocking mode.

Testing the Python snippets from that bug both on the host machine and in the VM did not show any issues, thus something must change the blocking mode during compilation.

And, indeed, after more digging I stumbled over

https://github.com/nodejs/node/issues/14752
https://github.com/bnoordhuis/io.js/commit/256614ad5335419b75c798bf0d136bf1a0395e81

and I could reproduce the issue with the Python snippets from the Travis bug report and work around it.

comment:8 Changed 5 weeks ago by gk

Compiling our 32bits were failing, too. The first show-stopper was a failure related to us setting host and target to i686-linux-gnu:

 0:08.82 ERROR: The rust compiler host (x86_64-unknown-linux-gnu) is not suitable for the configure host (i686-pc-linux-gnu/i686-unknown-linux-gnu).

That gets solved by just keeping ac_add_options --target=i686-linux-gnu.

Then gold is breaking when linking libxul.so

41:17.86 /var/tmp/dist/binutils/bin/ld.gold.real: internal error in relocate_section, at i386.cc:3684
41:18.04 clang-8: error: linker command failed with exit code 1 (use -v to see invocation)

It seems we hit gold_assert(sh_type == elfcpp::SHT_REL); and others might have encountered the same, see: https://patchwork.openembedded.org/patch/155471/

I tried to figure out whether a GCC/binutils toolchain fares better but that breaks already early with an ICE:

25:09.35 /var/tmp/build/firefox-a194f586c005/gfx/skia/skia/third_party/skcms/src/Transform_inl.h:640:13: internal compiler error: in convert_move, at expr.c:218
25:09.35  static void exec_ops(const Op* ops, const void** args,
25:09.35              ^~~~~~~~
25:10.21 0x596b7a convert_move(rtx_def*, rtx_def*, int)
25:10.21 	../.././gcc/expr.c:218
25:10.21 0x8a06c2 convert_modes(machine_mode, machine_mode, rtx_def*, int)
25:10.22 	../.././gcc/expr.c:709
25:10.22 0xbc32f2 emit_partition_copy
25:10.22 	../.././gcc/tree-outof-ssa.c:220
25:10.22 0xbc32f2 insert_part_to_rtx_on_edge
25:10.22 	../.././gcc/tree-outof-ssa.c:389
25:10.22 0xbc32f2 elim_create
25:10.22 	../.././gcc/tree-outof-ssa.c:675
25:10.22 0xbc32f2 eliminate_phi
25:10.22 	../.././gcc/tree-outof-ssa.c:733
25:10.22 0xbc32f2 expand_phi_nodes(ssaexpand*)
25:10.22 	../.././gcc/tree-outof-ssa.c:909
25:10.23 0x7b23be execute
25:10.23 	../.././gcc/cfgexpand.c:6416

Using bfd with ac_add_options --enable-linker=bfd works and I can successfully compile and link the browser. Using lld fails, too:

41:04.50 ld.lld: error: dwarf.c:(.debug_info+0x20DE295A): has non-ABS relocation R_386_GOTOFF against symbol '.LC25'
41:08.83 clang-8: error: linker command failed with exit code 1 (use -v to see invocation)

Might be worth figuring out at least why gold is not happy here.

Last edited 5 weeks ago by gk (previous) (diff)

comment:9 Changed 5 weeks ago by gk

Keywords: TorBrowserTeam201906 GeorgKoppen201906 added
Priority: MediumVery High

comment:10 Changed 5 weeks ago by gk

One big drawback we still have for the 32bit builds is that mar and mbsdiff are still 64bit binaries and thus, signing on 32bit systems is not working out of the box.

comment:11 Changed 3 weeks ago by gk

Keywords: GeorgKoppen201907 added; GeorgKoppen201906 removed

Moving my tickets to July.

comment:12 Changed 2 weeks ago by gk

Keywords: TorBrowserTeam201907 added; TorBrowserTeam201906 removed

Moving tickets to July

comment:13 Changed 4 days ago by gk

Keywords: tbb-9.0-must-nightly added

Starting with 9.0 keywords

Note: See TracTickets for help on using tickets.