Adapt Linux toolchain for Firefox 68 ESR
This is the (parent) ticket for adapting our toolchain for Linux bundles built from Firefox 68 ESR.
Activity
Trac:
Parent Ticket: #30320 (moved)
Child Ticket(s): #31618 (moved), #25930 (moved), #30736 (moved), #31448 (moved), #31449 (moved)
Okay, I think I have all prerequisites in place. Now on to get the whole thing actually compiled. Right now it is at least failing during Stylo compilation with the following backtrace:
27:59.16 thread 'stack backtrace: 27:59.32 0: 0x557146665c03 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h28991c1d2d7fb0bc 27:59.35 at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39 27:59.35 1: 0x5571466608ab - std::sys_common::backtrace::_print::hdb8f9a6bc6550c73 27:59.35 at src/libstd/sys_common/backtrace.rs:70 27:59.35 2: 0x557146663986 - std::panicking::default_hook::{{closure}}::hf5bd0e166d7070c2 27:59.35 at src/libstd/sys_common/backtrace.rs:58 27:59.35 at src/libstd/panicking.rs:200 27:59.35 3: 0x557146663704 - std::panicking::default_hook::h551580c4307ee3ab 27:59.35 at src/libstd/panicking.rs:215 27:59.35 4: 0x55714666408f - std::panicking::rust_panic_with_hook::h3b41ee611369f0b1 27:59.35 at src/libstd/panicking.rs:478 27:59.35 5: 0x557146663c11 - std::panicking::continue_panic_fmt::h86542a6b075513dd 27:59.35 at src/libstd/panicking.rs:385 27:59.35 6: 0x557146663b5e - std::panicking::begin_panic_fmt::hd511eaf3357d07a6 27:59.35 at src/libstd/panicking.rs:340 27:59.35 7: 0x557146658103 - std::io::stdio::_print::h9035f66296b9efd1 27:59.35 at src/libstd/io/stdio.rs:735 27:59.35 at src/libstd/io/stdio.rs:744 27:59.35 8: 0x557146251c08 - cargo::core::compiler::job_queue::JobQueue::drain_the_queue::hd94276b883027b45 27:59.35 9: 0x55714611bc94 - std::panicking::try::do_call::h5f91376082cf7b1e 27:59.35 10: 0x55714666fb79 - __rust_maybe_catch_panic 27:59.35 at src/libpanic_unwind/lib.rs:87 27:59.35 11: 0x5571462196c1 - cargo::core::compiler::context::Context::compile::hd97932f2c980d56c 27:59.35 12: 0x5571462c5bb7 - cargo::ops::cargo_compile::compile_ws::h63581a7f0cd1f2ed 27:59.35 13: 0x5571462c3a9e - cargo::ops::cargo_compile::compile::h632a606c9260ae50 27:59.35 14: 0x5571460bdceb - cargo::commands::rustc::exec::h614f2a41cf15f27d 27:59.35 15: 0x5571460c27d1 - cargo::main::hca1a1dbe3739f08f 27:59.35 16: 0x557146085fc2 - std::rt::lang_start::{{closure}}::he6e482ffcf9ae8a6 27:59.35 17: 0x557146663a92 - std::panicking::try::do_call::hdb24dd18b35dbd0e 27:59.35 at src/libstd/rt.rs:49 27:59.35 at src/libstd/panicking.rs:297 27:59.36 18: 0x55714666fb79 - __rust_maybe_catch_panic 27:59.36 at src/libpanic_unwind/lib.rs:87 27:59.36 19: 0x55714666465c - std::rt::lang_start_internal::hc9e10b9bd186777b 27:59.36 at src/libstd/panicking.rs:276 27:59.36 at src/libstd/panic.rs:388 27:59.36 at src/libstd/rt.rs:48 27:59.36 20: 0x5571460c5dc1 - main 27:59.36 21: 0x2b2fb08e4eac - __libc_start_main 27:59.36 22: 0x557146082d68 - <unknown>There are a bunch of warnings but no errors shown besides
28:03.67 make[4]: *** [force-cargo-library-build] Error 101-
Replying to gk:
Okay, I think I have all prerequisites in place. Now on to get the whole thing actually compiled. Right now it is at least failing during Stylo compilation with the following backtrace: {{{ 27:59.16 thread 'stack backtrace: 27:59.32 0: 0x557146665c03 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h28991c1d2d7fb0bc 27:59.35 at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39 27:59.35 1: 0x5571466608ab - std::sys_common::backtrace::_print::hdb8f9a6bc6550c73 27:59.35 at src/libstd/sys_common/backtrace.rs:70 27:59.35 2: 0x557146663986 - std::panicking::default_hook::{{closure}}::hf5bd0e166d7070c2 27:59.35 at src/libstd/sys_common/backtrace.rs:58 27:59.35 at src/libstd/panicking.rs:200 27:59.35 3: 0x557146663704 - std::panicking::default_hook::h551580c4307ee3ab 27:59.35 at src/libstd/panicking.rs:215 27:59.35 4: 0x55714666408f - std::panicking::rust_panic_with_hook::h3b41ee611369f0b1 27:59.35 at src/libstd/panicking.rs:478 27:59.35 5: 0x557146663c11 - std::panicking::continue_panic_fmt::h86542a6b075513dd 27:59.35 at src/libstd/panicking.rs:385 27:59.35 6: 0x557146663b5e - std::panicking::begin_panic_fmt::hd511eaf3357d07a6 27:59.35 at src/libstd/panicking.rs:340 27:59.35 7: 0x557146658103 - std::io::stdio::_print::h9035f66296b9efd1 27:59.35 at src/libstd/io/stdio.rs:735 27:59.35 at src/libstd/io/stdio.rs:744 27:59.35 8: 0x557146251c08 - cargo::core::compiler::job_queue::JobQueue::drain_the_queue::hd94276b883027b45 27:59.35 9: 0x55714611bc94 - std::panicking::try::do_call::h5f91376082cf7b1e 27:59.35 10: 0x55714666fb79 - __rust_maybe_catch_panic 27:59.35 at src/libpanic_unwind/lib.rs:87 27:59.35 11: 0x5571462196c1 - cargo::core::compiler::context::Context::compile::hd97932f2c980d56c 27:59.35 12: 0x5571462c5bb7 - cargo::ops::cargo_compile::compile_ws::h63581a7f0cd1f2ed 27:59.35 13: 0x5571462c3a9e - cargo::ops::cargo_compile::compile::h632a606c9260ae50 27:59.35 14: 0x5571460bdceb - cargo::commands::rustc::exec::h614f2a41cf15f27d 27:59.35 15: 0x5571460c27d1 - cargo::main::hca1a1dbe3739f08f 27:59.35 16: 0x557146085fc2 - std::rt::lang_start::{{closure}}::he6e482ffcf9ae8a6 27:59.35 17: 0x557146663a92 - std::panicking::try::do_call::hdb24dd18b35dbd0e 27:59.35 at src/libstd/rt.rs:49 27:59.35 at src/libstd/panicking.rs:297 27:59.36 18: 0x55714666fb79 - __rust_maybe_catch_panic 27:59.36 at src/libpanic_unwind/lib.rs:87 27:59.36 19: 0x55714666465c - std::rt::lang_start_internal::hc9e10b9bd186777b 27:59.36 at src/libstd/panicking.rs:276 27:59.36 at src/libstd/panic.rs:388 27:59.36 at src/libstd/rt.rs:48 27:59.36 20: 0x5571460c5dc1 - main 27:59.36 21: 0x2b2fb08e4eac - __libc_start_main 27:59.36 22: 0x557146082d68 - }}} There are a bunch of warnings but no errors shown besides {{{ 28:03.67 make[4]: *** [force-cargo-library-build] Error 101 }}}
Looks like https://bugs.gentoo.org/680934, fun.
-
https://github.com/rust-lang/rust/issues/61468 https://bugs.gentoo.org/687028
seem to be relevant as well.
-
Replying to gk:
https://github.com/rust-lang/rust/issues/61468 https://bugs.gentoo.org/687028
seem to be relevant as well.
Alright, some follow-ups: I tried a bunch of random things to no avail (using precompiled Rust versions 1.34.1, 1.35.0, just compiling Firefox with
-j1, trying to make surecargois compiling only with-j1, resuming the build from within the VM, which surprisingly worked) and ended up with omitting--verboseto ourmach buildincantation which "worked". Trying to figure out the underlying problem Alex Crichton put me on the right path by linking to https://github.com/travis-ci/travis-ci/issues/4704. In particular https://github.com/travis-ci/travis-ci/issues/4704#issuecomment-348435959 has been useful:Pretty much every commandline tool expects stdout to be in blocking mode, and does not properly retry when in nonblocking mode.Testing the Python snippets from that bug both on the host machine and in the VM did not show any issues, thus something must change the blocking mode during compilation.
And, indeed, after more digging I stumbled over
https://github.com/nodejs/node/issues/14752 https://github.com/bnoordhuis/io.js/commit/256614ad5335419b75c798bf0d136bf1a0395e81
and I could reproduce the issue with the Python snippets from the Travis bug report and work around it.
-
Compiling our 32bits were failing, too. The first show-stopper was a failure related to us setting
hostandtargettoi686-linux-gnu:0:08.82 ERROR: The rust compiler host (x86_64-unknown-linux-gnu) is not suitable for the configure host (i686-pc-linux-gnu/i686-unknown-linux-gnu).That gets solved by just keeping
ac_add_options --target=i686-linux-gnu.Then gold is breaking when linking
libxul.so41:17.86 /var/tmp/dist/binutils/bin/ld.gold.real: internal error in relocate_section, at i386.cc:3684 41:18.04 clang-8: error: linker command failed with exit code 1 (use -v to see invocation)It seems we hit
gold_assert(sh_type == elfcpp::SHT_REL);and others might have encountered the same, see: https://patchwork.openembedded.org/patch/155471/I tried to figure out whether a GCC/binutils toolchain fares better but that breaks already early with an ICE:
25:09.35 /var/tmp/build/firefox-a194f586c005/gfx/skia/skia/third_party/skcms/src/Transform_inl.h:640:13: internal compiler error: in convert_move, at expr.c:218 25:09.35 static void exec_ops(const Op* ops, const void** args, 25:09.35 ^~~~~~~~ 25:10.21 0x596b7a convert_move(rtx_def*, rtx_def*, int) 25:10.21 ../.././gcc/expr.c:218 25:10.21 0x8a06c2 convert_modes(machine_mode, machine_mode, rtx_def*, int) 25:10.22 ../.././gcc/expr.c:709 25:10.22 0xbc32f2 emit_partition_copy 25:10.22 ../.././gcc/tree-outof-ssa.c:220 25:10.22 0xbc32f2 insert_part_to_rtx_on_edge 25:10.22 ../.././gcc/tree-outof-ssa.c:389 25:10.22 0xbc32f2 elim_create 25:10.22 ../.././gcc/tree-outof-ssa.c:675 25:10.22 0xbc32f2 eliminate_phi 25:10.22 ../.././gcc/tree-outof-ssa.c:733 25:10.22 0xbc32f2 expand_phi_nodes(ssaexpand*) 25:10.22 ../.././gcc/tree-outof-ssa.c:909 25:10.23 0x7b23be execute 25:10.23 ../.././gcc/cfgexpand.c:6416Using
bfdwithac_add_options --enable-linker=bfdworks and I can successfully compile and link the browser. Usinglldfails, too:41:04.50 ld.lld: error: dwarf.c:(.debug_info+0x20DE295A): has non-ABS relocation R_386_GOTOFF against symbol '.LC25' 41:08.83 clang-8: error: linker command failed with exit code 1 (use -v to see invocation)Might be worth figuring out at least why
goldis not happy here. 
My branch
linux_esr68_v6contains the latest version of the patches for the Linux changes: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/log/?h=linux_esr68_v6The top commit from this branch is only for testing and should not be merged. It should be replaced by a commit updating the firefox version (after we pushed an esr68 branch to
tor-browser.git).-
Replying to boklm:
My branch
linux_esr68_v6contains the latest version of the patches for the Linux changes: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/log/?h=linux_esr68_v6The top commit from this branch is only for testing and should not be merged. It should be replaced by a commit updating the firefox version (after we pushed an esr68 branch to
tor-browser.git).Thanks. The language pack related commit looks good to me, nice find! I cherry-picked that one to
masteras commit 4a632f8ac10f5d01ac9ef075fc698bd7454ab6cb. I looked over the ESR 68 changes again and tweaked the commit slightly, e.g. by passing--enabled-proxy-bypass-protectionto i686 as well. It's now commit c26b8196c90e908a83104c639865a2aca6598efe. Please have a look once you are back.There are a number of XXX's and issues we still should try to address as well. For verbosity they are:
- We are installing
pythonjust formach. I guess we should use the one we built ourselves instead? - gold and lld break for 32bit and we need to resort to bfd. This needs investigation (I bet Mozilla is not affected here)
- We don't get 32bit signing related binaries anymore but we need those. We need to fix that.
- We need to enable the updater again once the rebases patches are available
- I think we still should build the debug build with GCC enabled to catch potential issues Mozilla is not catching anymore after they switched to clang.
- We are installing
-
Replying to gk:
Replying to boklm:
My branch
linux_esr68_v6contains the latest version of the patches for the Linux changes: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/log/?h=linux_esr68_v6The top commit from this branch is only for testing and should not be merged. It should be replaced by a commit updating the firefox version (after we pushed an esr68 branch to
tor-browser.git).Thanks. The language pack related commit looks good to me, nice find! I cherry-picked that one to
masteras commit 4a632f8ac10f5d01ac9ef075fc698bd7454ab6cb. I looked over the ESR 68 changes again and tweaked the commit slightly, e.g. by passing--enabled-proxy-bypass-protectionto i686 as well. It's now commit c26b8196c90e908a83104c639865a2aca6598efe. Please have a look once you are back.There are a number of XXX's and issues we still should try to address as well. For verbosity they are:
- We are installing
pythonjust formach. I guess we should use the one we built ourselves instead?
- gold and lld break for 32bit and we need to resort to bfd. This needs investigation (I bet Mozilla is not affected here)
- We don't get 32bit signing related binaries anymore but we need those. We need to fix that.
#31449 (moved) and the only real blocker for this ticket.
- We need to enable the updater again once the rebases patches are available
We'll do that on the fly once the updater patches landed.
- I think we still should build the debug build with GCC enabled to catch potential issues Mozilla is not catching anymore after they switched to clang.
- We are installing
-
Reopening temporarily to set
Actual Pointson #31448 (moved) (as trac does not allow any change on a ticket if the parent is closed).Trac:
Status: closed to reopened
Resolution: fixed to N/A mentioned in issue #31448 (moved)
mentioned in issue #31449 (moved)
mentioned in issue #31729 (moved)
mentioned in issue #30320 (moved)
mentioned in issue #30736 (moved)
mentioned in issue #31618 (moved)
