Tor Browser for Linux/ARMv7 (x86_64 build arch)

Trac:
Parent Ticket: #12631 (moved)

added TorBrowserTeam202006R component::applications/tor browser gitlab-tb-tor-browser-build owner::tbb-team parent::12631 priority::medium reviewer::gk severity::normal status::needs-review tbb-rbm type::project labels

Trac:
firefox-linux-arm-2019-10-29.log

Here's my current patch: https://notabug.org/JeremyRand/tor-browser-build/src/armhf-esr68

At the moment, firefox fails to build; log is at http://ea5faa5po25cf7fb.onion/projects/tor/attachment/ticket/32355/firefox-linux-arm-2019-10-29.log . The failure seems to be a Rust panic while building the Rust style package. It's not clear to me what the cause of this is... does anyone happen to recognize this error or have any ideas on what could be wrong?

You see the problem above the panic:

10:15.93 [style 0.0.1] /usr/include/stdlib.h:96:1: error: unknown type name '__BEGIN_NAMESPACE_STD'
10:15.93 [style 0.0.1] /usr/include/stdlib.h:98:1: error: expected unqualified-id
10:15.93 [style 0.0.1] /usr/include/stdlib.h:102:5: error: C++ requires a type specifier for all declarations
10:15.94 [style 0.0.1] /usr/include/stdlib.h:113:1: error: unknown type name '__END_NAMESPACE_STD'
10:15.94 [style 0.0.1] /usr/include/stdlib.h:116:22: error: expected ';' after top level declarator
10:15.94 [style 0.0.1] /usr/include/stdlib.h:124:1: error: unknown type name '__END_NAMESPACE_C99'
10:15.94 [style 0.0.1] /usr/include/stdlib.h:140:1: error: expected unqualified-id
10:15.94 [style 0.0.1] /usr/include/stdlib.h:143:1: error: unknown type name '__BEGIN_NAMESPACE_STD'
10:15.94 [style 0.0.1] /usr/include/stdlib.h:145:1: error: expected unqualified-id
10:15.94 [style 0.0.1] /usr/include/stdlib.h:153:1: error: unknown type name '__END_NAMESPACE_STD'
10:15.94 [style 0.0.1] /usr/include/stdlib.h:156:22: error: expected ';' after top level declarator
10:15.94 [style 0.0.1] /usr/include/stdlib.h:160:1: error: unknown type name '__END_NAMESPACE_C99'
10:15.94 [style 0.0.1] /usr/include/stdlib.h:163:22: error: expected ';' after top level declarator
10:15.94 [style 0.0.1] /usr/include/stdlib.h:168:1: error: unknown type name '__END_NAMESPACE_STD'
10:15.94 [style 0.0.1] /usr/include/stdlib.h:171:22: error: expected ';' after top level declarator
10:15.94 [style 0.0.1] /usr/include/stdlib.h:179:1: error: unknown type name '__END_NAMESPACE_C99'
10:15.94 [style 0.0.1] /usr/include/stdlib.h:182:22: error: expected ';' after top level declarator
10:15.94 [style 0.0.1] /usr/include/stdlib.h:191:1: error: unknown type name '__END_NAMESPACE_STD'
10:15.94 [style 0.0.1] /usr/include/stdlib.h:207:22: error: expected ';' after top level declarator
10:15.94 [style 0.0.1] fatal error: too many errors emitted, stopping now [-ferror-limit=]
10:15.94 [style 0.0.1] /usr/include/stdlib.h:96:1: error: unknown type name '__BEGIN_NAMESPACE_STD', err: true
10:15.94 [style 0.0.1] /usr/include/stdlib.h:98:1: error: expected unqualified-id, err: true
10:15.94 [style 0.0.1] /usr/include/stdlib.h:102:5: error: C++ requires a type specifier for all declarations, err: true
10:15.94 [style 0.0.1] /usr/include/stdlib.h:113:1: error: unknown type name '__END_NAMESPACE_STD', err: true
10:15.94 [style 0.0.1] /usr/include/stdlib.h:116:22: error: expected ';' after top level declarator, err: true
10:15.94 [style 0.0.1] /usr/include/stdlib.h:124:1: error: unknown type name '__END_NAMESPACE_C99', err: true
10:15.94 [style 0.0.1] /usr/include/stdlib.h:140:1: error: expected unqualified-id, err: true
10:15.94 [style 0.0.1] /usr/include/stdlib.h:143:1: error: unknown type name '__BEGIN_NAMESPACE_STD', err: true
10:15.94 [style 0.0.1] /usr/include/stdlib.h:145:1: error: expected unqualified-id, err: true
10:15.94 [style 0.0.1] /usr/include/stdlib.h:153:1: error: unknown type name '__END_NAMESPACE_STD', err: true
10:15.95 [style 0.0.1] /usr/include/stdlib.h:156:22: error: expected ';' after top level declarator, err: true
10:15.95 [style 0.0.1] /usr/include/stdlib.h:160:1: error: unknown type name '__END_NAMESPACE_C99', err: true
10:15.95 [style 0.0.1] /usr/include/stdlib.h:163:22: error: expected ';' after top level declarator, err: true
10:15.95 [style 0.0.1] /usr/include/stdlib.h:168:1: error: unknown type name '__END_NAMESPACE_STD', err: true
10:15.95 [style 0.0.1] /usr/include/stdlib.h:171:22: error: expected ';' after top level declarator, err: true
10:15.95 [style 0.0.1] /usr/include/stdlib.h:179:1: error: unknown type name '__END_NAMESPACE_C99', err: true
10:15.95 [style 0.0.1] /usr/include/stdlib.h:182:22: error: expected ';' after top level declarator, err: true
10:15.95 [style 0.0.1] /usr/include/stdlib.h:191:1: error: unknown type name '__END_NAMESPACE_STD', err: true
10:15.95 [style 0.0.1] /usr/include/stdlib.h:207:22: error: expected ';' after top level declarator, err: true
10:15.95 [style 0.0.1] fatal error: too many errors emitted, stopping now [-ferror-limit=], err: true

and

10:15.97 [style 0.0.1] thread 'main' panicked at 'Failed to generate bindings, flags: ["/var/tmp/build/firefox-5d24d4b6873d/obj-arm/dist/include/GeckoProfiler.h", "--rust-target", "1.25", "--bitfield-enum", "nsChangeHint", "--bitfield-enum", "mozilla::OriginFlags", "--rustified-enum", "nsCompatibility", "--rustified-enum", "mozilla::EffectCompositor_CascadeLevel", "--rustified-enum", "mozilla::SheetType", "--rustified-enum", "mozilla::dom::CallerType", "--rustified-enum", "mozilla::dom::IterationCompositeOperation", "--rustified-enum", "mozilla::dom::CompositeOperation", "--rustified-enum", "mozilla::InheritTarget", "--rustified-enum", "mozilla::css::DocumentMatchingFunction", "--rustified-enum", "mozilla::css::SheetParsingMode", "--rustified-enum", "mozilla::StyleContentType", "--rustified-enum", "nsStyleSVGOpacitySource", "--rustified-enum", "nsStyleUnit", "--rustified-enum", "nsCSSKeyword", "--rustified-enum", "mozilla::dom::Document_DocumentTheme", "--rustified-enum", "mozilla::dom::Document_Type", "--rustified-enum", "nsCSSUnit", "--rustified-enum", "nsCSSFontDesc", "--rustified-enum", "nsCSSPropertyID", "--rustified-enum", "nsCSSCounterDesc", "--rustified-enum", "nsMediaFeature_RangeType", "--rustified-enum", "nsMediaFeature_ValueType", "--rustified-enum", "nsresult", "--rustified-enum", "nsAtom_AtomKind", "--rustified-enum", "nsStyleImageLayers_LayerType", "--rustified-enum", "mozilla::ServoElementSnapshotFlags", "--rustified-enum", "mozilla::Side", "--rustified-enum", "mozilla::dom::PlaybackDirection", "--rustified-enum", "mozilla::dom::FillMode", "--rustified-enum", "mozilla::HalfCorner", "--rustified-enum", "mozilla::StyleFloatEdge", "--rustified-enum", "mozilla::StyleShapeRadius", "--rustified-enum", "mozilla::StyleWindowDragging", "--rustified-enum", "mozilla::StyleAnimationPlayState", "--rustified-enum", "mozilla::StyleOrient", "--rustified-enum", "mozilla::StyleBoxSizing", "--rustified-enum", "mozilla::StyleClear", "--rustified-enum", "mozilla::StyleColumnFill", "--rustified-enum", "mozilla::StyleColumnSpan", "--rustified-enum", "mozilla::StyleFloat", "--rustified-enum", "mozilla::StyleImageOrientation", "--rustified-enum", "mozilla::StyleUserModify", "--rustified-enum", "mozilla::StyleUserInput", "--rustified-enum", "mozilla::StyleBoxDirection", "--rustified-enum", "mozilla::StyleTextJustify", "--rustified-enum", "mozilla::StyleHyphens", "--rustified-enum", "mozilla::StyleShapeSourceType", "--rustified-enum", "mozilla::StyleBasicShapeType", "--rustified-enum", "nsStyleImageLayers_Size_DimensionType", "--rustified-enum", "mozilla::StyleStackSizing", "--rustified-enum", "mozilla::StyleBorderCollapse", "--rustified-enum", "mozilla::StyleBorderImageRepeat", "--rustified-enum", "mozilla::StyleBoxPack", "--rustified-enum", "mozilla::StyleBoxOrient", "--rustified-enum", "mozilla::StyleBoxAlign", "--rustified-enum", "mozilla::StyleUserFocus", "--rustified-enum", "mozilla::StyleUserSelect", "--rustified-enum", "mozilla::StyleImageLayerRepeat", "--rustified-enum", "mozilla::StyleImageLayerAttachment", "--rustified-enum", "mozilla::StyleBoxDecorationBreak", "--rustified-enum", "mozilla::StyleBorderStyle", "--rustified-enum", "mozilla::StyleRuleInclusion", "--rustified-enum", "mozilla::StyleGridTrackBreadth", "--rustified-enum", "mozilla::StyleOverscrollBehavior", "--rustified-enum", "mozilla::StyleOverflowAnchor", "--rustified-enum", "mozilla::StyleScrollbarWidth", "--rustified-enum", "mozilla::StyleWhiteSpace", "--rustified-enum", "mozilla::StyleTextRendering", "--rustified-enum", "mozilla::StyleColorAdjust", "--rustified-enum", "mozilla::StyleFlexDirection", "--rustified-enum", "nsStyleImageType", "--rustified-enum", "nsStyleSVGPaintType", "--rustified-enum", "nsStyleSVGFallbackType", "--rustified-enum", "nsINode_BooleanFlag", "--rustified-enum", "mozilla::PseudoStyleType", "--rustified-enum", "mozilla::LookAndFeel_ColorID", "--rustified-enum", "mozilla::LookAndFeel_FontID", "--rustified-enum", "nsStyleTransformMatrix::MatrixTransformOperator", "--rustified-enum", "mozilla::StyleGeometryBox", "--rustified-enum", "

What command did you use to build the Firefox part?

I suspect you are passing the wrong flags to BINDGEN_CFLAGS actually.

Would it be acceptable (at least for the purposes of an initial merge) for the Linux/ARMv7 target to use the clang/rustc packages from Debian Buster rather than the rbm-built equivalents (at least for building the firefox project)? I'm having no luck getting Firefox to build for Linux/ARMv7 with the self-built clang/rustc (seems to be some kind of weird path/sysroot issue), but I'm able to build it fine with the Buster-packaged compilers in rbm.

I assume that the main reason for building clang/rustc inside rbm is for bootstrapping-related security, but given that #32523 (moved) might make that obsolete anyway, I tend to figure that anything that builds working binaries is better than nothing for now.

Replying to JeremyRand:

Would it be acceptable (at least for the purposes of an initial merge) for the Linux/ARMv7 target to use the clang/rustc packages from Debian Buster rather than the rbm-built equivalents (at least for building the firefox project)? I'm having no luck getting Firefox to build for Linux/ARMv7 with the self-built clang/rustc (seems to be some kind of weird path/sysroot issue), but I'm able to build it fine with the Buster-packaged compilers in rbm.

So, both clang/rustc compiled yourself are problematic or just one of them? I.e. what happens if you take, say, the clang we compile with the buster-packaged rustc and vice versa? Is the issue still the same as in comment:2? If not what is the problem now?

Oh, and do you have a branch somewhere folks can poke at?

Good questions @gk, I'll do some further diagnosis and hopefully see what happens if one is from Buster and the other is rbm-built. Might not get to it for a few weeks, things are hectic here (and this port is mostly a weekend project for me). I don't have my current work pushed to NotABug yet, but will hopefully do so next weekend.

Updated branch at https://notabug.org/JeremyRand/tor-browser-build/src/armhf-esr68 (commit hash 224663ea2f7b5096f25ef7ed3e7170d21b16fbe5 in the armhf-esr68 branch). Successfully got the rbm-built clang working for building firefox for linux-arm. Still very much a WIP branch, but it does build working binaries.

Updated branch at https://notabug.org/JeremyRand/tor-browser-build/src/armhf-esr68 (commit hash bcd0e21237a0aeb598d1cf388b08d6f4e28eecd0). Builds binaries that work fine on my Asus C201 (running PrawnOS, based on Debian Buster). Binutils, Clang, Rust, Rust's stdlib, and cbindgen are all built from source in rbm, as with x86. libgcc/libstdc++ from Debian Buster's repos are used as build-time dependencies (removing them is likely to be nontrivial).

As discussed with Georg on IRC, I'm now requesting informal review, not for the purpose of determining eligibility to be merged at this time, but for the purpose of determining whether there are any likely conflicts between this branch and the eventual ESR78 port of Tor Browser. Georg informs me that this will definitely not be considered for merge until after ESR78 is active; that's totally fine from my point of view. I would appreciate any informal review that could point out things that might be merge blockers after ESR78 is active, or things that may need a rewrite/refactor to be compatible with ESR78. (However, I recognize that this is a low priority for the Tor Browser team, probably more so due to the recent staff losses, so I will accept whatever delays happen in reviewing this.) I am already aware that the use of network access while building the tor project is a blocker; I intend to fix that in the same way that I already fixed it in the firefox project (i.e. switching the build container to Debian Buster, and installing the relevant armhf packages via apt-get as part of container construction).

Cheers!

Trac:
Status: new to needs_review
Reviewer: N/A to gk
Keywords: TorBrowserTeam201904 deleted, TorBrowserTeam202005R added

Moving review tickets.

Trac:
Keywords: TorBrowserTeam202005R deleted, TorBrowserTeam202006R added

That's impressive work, thanks! I have three high-level questions for now:

IIRC the linux-cross target was the idea to abstract the cross-compilation part away from ARM and other platforms. Do I remember that right? Right now we have for that:

  linux-cross:
    var:
      linux-cross: 1
      container:
        arch: amd64
      # TODO: Maybe re-enable snowflake on linux-cross later?
      snowflake: 0
      # TODO: Maybe re-enable fteproxy on linux-cross later?
      fteproxy: 0

That does not strike me overly useful. fteproxy is gone. We'll want to have snowflake support on those platforms anyway and I doubt that the container won't be amd64. So, overall we don't seem to need that abstraction? Or am I missing something?

We build usually on latest Debian stable for our targets when we are cross-compiling. I think it is fine thinking about something else than building on Debian Wheezy for this project, in particular if it would make things easier (which it seems it would). We are only still on Wheezy for non-cross builds because of CentOS 6 and we'll soon move away from it because CentOS 6 will finally be dead later this year. We'll likely end up using Jessie or even Stretch (but very likely not Buster). I see you are using Glibc 2.26 in your cross-build which is even newer than what is included in Stretch. Thus, you probably already excluded a bunch of potentially still supported distro versions for armv7 (which is fine to me right now). So, a good step forward here we would thinking about what we/you want to support and what minimum Debian version that would mean (it's not Wheezy :)) and then start using that one (as I said above I bet newer Debian versions would greatly help in the cross-compilation effort)
We use essentially for all of our cross-compilation efforts clang nowadays and not GCC as the former has quite good support (which got better over the years) and GCC is not enough (anymore) to build for macOS/Winodws/Android. Would your linux arm work benefit from using clang here too? Or would that be harder? You are using clang for the Firefox part anyway (which is already the by far most complicated piece in our projects).

Add magic gitlab keyword.

Trac:
Keywords: N/A deleted, gitlab-tb-tor-browser-build added

moved to tpo/applications/tor-browser-build#32355

Tor Browser for Linux/ARMv7 (x86_64 build arch)

Child items 0

Activity