Opened 5 months ago

Last modified 43 hours ago

#31588 new task

Be smarter about vendoring for Rust projects

Reported by: gk Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, TorBrowserTeam202002
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

In #30490 we added cbindgen to our projects which resulted in a large extra archive with all the vendored stuff that is specified. However, we probably don't need a lot of that stuff (like all the big Windows .a files). We should figure out a smarter way of handling that than just shipping everything.

Child Tickets

Change History (8)

comment:1 Changed 2 months ago by gk

Summary: Be smarter about vendoring for cbindgenBe smarter about vendoring for Rust projects

#32436 has similar problems. So, let's be more generic and make this ticket into getting to a proper vendoring process/strategy.

comment:2 Changed 2 months ago by boklm

If Mozilla is vendoring most of the rust dependencies we need, I think we could generate a tarball from tor-browser.git/third_party/rust, and include it in the cbindgen project, and in other rust projects that we build. That way we don't need to manually update a tarball of vendored projects.

If we want to avoid generating a new tarball (which will probably involve generating a tarball containing all the firefox tree, extracting it, and generating a new tarball from the third_party/rust directory only), we can re-use the src-firefox-$version.tar.xz tarball which we already generate.

comment:3 in reply to:  2 ; Changed 2 months ago by gk

Replying to boklm:

If Mozilla is vendoring most of the rust dependencies we need, I think we could generate a tarball from tor-browser.git/third_party/rust, and include it in the cbindgen project, and in other rust projects that we build. That way we don't need to manually update a tarball of vendored projects.

If we want to avoid generating a new tarball (which will probably involve generating a tarball containing all the firefox tree, extracting it, and generating a new tarball from the third_party/rust directory only), we can re-use the src-firefox-$version.tar.xz tarball which we already generate.

Yeah, I've thought about that. The problem here is, though, those the projects in question are external ones which are *not* needed to build Mozilla's Rust code. Thus, Mozilla has not vendored them in but has them rather as a build dependency. For cbindgen we could think about using Debian packages at least once we move don't have versions anymore where cbindgen is not shipped for in a sufficiently recent version. For lucetc this option is not available.

comment:4 in reply to:  3 ; Changed 2 months ago by boklm

Replying to gk:

Replying to boklm:

If Mozilla is vendoring most of the rust dependencies we need, I think we could generate a tarball from tor-browser.git/third_party/rust, and include it in the cbindgen project, and in other rust projects that we build. That way we don't need to manually update a tarball of vendored projects.

If we want to avoid generating a new tarball (which will probably involve generating a tarball containing all the firefox tree, extracting it, and generating a new tarball from the third_party/rust directory only), we can re-use the src-firefox-$version.tar.xz tarball which we already generate.

Yeah, I've thought about that. The problem here is, though, those the projects in question are external ones which are *not* needed to build Mozilla's Rust code. Thus, Mozilla has not vendored them in but has them rather as a build dependency. For cbindgen we could think about using Debian packages at least once we move don't have versions anymore where cbindgen is not shipped for in a sufficiently recent version. For lucetc this option is not available.

Hmm, I don't understand what you mean here. When I look at the content of cbindgen-vendor.tar.bz2, all the directories included in it are also present in tor-browser.git/third_party/rust. Or are you talking about other projects than cbindgen?

comment:5 in reply to:  4 Changed 2 months ago by gk

Replying to boklm:

Replying to gk:

Replying to boklm:

If Mozilla is vendoring most of the rust dependencies we need, I think we could generate a tarball from tor-browser.git/third_party/rust, and include it in the cbindgen project, and in other rust projects that we build. That way we don't need to manually update a tarball of vendored projects.

If we want to avoid generating a new tarball (which will probably involve generating a tarball containing all the firefox tree, extracting it, and generating a new tarball from the third_party/rust directory only), we can re-use the src-firefox-$version.tar.xz tarball which we already generate.

Yeah, I've thought about that. The problem here is, though, those the projects in question are external ones which are *not* needed to build Mozilla's Rust code. Thus, Mozilla has not vendored them in but has them rather as a build dependency. For cbindgen we could think about using Debian packages at least once we move don't have versions anymore where cbindgen is not shipped for in a sufficiently recent version. For lucetc this option is not available.

Hmm, I don't understand what you mean here. When I look at the content of cbindgen-vendor.tar.bz2, all the directories included in it are also present in tor-browser.git/third_party/rust. Or are you talking about other projects than cbindgen?

I am talking right now about cbindgen and lucetc. But the specific third_party/rust part I had in mind is only applying to the former.

Yes, the packages are there but not all the versions are the same as cargo vendor gives us. This, the risk is high that either compilation fails or some other weird behavior would happen. If the packages available *and* the versions they are in matched, I agree, using the code from third_party/rust would work. But, alas, that's not the case.

comment:6 Changed 2 months ago by boklm

If we trust that cargo vendor will provide reproducible output, and is correctly checking downloaded files, then maybe we could add some step in the build where we allow network access in the container, in order to run cargo vendor and generate a vendor tarball.

comment:7 Changed 2 months ago by sysrqb

Keywords: TorBrowserTeamTriaged added

Marking as triaged, but not putting it on the roadmap yet.

comment:8 in reply to:  6 Changed 43 hours ago by sysrqb

Keywords: TorBrowserTeam202002 added; TorBrowserTeamTriaged removed

Replying to boklm:

If we trust that cargo vendor will provide reproducible output, and is correctly checking downloaded files, then maybe we could add some step in the build where we allow network access in the container, in order to run cargo vendor and generate a vendor tarball.

Moving this to February. If we make the above assumptions (and they need to be confirmed), then after #28325 is solved this ticket should (hopefully) follow soon after.

Note: See TracTickets for help on using tickets.