Opened 2 years ago

Last modified 5 months ago

#28325 new task

Use go 1.11 module versioning support

Reported by: boklm Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, tbb-maint, TorBrowserTeam202004, gitlab-tb-tor-browser-build
Cc: cohosh, boklm, tbb-team Actual Points:
Parent ID: Points: 3
Reviewer: sysrqb Sponsor:

Description

Go 1.11 is adding preliminary support for Go modules, replacing the GOPATH-based approach.

obfs4 is adding go.mod and go.sum files, which use the Go modules support to fetch the dependencies. We disable that in #28310 as the go libraries we build are not currently recognized as Go modules.

We should change how we build our Go libraries so that they can be used as Go modules.

Child Tickets

Change History (26)

comment:1 Changed 15 months ago by cohosh

Cc: cohosh added

We could really use this for the snowflake reproducible build. Our solution to the Windows build issue is to use a different webrtc library (which will probably be less painful for us down the road anyway). However, that library has 30+ dependencies and is a pain to build right now (see https://trac.torproject.org/projects/tor/ticket/28942#comment:39).

Is this item roadmapped or is there some information on how difficult this task would be?

comment:2 in reply to:  1 ; Changed 14 months ago by gk

Replying to cohosh:

We could really use this for the snowflake reproducible build. Our solution to the Windows build issue is to use a different webrtc library (which will probably be less painful for us down the road anyway). However, that library has 30+ dependencies and is a pain to build right now (see https://trac.torproject.org/projects/tor/ticket/28942#comment:39).

Is this item roadmapped or is there some information on how difficult this task would be?

It's not roadmapped as we did not have any prio to do this so far. Not sure how difficult it would be but my guess it not too difficult. Is that ticket blocking you? Or, asked differently: by when would you need this solved?

comment:4 in reply to:  2 Changed 14 months ago by cohosh

Replying to gk:

Replying to cohosh:

We could really use this for the snowflake reproducible build. Our solution to the Windows build issue is to use a different webrtc library (which will probably be less painful for us down the road anyway). However, that library has 30+ dependencies and is a pain to build right now (see https://trac.torproject.org/projects/tor/ticket/28942#comment:39).

Is this item roadmapped or is there some information on how difficult this task would be?

It's not roadmapped as we did not have any prio to do this so far. Not sure how difficult it would be but my guess it not too difficult. Is that ticket blocking you? Or, asked differently: by when would you need this solved?

I'm still unfamiliar with what exactly we need for reproducible builds. If all we need is a commit hash for every go library that snowflake depends on, I could probably spend a day or two writing a messy build script for the pion go version of snowflake. It's annoying but doable.

I think the pion version is becoming more and more necessary given #31446 and us wanting to roll out support for windows users. So my answer is: if a hacky build script as described above would work, it's not necessary. If not, then we are blocked on this for moving away from the messy chromium webrtc sources.

comment:5 Changed 14 months ago by boklm

I am still not sure how we should support fetching of sources for go modules in our build. In order to not block progress on using pion, if it has too many dependencies to manually create one project for each dependency, I think an other option is to run go mod vendor to fetch all dependencies, create a tarball of the vendor directory it created, then put the tarball online somewhere and use it as a dependency for the pion build.

comment:6 Changed 14 months ago by gk

Priority: MediumHigh

comment:7 Changed 12 months ago by gk

Cc: boklm added
Keywords: TorBrowserTeam201911 added

Let's pick this up for November/December.

Last edited 12 months ago by gk (previous) (diff)

comment:8 Changed 12 months ago by pili

Cc: tbb-team added
Owner: changed from tbb-team to boklm
Status: newassigned

Assigning tickets to boklm for the next few months

comment:9 Changed 11 months ago by boklm

It seems this issue is similar to #31588.

One option to fix this would be to add a special build step where we allow network access in the container, and run go mod vendor and create a vendor tarball that we later use in the build.

comment:10 Changed 11 months ago by pili

Keywords: TorBrowserTeam201912 added; TorBrowserTeam201911 removed

Moving tickets to December

comment:11 Changed 11 months ago by pili

Points: 3

comment:12 Changed 10 months ago by sysrqb

Keywords: TorBrowserTeam202001 added; TorBrowserTeam201912 removed

comment:13 in reply to:  9 Changed 9 months ago by sysrqb

Keywords: TorBrowserTeam202002 added; TorBrowserTeam202001 removed

Replying to boklm:

It seems this issue is similar to #31588.

One option to fix this would be to add a special build step where we allow network access in the container, and run go mod vendor and create a vendor tarball that we later use in the build.

I like this idea, and I think this is the easiest and most obvious solution. It should be reusable for rust, too.

dcf's comment is useful here, as well.

This FAQ https://github.com/golang/go/wiki/Modules#how-do-i-use-vendoring-with-modules-is-vendoring-going-away and the entry two below it are helpful, too.

My main concern with this is it becomes difficult detecting when a sub-dependency (dependency-of-a-dependency) is modified. This should not be a problem, because the chain of hashes for each dependency should prevent this. However, I'd prefer a defense-in-depth approach and not take unnecessary risks. We can pin the expected hash of the vendored tarball, and we can at least detect when any of the files change (this is assuming the output from go mod vendor is deterministic, of course), but it will require manual investigation to identify which dependency caused the failure.

I'm moving this to February, but we should really, really implement a solution for this soon. boklm, I'm putting this on your plate, too.

comment:14 Changed 9 months ago by boklm

An other option I think is to take the script dcf made for #28942 and improve it:
https://trac.torproject.org/projects/tor/attachment/ticket/28942/gomodtorbm

I think one possible improvement would be to define all go dependencies projects completely in the input_files section without creating them in the projects/ directory.

comment:15 Changed 9 months ago by sysrqb

Keywords: ReleaseTrainMigration added

Release Train Migration.

comment:16 Changed 8 months ago by pili

Keywords: TorBrowserTeam202003 added; TorBrowserTeam202002 removed

We are no longer in February, moving tickets

comment:17 Changed 7 months ago by pili

Sponsor: Sponsor58

comment:18 Changed 7 months ago by pili

Keywords: TorBrowserTeam202004 added; TorBrowserTeam202003 removed

We are no longer in March

comment:19 Changed 7 months ago by pili

Reviewer: sysrqb

comment:20 Changed 7 months ago by pili

Parent ID: #33659

comment:21 Changed 6 months ago by gk

Okay, just for the dependency update I created #33953 to disentangle some pieces that got moved into this ticket, too.

Last edited 5 months ago by gk (previous) (diff)

comment:22 Changed 6 months ago by cypherpunks

After upgrading to Go 1.14 only:
"Module support in the go command is now ready for production use"

comment:23 Changed 6 months ago by gaba

Owner: changed from boklm to tbb-team

Release all this tickets back into tbb-team.

comment:24 Changed 5 months ago by gk

Keywords: tbb-maint added; ReleaseTrainMigration removed

comment:25 Changed 5 months ago by gk

Parent ID: #33659
Priority: HighMedium
Sponsor: Sponsor58
Status: assignednew

comment:26 Changed 5 months ago by gk

Keywords: gitlab-tb-tor-browser-build added

Add magic gitlab keyword.

Note: See TracTickets for help on using tickets.