Opened 2 months ago

Closed 5 weeks ago

#32520 closed defect (fixed)

Output of go project contains nonreproducible datetime values

Reported by: JeremyRand Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, TorBrowserTeam201912R
Cc: Actual Points:
Parent ID: Points:
Reviewer: boklm Sponsor:

Description

Steps to reproduce:

  1. Run ./rbm/rbm build go --target release --target torbrowser-linux-x86_64 twice.
  2. Compare the hashes of the results.

Expected behavior:

The hashes should be reproducible.

Observed behavior:

The hashes are not reproducible.

Other info:

I'm attaching a diffoscope. Most of the nonreproducibility seems to be due to datetime values. I suspect, but have not verified, that these datetime values are being inserted by the (currently unmaintained) Go 1.4.x compiler, and that therefore we can't expect an upstream fix. libfaketime seems like the most straightforward way to fix the issue. Would a patch be accepted that uses libfaketime to make the datetime values in the go project's output reproducible?

Child Tickets

Attachments (1)

go-1.12.13-0189d4.tar.gz.diffoscope (1.1 MB) - added by JeremyRand 2 months ago.

Download all attachments as: .zip

Change History (13)

Changed 2 months ago by JeremyRand

comment:1 Changed 2 months ago by gk

Keywords: tbb-rbm added

comment:2 Changed 2 months ago by sysrqb

Keywords: TorBrowserTeam201911 added
Status: newneeds_information

This isn't a duplicate of #32053?

comment:3 in reply to:  2 Changed 2 months ago by sysrqb

Replying to sysrqb:

This isn't a duplicate of #32053?

Hrm. No, that doesn't make sense (rust vs. go).

You're saying that you compared the build artifacts of the go compiler from two builds and found they aren't reproducible, while the resulting tor browser is reproducible?

comment:4 Changed 2 months ago by boklm

Since the go compiler that we build is only used locally, and not distributed, does it matter if the build is not reproducible? Or do you have a different use case where this matters?

comment:5 Changed 2 months ago by JeremyRand

You're saying that you compared the build artifacts of the go compiler from two builds and found they aren't reproducible, while the resulting tor browser is reproducible?

@sysrqb yes, the Go compiler isn't building reproducibly, while the projects that use it as an input do build reproducibly.

Since the go compiler that we build is only used locally, and not distributed, does it matter if the build is not reproducible? Or do you have a different use case where this matters?

@boklm It's certainly much less of a big deal than if the final Tor Browser output were nonreproducible, but there are still cases where it makes a difference.

For example, let's say that Alice and Bob are trying to build Tor Browser, and they get different hashes for obfs4. If they get the same hashes for the Go compiler, then that helps them narrow down the bug to the obfs4 build scripts, whereas if the Go compiler isn't reproducible either, then they have to consider the possibility that the Go compiler's nondeterminism is somehow leaking into obfs4.

As another example, if 2 different communities (let's say Tor and Namecoin) are both building binaries with Go, then making the Go compiler reproducible allows the Tor and Namecoin communities to cross-validate that the Go compiler is reproducible, which means more people will be attesting to the non-backdoored-ness of the Go compiler. (Given that today there are very few people other than employees of The Tor Project who are trying to reproduce Tor Browser hashes, this would be a useful improvement.)

As a third example, it's conceivable that some users may want to download binaries of the Go compiler without trusting that Google hasn't backdoored them; if the Go compiler can be built reproducibly, then this enables any 3rd party to provide downloads of those reproducible binaries without becoming a trusted 3rd party.

So yes, there are cases where it matters, though admittedly they are not as major as the standard use case of "allow 2 Tor Project employees to reproduce the final Tor Browser binaries", where you're correct that it doesn't matter.

comment:6 Changed 2 months ago by JeremyRand

(By the way, just to be clear -- I recognize that this is not the most urgent thing for you guys, which is why I asked if you'd accept a patch for it, rather than asking you to write a patch yourselves. :) )

comment:7 in reply to:  6 Changed 2 months ago by gk

Status: needs_informationnew

Replying to JeremyRand:

(By the way, just to be clear -- I recognize that this is not the most urgent thing for you guys, which is why I asked if you'd accept a patch for it, rather than asking you to write a patch yourselves. :) )

Yes, by all means, if you would like to write a patch, this would be very much appreciated. Please do so. :)

comment:8 Changed 7 weeks ago by JeremyRand

Most of the nonreproducibility seems to be in the go/pkg/obj/go-build directory. Google's official binary packages don't even include this directory. Is that directory necessary for anything? If it's not necessary, would a patch be accepted that removes that directory from the output?

comment:9 Changed 7 weeks ago by JeremyRand

Fix at https://notabug.org/JeremyRand/tor-browser-build/src/reproducible-go (Git commit hash 0ab49005efd962074e80cafb52fe4dd9cb286229). Tested to work fine for the linux-x86_64 target (builds without errors and the resulting Tor Browser connects to obfs4 without issues). I can't think of any reason it would cause problems for other targets, but haven't tested on them yet. A side benefit is that it decreases the size of the go project's output by circa 34 MiB.

comment:10 Changed 7 weeks ago by gk

Keywords: TorBrowserTeam201912R added; TorBrowserTeam201911 removed
Status: newneeds_review

comment:11 Changed 7 weeks ago by pili

Reviewer: boklm

boklm to review

comment:12 in reply to:  9 Changed 5 weeks ago by boklm

Resolution: fixed
Status: needs_reviewclosed

Replying to JeremyRand:

Fix at https://notabug.org/JeremyRand/tor-browser-build/src/reproducible-go (Git commit hash 0ab49005efd962074e80cafb52fe4dd9cb286229). Tested to work fine for the linux-x86_64 target (builds without errors and the resulting Tor Browser connects to obfs4 without issues). I can't think of any reason it would cause problems for other targets, but haven't tested on them yet. A side benefit is that it decreases the size of the go project's output by circa 34 MiB.

This looks good to me, thanks. I merged it to master with commit 0c7a319ca0be61832c3c1d0cec769f0feb089c8d.

Note: See TracTickets for help on using tickets.