Opened 3 years ago

Closed 3 years ago

#20758 closed task (fixed)

Make Linux sandbox build deterministic

Reported by: gk Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Major Keywords: TorBrowserTeam201612R, GeorgKoppen201612
Cc: boklm Actual Points:
Parent ID: #19750 Points:
Reviewer: Sponsor:

Description

It seems there are multiple reproducibility issues with the tools we use for building the Linux sandbox (as mentioned in #20352):

-00b58fb0: 702e 676f 0000 2f74 6d70 2f67 6234 3537  p.go../tmp/gb457
-00b58fc0: 3938 3136 3238 2f67 6974 6875 622e 636f  981628/github.co
+00b58fb0: 702e 676f 0000 2f74 6d70 2f67 6231 3538  p.go../tmp/gb158
+00b58fc0: 3532 3636 3333 2f67 6974 6875 622e 636f  526633/github.co

But that's not the only one it seems: :(

-000001b0: 0300 0000 474e 5500 c50f b1fd 318e 7ecb  ....GNU.....1.~.
-000001c0: a61b 168e a2ec 32b6 0391 4dfb 0400 0000  ......2...M.....
-000001d0: 1400 0000 0300 0000 474e 5500 da00 559f  ........GNU...U.
-000001e0: d622 3c47 638e 0f4f 4ad1 3d2d 7e93 91e8  ."<Gc..OJ.=-~...
+000001b0: 0300 0000 474e 5500 0e56 fb68 814e 9f95  ....GNU..V.h.N..
+000001c0: b667 fc03 2349 a2d9 b0e8 922a 0400 0000  .g..#I.....*....
+000001d0: 1400 0000 0300 0000 474e 5500 6ece 2446  ........GNU.n.$F
+000001e0: 2790 d47f cfae 4b3b 3066 f673 b22b 146c  '.....K;0f.s.+.l

Child Tickets

Change History (7)

comment:1 Changed 3 years ago by gk

Testing the gb example on https://getgb.io/examples/sample-project/ resulted in identical binaries. So it seems there needs to be a certain complexity involved to trigger the problem.

Yawning had the hunch that part of the problem might be that go-bindata isn't laying out the source containing the static assets in a deterministic way.

comment:2 Changed 3 years ago by gk

Just doing make static-assets does not result in any differences. Thus, the issues seem to arise during gb build -tags $(GTK3TAG).

comment:3 Changed 3 years ago by gk

Cc: boklm added
Keywords: TorBrowserTeam201611R added; TorBrowserTeam201611 removed
Status: newneeds_review

Okay this was easier than thought: bug_20758 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_20758&id=e440393104ee04128a8660fecd0cce2f25bcaed8) in my tor-browser-bundle repo has a fix for review. It solves all the problems I encountered while doing several builds of the sandbox on the same machine and comparing the outputs. It is atop of the patches for #20352.

Last edited 3 years ago by gk (previous) (diff)

comment:4 in reply to:  3 ; Changed 3 years ago by yawning

Replying to gk:

Okay this was easier than thought: bug_20758 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_20758&id=e440393104ee04128a8660fecd0cce2f25bcaed8) in my tor-browser-bundle repo has a fix for review. It solves all the problems I encountered while doing several builds of the sandbox on the same machine and comparing the outputs. It is atop of the patches for #20352.

Looks ok to me.

If you're interested in nitpicking (though this probably won't get upstreamed):

  • Run go fmt prior to add/commit to reformat the source code (The commit message mentioned indentation).
  • I'd probably do something like:
      workdir := filepath.Join(os.TempDir(), "gb")
      err := os.Mkdir(workdir, 0777)
    

comment:5 Changed 3 years ago by gk

Keywords: TorBrowserTeam201612R added; TorBrowserTeam201611R removed

Moving our review tickets to December.

comment:6 Changed 3 years ago by gk

Keywords: GeorgKoppen201612 added; GeorgKoppen201611 removed

Moving my tickets

comment:7 in reply to:  4 Changed 3 years ago by gk

Resolution: fixed
Status: needs_reviewclosed

Replying to yawning:

Replying to gk:

Okay this was easier than thought: bug_20758 (https://gitweb.torproject.org/user/gk/tor-browser-bundle.git/commit/?h=bug_20758&id=e440393104ee04128a8660fecd0cce2f25bcaed8) in my tor-browser-bundle repo has a fix for review. It solves all the problems I encountered while doing several builds of the sandbox on the same machine and comparing the outputs. It is atop of the patches for #20352.

Looks ok to me.

If you're interested in nitpicking (though this probably won't get upstreamed):

  • Run go fmt prior to add/commit to reformat the source code (The commit message mentioned indentation).
  • I'd probably do something like:
      workdir := filepath.Join(os.TempDir(), "gb")
      err := os.Mkdir(workdir, 0777)
    

Yes, I am, thanks. I fixed both issues and pushed the result to master (commit 27cf6fc8837ab98cb702d8c4db093e7b8f9d1c07).

Note: See TracTickets for help on using tickets.