Opened 3 months ago

Closed 6 weeks ago

Last modified 8 days ago

#27827 closed defect (fixed)

Reproducibility issue of the snowflake osx64 build

Reported by: boklm Owned by: tbb-team
Priority: Very High Milestone:
Component: Obfuscation/Snowflake Version:
Severity: Normal Keywords: tbb-rbm, TorBrowserTeam201810R
Cc: dcf, arthuredelstein, arlolra, boklm Actual Points:
Parent ID: #19001 Points:
Reviewer: Sponsor: Sponsor19

Description (last modified by dcf)

The build of Snowflake for MacOS is often producing the same result, but not always.

Arthur has been rebuilding Snowflake 8 times, with 4 different results:
https://gist.github.com/arthuredelstein/73860df088c565ea0b2ca6eef586063a

fish script:

for x in (seq 8)
    rm out/snowflake/snowflake-6077141f4aff-osx-x86_64-3b578d.tar.gz
    ./rbm/rbm build snowflake  --target alpha --target torbrowser-osx-x86_64
    tar xvf out/snowflake/snowflake-6077141f4aff-osx-x86_64-3b578d.tar.gz
    echo (sha256sum ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
end

Results:

b060b42cfd0c8fb2781dbb0fd45d42804dbb414473fec0597d9c2fb7d6d12aa8  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
22557c38d913e478e480dd3581efc00019fe2989c4273d9207f1719c34b6e399  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
22557c38d913e478e480dd3581efc00019fe2989c4273d9207f1719c34b6e399  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
6d008bc7d29e8543608491b67d4b11da7bd6589741d9f52ac5fd50dd39d84f29  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client

Child Tickets

Change History (17)

comment:2 Changed 3 months ago by boklm

For the Tor Browser 7.5a2 release, as we don't have time to fix this bug for this release, we will be using the 1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3 version of snowflake-client (which was also the version used in the previous alpha releases).

comment:3 Changed 3 months ago by dcf

Description: modified (diff)

The difference is different ordering of some symbols/path in the snowflake-client binary. Notice how (except for cgo-gcc-prolog), it's different orderings of three blocks, color coded below.

Notice also that each block contains a /tmp/go-buildXXXXXXXXX string. These come from the snowflake build descriptor: the XXXXXXXXX overwrites a random number. What I suspect is happening is, the go compiler is generating three distinct random /tmp/go-buildXXXXXXXXX paths and sorting the blocks on them; then the build descriptor overwrites them all. Therefore each build randomly gets one of the six possible permutations.

The overwriting is a workaround for the upstream Go bug #9206, which is now marked closed. The first thing I would try is upgrading to a newer Go and seeing if it just gets fixed.

1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3

22557c38d913e478e480dd3581efc00019fe2989c4273d9207f1719c34b6e399

6d008bc7d29e8543608491b67d4b11da7bd6589741d9f52ac5fd50dd39d84f29

/tmp/go-build/net/_obj/
_cgo_export.c
/tmp/go-buildXXXXXXXXX/net/_obj/_cgo_export.o
__cgo_9f1b05c52f96_Cfunc__Cmalloc
cgo_resnew.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_resnew.cgo2.o
__cgo_9f1b05c52f96_C2func_getnameinfo
cgo-gcc-prolog
__cgo_9f1b05c52f96_Cfunc_getnameinfo
cgo_unix.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_unix.cgo2.o
__cgo_9f1b05c52f96_C2func_getaddrinfo
__cgo_9f1b05c52f96_Cfunc_free
__cgo_9f1b05c52f96_Cfunc_freeaddrinfo
__cgo_9f1b05c52f96_Cfunc_gai_strerror
__cgo_9f1b05c52f96_Cfunc_getaddrinfo
/tmp/go-build/crypto/x509/_obj/
root_cgo_darwin.cgo2.c
/tmp/go-buildXXXXXXXXX/crypto/x509/_obj/root_cgo_darwin.cgo2.o
_FetchPEMRoots_MountainLion
/var/tmp/dist/go/src/crypto/x509/root_cgo_darwin.go
_useOldCode
_FetchPEMRoots
__cgo_62033c69288a_Cfunc_CFDataGetBytePtr
__cgo_62033c69288a_Cfunc_CFDataGetLength
__cgo_62033c69288a_Cfunc_CFRelease
__cgo_62033c69288a_Cfunc_FetchPEMRoots
/var/tmp/dist/go/src/runtime/cgo/
gcc_context.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_context.o
__cgo_release_context
gcc_darwin_amd64.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_darwin_amd64.o
_x_cgo_init
__cgo_sys_thread_start
_threadentry
_k1
gcc_libinit.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_libinit.o
_x_cgo_sys_thread_create
__cgo_try_pthread_create
__cgo_wait_runtime_init_done
_x_cgo_notify_runtime_init_done
_x_cgo_set_context_function
__cgo_get_context_function
_runtime_init_mu
_runtime_init_cond
_runtime_init_done
_cgo_context_function
gcc_setenv.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_setenv.o
_x_cgo_setenv
_x_cgo_unsetenv
gcc_util.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_util.o
_x_cgo_thread_start
gcc_amd64.S
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_amd64.o
_crosscall_amd64
/tmp/go-build/net/_obj/
_cgo_export.c
/tmp/go-buildXXXXXXXXX/net/_obj/_cgo_export.o
__cgo_9f1b05c52f96_Cfunc__Cmalloc
cgo_resnew.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_resnew.cgo2.o
__cgo_9f1b05c52f96_C2func_getnameinfo
cgo-gcc-prolog
__cgo_9f1b05c52f96_Cfunc_getnameinfo
cgo_unix.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_unix.cgo2.o
__cgo_9f1b05c52f96_C2func_getaddrinfo
__cgo_9f1b05c52f96_Cfunc_free
__cgo_9f1b05c52f96_Cfunc_freeaddrinfo
__cgo_9f1b05c52f96_Cfunc_gai_strerror
__cgo_9f1b05c52f96_Cfunc_getaddrinfo
/var/tmp/dist/go/src/runtime/cgo/
gcc_context.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_context.o
__cgo_release_context
gcc_darwin_amd64.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_darwin_amd64.o
_x_cgo_init
__cgo_sys_thread_start
_threadentry
_k1
gcc_libinit.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_libinit.o
_x_cgo_sys_thread_create
__cgo_try_pthread_create
__cgo_wait_runtime_init_done
_x_cgo_notify_runtime_init_done
_x_cgo_set_context_function
__cgo_get_context_function
_runtime_init_mu
_runtime_init_cond
_runtime_init_done
_cgo_context_function
gcc_setenv.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_setenv.o
_x_cgo_setenv
_x_cgo_unsetenv
gcc_util.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_util.o
_x_cgo_thread_start
gcc_amd64.S
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_amd64.o
_crosscall_amd64
/tmp/go-build/crypto/x509/_obj/
root_cgo_darwin.cgo2.c
/tmp/go-buildXXXXXXXXX/crypto/x509/_obj/root_cgo_darwin.cgo2.o
_FetchPEMRoots_MountainLion
/var/tmp/dist/go/src/crypto/x509/root_cgo_darwin.go
_useOldCode
_FetchPEMRoots
__cgo_62033c69288a_Cfunc_CFDataGetBytePtr
__cgo_62033c69288a_Cfunc_CFDataGetLength
__cgo_62033c69288a_Cfunc_CFRelease
__cgo_62033c69288a_Cfunc_FetchPEMRoots
/var/tmp/dist/go/src/runtime/cgo/
gcc_context.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_context.o
__cgo_release_context
gcc_darwin_amd64.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_darwin_amd64.o
_x_cgo_init
__cgo_sys_thread_start
_threadentry
_k1
gcc_libinit.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_libinit.o
_x_cgo_sys_thread_create
__cgo_try_pthread_create
__cgo_wait_runtime_init_done
_x_cgo_notify_runtime_init_done
_x_cgo_set_context_function
__cgo_get_context_function
_runtime_init_mu
_runtime_init_cond
_runtime_init_done
_cgo_context_function
gcc_setenv.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_setenv.o
_x_cgo_setenv
_x_cgo_unsetenv
gcc_util.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_util.o
_x_cgo_thread_start
gcc_amd64.S
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_amd64.o
_crosscall_amd64
/tmp/go-build/crypto/x509/_obj/
root_cgo_darwin.cgo2.c
/tmp/go-buildXXXXXXXXX/crypto/x509/_obj/root_cgo_darwin.cgo2.o
_FetchPEMRoots_MountainLion
/var/tmp/dist/go/src/crypto/x509/root_cgo_darwin.go
_useOldCode
_FetchPEMRoots
__cgo_62033c69288a_Cfunc_CFDataGetBytePtr
cgo-gcc-prolog
__cgo_62033c69288a_Cfunc_CFDataGetLength
__cgo_62033c69288a_Cfunc_CFRelease
__cgo_62033c69288a_Cfunc_FetchPEMRoots
/tmp/go-build/net/_obj/
_cgo_export.c
/tmp/go-buildXXXXXXXXX/net/_obj/_cgo_export.o
__cgo_9f1b05c52f96_Cfunc__Cmalloc
cgo_resnew.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_resnew.cgo2.o
__cgo_9f1b05c52f96_C2func_getnameinfo
__cgo_9f1b05c52f96_Cfunc_getnameinfo
cgo_unix.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_unix.cgo2.o
__cgo_9f1b05c52f96_C2func_getaddrinfo
__cgo_9f1b05c52f96_Cfunc_free
__cgo_9f1b05c52f96_Cfunc_freeaddrinfo
__cgo_9f1b05c52f96_Cfunc_gai_strerror
__cgo_9f1b05c52f96_Cfunc_getaddrinfo
Last edited 3 months ago by dcf (previous) (diff)

comment:4 Changed 3 months ago by gk

Keywords: tbb-rbm TorBrowserTeam201809 added
Priority: MediumHigh

comment:5 Changed 2 months ago by gk

Keywords: TorBrowserTeam201810 added; TorBrowserTeam201809 removed

Moving tickets to October

comment:6 Changed 2 months ago by gk

Cc: tbb-team removed
Owner: set to tbb-team
Priority: HighVery High
Status: newassigned

comment:7 Changed 2 months ago by boklm

In branch bug_27827, I made a patch to build with go 1.11:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27827&id=ca1921e4566ef28d134b6fe88dea3c0df1d6ad1a

The build finished correctly, but I didn't try yet to do multiple builds to see if we get the same binary.

However it seems the sed on /tmp/go-build000000000 is still needed as I didn't get an error during the build.

comment:8 Changed 8 weeks ago by dcf

I did several builds with boklm's go1.11 patch and analyzed the differences.

Now, there are no /tmp/go-build000000000 paths anymore (where 000000000 stands for random digits), but there is a single /tmp/go-link-000000000/go.o. This shouldn't be a problem, because the sed substitution rewrites it to /tmp/go-link-XXXXXXXXX/go.o and there is only one so there is no problem with ordering.

But there's another difference in the binaries, the Go build ID. Here are sample values that are baked into the binary:

Go build ID: "kTjwWGrY9n3mGwOUpwVM/N2WdKU5WHR85aFCEJdn9/aRTcrk1SBq_sas7IMHGu/naljN2FzcmFC20pFl5NO"
Go build ID: "kTjwWGrY9n3mGwOUpwVM/N2WdKU5WHR85aFCEJdn9/aRTcrk1SBq_sas7IMHGu/XQTshh9tsXMqYLXM55kx"
Go build ID: "kTjwWGrY9n3mGwOUpwVM/N2WdKU5WHR85aFCEJdn9/aRTcrk1SBq_sas7IMHGu/Etnddau3jLCah3D-CG_-"

I found some documentation that says the build ID is, in part, a hash of all the input filenames. So I suspect it's including the random /tmp/go-link-000000000/go.o in the hash, because the build ID was different in all 9 builds I did.

One option is just to do another sed substitution on the build ID. But before that, I'm going to try building with go1.10 to see if this is a new bug introduced in go1.11.

Potentially related Go issues: GH#16860 GH#28008 GH#22382.

EDIT: same symptoms with go1.10.4.

Last edited 8 weeks ago by dcf (previous) (diff)

comment:9 Changed 8 weeks ago by dcf

Status: assignedneeds_review

I have a mostly working branch bug_27827.

It updates to go1.11.1. This update removed all the /tmp/go-build000000000 paths that we previously had to overwrite. However it left a single /tmp/go-link-000000000/go.o path. This path was also affecting the Go build ID (a hash of the build inputs), so it couldn't be fixed just by overwriting. However, I found a -tmpdir flag that lets us replace the random path with a static path, and that fixed it.

There is still a reproducibility problem, which is the gzip timestamp (bytes 4, 5, 6, 7) in the tar.gz output file differ across builds. I.e., the complete diffoscope output is

--- bug_27827/tmpdir/snowflake.tar.gz.1
+++ bug_27827/tmpdir/snowflake.tar.gz.2
├── metadata
│ @@ -1 +1 @@
│ -gzip compressed data, last modified: Wed Oct 17 16:30:51 2018, from Unix
│ +gzip compressed data, last modified: Wed Oct 17 16:34:52 2018, from Unix

I'm guessing that this is some other problem unrelated to go or snowflake.

comment:10 Changed 8 weeks ago by gk

Keywords: TorBrowserTeam201810R added; TorBrowserTeam201810 removed

comment:11 Changed 8 weeks ago by gk

Resolution: fixed
Status: needs_reviewclosed

Looks good to me. I merged the branch with commit 889be4318d80f11f2b5b2b845ab11868149c48f1 into tor-browser-build's master.

comment:12 Changed 8 weeks ago by gk

Cc: boklm added
Keywords: TorBrowserTeam201810 added; TorBrowserTeam201810R removed
Resolution: fixed
Status: closedreopened

That patch is actually busting our Linux builds, reopening:

# github.com/keroserene/go-webrtc
peerconnection.cc:374: error: undefined reference to 'webrtc::JsepSessionDescription::JsepSessionDescription(std::string const&)'
peerconnection.cc:377: error: undefined reference to 'webrtc::SdpDeserialize(std::string const&, webrtc::JsepSessionDescription*, webrtc::SdpParseError*)'
peerconnection.cc:416: error: undefined reference to 'webrtc::CreateIceCandidate(std::string const&, int, std::string const&, webrtc::SdpParseError*)'
peerconnection.cc:55: error: undefined reference to 'rtc::Thread::SetName(std::string const&, void const*)'
peerconnection.cc:56: error: undefined reference to 'rtc::Thread::SetName(std::string const&, void const*)'
./include/webrtc/base/array_view.h:141: error: undefined reference to 'rtc::FatalMessage::FatalMessage(char const*, int, std::string*)'
collect2: error: ld returned 1 exit status

comment:13 Changed 8 weeks ago by dcf

Sorry for not checking the linux build. I'll have a little time to look at this on Tuesday, but if I can't figure it out then, it'll be another week before I can look at it.

comment:14 Changed 6 weeks ago by dcf

Keywords: TorBrowserTeam201810R added; TorBrowserTeam201810 removed
Status: reopenedneeds_review

Please see bug_27827_v2. The difference is that now we set -D_GLIBCXX_USE_CXX11_ABI=1 in snowflake/build, just like in go-webrtc/config. (The error message in comment:12 was caused by incompatible ABIs; i.e. std::string versus std::__cxx11::basic_string.) I don't know why we need to set it in snowflake/build now (or alternatively, why we didn't have to set it with go1.9).

comment:15 Changed 6 weeks ago by gk

Resolution: fixed
Status: needs_reviewclosed

Thanks. Works now on my build machines. Merged to master (commit 1ca5a195c785f70cac791117a4e12d49482a206f and 28abf57bd5bd307262741ac679a80e98aa389d20).

comment:16 Changed 8 days ago by pili

Parent ID: #19001

comment:17 Changed 8 days ago by pili

Sponsor: Sponsor19
Note: See TracTickets for help on using tickets.