Reproducibility issue of the snowflake osx64 build

added TorBrowserTeam201810R component::circumvention/snowflake owner::tbb-team parent::19001 priority::very high resolution::fixed severity::normal sponsor::19 status::closed tbb-rbm type::defect labels

I uploaded two versions of the snowflake-client build: https://people.torproject.org/~boklm/tmp/bug_27827/1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3.tar.gz https://people.torproject.org/~boklm/tmp/bug_27827/6d008bc7d29e8543608491b67d4b11da7bd6589741d9f52ac5fd50dd39d84f29.tar.gz

For the Tor Browser 7.5a2 release, as we don't have time to fix this bug for this release, we will be using the 1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3 version of snowflake-client (which was also the version used in the previous alpha releases).

The difference is different ordering of some symbols/path in the snowflake-client binary. Notice how (except for cgo-gcc-prolog), it's different orderings of three blocks, color coded below.

Notice also that each block contains a /tmp/go-buildXXXXXXXXX string. These come from the snowflake build descriptor: the XXXXXXXXX overwrites a random number. What I suspect is happening is, the go compiler is generating three distinct random /tmp/go-buildXXXXXXXXX paths and sorting the blocks on them; then the build descriptor overwrites them all. Therefore each build randomly gets one of the six possible permutations.

The overwriting is a workaround for the upstream Go bug #9206, which is now marked closed. The first thing I would try is upgrading to a newer Go and seeing if it just gets fixed.

1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3

22557c38d913e478e480dd3581efc00019fe2989c4273d9207f1719c34b6e399

6d008bc7d29e8543608491b67d4b11da7bd6589741d9f52ac5fd50dd39d84f29

|-----------------------------------

{{{#!html
<small>
<pre style="background-color: red">
/tmp/go-build/net/_obj/
_cgo_export.c
/tmp/go-buildXXXXXXXXX/net/_obj/_cgo_export.o
__cgo_9f1b05c52f96_Cfunc__Cmalloc
cgo_resnew.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_resnew.cgo2.o
__cgo_9f1b05c52f96_C2func_getnameinfo
<span style="background-color: gray">cgo-gcc-prolog</span>
__cgo_9f1b05c52f96_Cfunc_getnameinfo
cgo_unix.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_unix.cgo2.o
__cgo_9f1b05c52f96_C2func_getaddrinfo
__cgo_9f1b05c52f96_Cfunc_free
__cgo_9f1b05c52f96_Cfunc_freeaddrinfo
__cgo_9f1b05c52f96_Cfunc_gai_strerror
__cgo_9f1b05c52f96_Cfunc_getaddrinfo
</pre><pre style="background-color: green">
/tmp/go-build/crypto/x509/_obj/
root_cgo_darwin.cgo2.c
/tmp/go-buildXXXXXXXXX/crypto/x509/_obj/root_cgo_darwin.cgo2.o
_FetchPEMRoots_MountainLion
/var/tmp/dist/go/src/crypto/x509/root_cgo_darwin.go
_useOldCode
_FetchPEMRoots
__cgo_62033c69288a_Cfunc_CFDataGetBytePtr
__cgo_62033c69288a_Cfunc_CFDataGetLength
__cgo_62033c69288a_Cfunc_CFRelease
__cgo_62033c69288a_Cfunc_FetchPEMRoots
</pre><pre style="background-color: lightblue">
/var/tmp/dist/go/src/runtime/cgo/
gcc_context.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_context.o
__cgo_release_context
gcc_darwin_amd64.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_darwin_amd64.o
_x_cgo_init
__cgo_sys_thread_start
_threadentry
_k1
gcc_libinit.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_libinit.o
_x_cgo_sys_thread_create
__cgo_try_pthread_create
__cgo_wait_runtime_init_done
_x_cgo_notify_runtime_init_done
_x_cgo_set_context_function
__cgo_get_context_function
_runtime_init_mu
_runtime_init_cond
_runtime_init_done
_cgo_context_function
gcc_setenv.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_setenv.o
_x_cgo_setenv
_x_cgo_unsetenv
gcc_util.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_util.o
_x_cgo_thread_start
gcc_amd64.S
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_amd64.o
_crosscall_amd64
</pre>

}}}

{{{#!html
<small>
<pre style="background-color: red">
/tmp/go-build/net/_obj/
_cgo_export.c
/tmp/go-buildXXXXXXXXX/net/_obj/_cgo_export.o
__cgo_9f1b05c52f96_Cfunc__Cmalloc
cgo_resnew.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_resnew.cgo2.o
__cgo_9f1b05c52f96_C2func_getnameinfo
<span style="background-color: gray">cgo-gcc-prolog</span>
__cgo_9f1b05c52f96_Cfunc_getnameinfo
cgo_unix.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_unix.cgo2.o
__cgo_9f1b05c52f96_C2func_getaddrinfo
__cgo_9f1b05c52f96_Cfunc_free
__cgo_9f1b05c52f96_Cfunc_freeaddrinfo
__cgo_9f1b05c52f96_Cfunc_gai_strerror
__cgo_9f1b05c52f96_Cfunc_getaddrinfo
</pre><pre style="background-color: lightblue">
/var/tmp/dist/go/src/runtime/cgo/
gcc_context.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_context.o
__cgo_release_context
gcc_darwin_amd64.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_darwin_amd64.o
_x_cgo_init
__cgo_sys_thread_start
_threadentry
_k1
gcc_libinit.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_libinit.o
_x_cgo_sys_thread_create
__cgo_try_pthread_create
__cgo_wait_runtime_init_done
_x_cgo_notify_runtime_init_done
_x_cgo_set_context_function
__cgo_get_context_function
_runtime_init_mu
_runtime_init_cond
_runtime_init_done
_cgo_context_function
gcc_setenv.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_setenv.o
_x_cgo_setenv
_x_cgo_unsetenv
gcc_util.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_util.o
_x_cgo_thread_start
gcc_amd64.S
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_amd64.o
_crosscall_amd64
</pre><pre style="background-color: green">
/tmp/go-build/crypto/x509/_obj/
root_cgo_darwin.cgo2.c
/tmp/go-buildXXXXXXXXX/crypto/x509/_obj/root_cgo_darwin.cgo2.o
_FetchPEMRoots_MountainLion
/var/tmp/dist/go/src/crypto/x509/root_cgo_darwin.go
_useOldCode
_FetchPEMRoots
__cgo_62033c69288a_Cfunc_CFDataGetBytePtr
__cgo_62033c69288a_Cfunc_CFDataGetLength
__cgo_62033c69288a_Cfunc_CFRelease
__cgo_62033c69288a_Cfunc_FetchPEMRoots
</pre>

}}}

{{{#!html
<small>
<pre style="background-color: lightblue">
/var/tmp/dist/go/src/runtime/cgo/
gcc_context.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_context.o
__cgo_release_context
gcc_darwin_amd64.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_darwin_amd64.o
_x_cgo_init
__cgo_sys_thread_start
_threadentry
_k1
gcc_libinit.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_libinit.o
_x_cgo_sys_thread_create
__cgo_try_pthread_create
__cgo_wait_runtime_init_done
_x_cgo_notify_runtime_init_done
_x_cgo_set_context_function
__cgo_get_context_function
_runtime_init_mu
_runtime_init_cond
_runtime_init_done
_cgo_context_function
gcc_setenv.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_setenv.o
_x_cgo_setenv
_x_cgo_unsetenv
gcc_util.c
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_util.o
_x_cgo_thread_start
gcc_amd64.S
/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/gcc_amd64.o
_crosscall_amd64
</pre><pre style="background-color: green">
/tmp/go-build/crypto/x509/_obj/
root_cgo_darwin.cgo2.c
/tmp/go-buildXXXXXXXXX/crypto/x509/_obj/root_cgo_darwin.cgo2.o
_FetchPEMRoots_MountainLion
/var/tmp/dist/go/src/crypto/x509/root_cgo_darwin.go
_useOldCode
_FetchPEMRoots
__cgo_62033c69288a_Cfunc_CFDataGetBytePtr
<span style="background-color: gray">cgo-gcc-prolog</span>
__cgo_62033c69288a_Cfunc_CFDataGetLength
__cgo_62033c69288a_Cfunc_CFRelease
__cgo_62033c69288a_Cfunc_FetchPEMRoots
</pre><pre style="background-color: red">
/tmp/go-build/net/_obj/
_cgo_export.c
/tmp/go-buildXXXXXXXXX/net/_obj/_cgo_export.o
__cgo_9f1b05c52f96_Cfunc__Cmalloc
cgo_resnew.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_resnew.cgo2.o
__cgo_9f1b05c52f96_C2func_getnameinfo
__cgo_9f1b05c52f96_Cfunc_getnameinfo
cgo_unix.cgo2.c
/tmp/go-buildXXXXXXXXX/net/_obj/cgo_unix.cgo2.o
__cgo_9f1b05c52f96_C2func_getaddrinfo
__cgo_9f1b05c52f96_Cfunc_free
__cgo_9f1b05c52f96_Cfunc_freeaddrinfo
__cgo_9f1b05c52f96_Cfunc_gai_strerror
__cgo_9f1b05c52f96_Cfunc_getaddrinfo
</pre>

}}}

Trac:
Description: The build of Snowflake for MacOS is often producing the same result, but not always.

Arthur has been rebuilding Snowflake 8 times, with 4 different results: https://gist.github.com/arthuredelstein/73860df088c565ea0b2ca6eef586063a

to

The build of Snowflake for MacOS is often producing the same result, but not always.

Arthur has been rebuilding Snowflake 8 times, with 4 different results: https://gist.github.com/arthuredelstein/73860df088c565ea0b2ca6eef586063a

fish script:

for x in (seq 8)
    rm out/snowflake/snowflake-6077141f4aff-osx-x86_64-3b578d.tar.gz
    ./rbm/rbm build snowflake  --target alpha --target torbrowser-osx-x86_64
    tar xvf out/snowflake/snowflake-6077141f4aff-osx-x86_64-3b578d.tar.gz
    echo (sha256sum ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
end

Results:

b060b42cfd0c8fb2781dbb0fd45d42804dbb414473fec0597d9c2fb7d6d12aa8  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
22557c38d913e478e480dd3581efc00019fe2989c4273d9207f1719c34b6e399  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
22557c38d913e478e480dd3581efc00019fe2989c4273d9207f1719c34b6e399  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
1ee0dd2a0b228988e22c663d62b696b23a6ac48dc742a57dfa8f854aa3992bc3  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client
6d008bc7d29e8543608491b67d4b11da7bd6589741d9f52ac5fd50dd39d84f29  ./Contents/MacOS/Tor/PluggableTransports/snowflake-client

Trac:
Priority: Medium to High
Keywords: N/A deleted, tbb-rbm, TorBrowserTeam201809 added

Moving tickets to October

Trac:
Keywords: TorBrowserTeam201809 deleted, TorBrowserTeam201810 added

Trac:
Owner: N/A to tbb-team
Priority: High to Very High
Status: new to assigned
Cc: tbb-team, dcf, arthuredelstein, arlolra to dcf, arthuredelstein, arlolra

In branch bug_27827, I made a patch to build with go 1.11: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27827&id=ca1921e4566ef28d134b6fe88dea3c0df1d6ad1a

The build finished correctly, but I didn't try yet to do multiple builds to see if we get the same binary.

However it seems the sed on /tmp/go-build000000000 is still needed as I didn't get an error during the build.

I did several builds with [comment:7 boklm's go1.11 patch] and analyzed the differences.

Now, there are no /tmp/go-build000000000 paths anymore (where 000000000 stands for random digits), but there is a single /tmp/go-link-000000000/go.o. This shouldn't be a problem, because the sed substitution rewrites it to /tmp/go-link-XXXXXXXXX/go.o and there is only one so there is no problem with ordering.

But there's another difference in the binaries, the Go build ID. Here are sample values that are baked into the binary:

Go build ID: "kTjwWGrY9n3mGwOUpwVM/N2WdKU5WHR85aFCEJdn9/aRTcrk1SBq_sas7IMHGu/naljN2FzcmFC20pFl5NO"
Go build ID: "kTjwWGrY9n3mGwOUpwVM/N2WdKU5WHR85aFCEJdn9/aRTcrk1SBq_sas7IMHGu/XQTshh9tsXMqYLXM55kx"
Go build ID: "kTjwWGrY9n3mGwOUpwVM/N2WdKU5WHR85aFCEJdn9/aRTcrk1SBq_sas7IMHGu/Etnddau3jLCah3D-CG_-"

I found some documentation that says the build ID is, in part, a hash of all the input filenames. So I suspect it's including the random /tmp/go-link-000000000/go.o in the hash, because the build ID was different in all 9 builds I did.

One option is just to do another sed substitution on the build ID. But before that, I'm going to try building with go1.10 to see if this is a new bug introduced in go1.11.

Potentially related Go issues: GH#16860 GH#28008 GH#22382.

EDIT: same symptoms with go1.10.4.

I have a mostly working branch bug_27827.

It updates to go1.11.1. This update removed all the /tmp/go-build000000000 paths that we previously had to overwrite. However it left a single /tmp/go-link-000000000/go.o path. This path was also affecting the Go build ID (a hash of the build inputs), so it couldn't be fixed just by overwriting. However, I found a -tmpdir flag that lets us replace the random path with a static path, and that fixed it.

There is still a reproducibility problem, which is the gzip timestamp (bytes 4, 5, 6, 7) in the tar.gz output file differ across builds. I.e., the complete diffoscope output is

--- bug_27827/tmpdir/snowflake.tar.gz.1
+++ bug_27827/tmpdir/snowflake.tar.gz.2
├── metadata
│ @@ -1 +1 @@
│ -gzip compressed data, last modified: Wed Oct 17 16:30:51 2018, from Unix
│ +gzip compressed data, last modified: Wed Oct 17 16:34:52 2018, from Unix

I'm guessing that this is some other problem unrelated to go or snowflake.

Trac:
Status: assigned to needs_review

Trac:
Keywords: TorBrowserTeam201810 deleted, TorBrowserTeam201810R added

Looks good to me. I merged the branch with commit 889be4318d80f11f2b5b2b845ab11868149c48f1 into tor-browser-build's master.

Trac:
Status: needs_review to closed
Resolution: N/A to fixed

That patch is actually busting our Linux builds, reopening:

# github.com/keroserene/go-webrtc
peerconnection.cc:374: error: undefined reference to 'webrtc::JsepSessionDescription::JsepSessionDescription(std::string const&)'
peerconnection.cc:377: error: undefined reference to 'webrtc::SdpDeserialize(std::string const&, webrtc::JsepSessionDescription*, webrtc::SdpParseError*)'
peerconnection.cc:416: error: undefined reference to 'webrtc::CreateIceCandidate(std::string const&, int, std::string const&, webrtc::SdpParseError*)'
peerconnection.cc:55: error: undefined reference to 'rtc::Thread::SetName(std::string const&, void const*)'
peerconnection.cc:56: error: undefined reference to 'rtc::Thread::SetName(std::string const&, void const*)'
./include/webrtc/base/array_view.h:141: error: undefined reference to 'rtc::FatalMessage::FatalMessage(char const*, int, std::string*)'
collect2: error: ld returned 1 exit status

Trac:
Cc: dcf, arthuredelstein, arlolra to dcf, arthuredelstein, arlolra, boklm
Keywords: TorBrowserTeam201810R deleted, TorBrowserTeam201810 added
Resolution: fixed to N/A
Status: closed to reopened

Sorry for not checking the linux build. I'll have a little time to look at this on Tuesday, but if I can't figure it out then, it'll be another week before I can look at it.

Please see bug_27827_v2. The difference is that now we set -D_GLIBCXX_USE_CXX11_ABI=1 in snowflake/build, just like in go-webrtc/config. (The error message in comment:12 was caused by incompatible ABIs; i.e. std::string versus std::__cxx11::basic_string.) I don't know why we need to set it in snowflake/build now (or alternatively, why we didn't have to set it with go1.9).

Trac:
Status: reopened to needs_review
Keywords: TorBrowserTeam201810 deleted, TorBrowserTeam201810R added

Thanks. Works now on my build machines. Merged to master (commit 1ca5a195c785f70cac791117a4e12d49482a206f and 28abf57bd5bd307262741ac679a80e98aa389d20).

Trac:
Status: needs_review to closed
Resolution: N/A to fixed

Trac:
Parent: N/A to #19001 (moved)

Trac:
Sponsor: N/A to Sponsor19

closed

moved to tpo/anti-censorship/pluggable-transports/snowflake#27827 (closed)

Reproducibility issue of the snowflake osx64 build

Child items ...

Activity