Android reproducible build of Snowflake
This ticket is for tracking the progress of porting Snowflake to Android.
Activity
Hans-Christoph did the bulk of porting both go-webrtc and Snowflake to Android. The process is narrated in #28205 (moved) and culminated in
- https://github.com/keroserene/go-webrtc/pull/90 (go-webrtc port to Android)
- https://github.com/keroserene/snowflake/pull/43 (Snowflake port to Android and deb packages)
The current status, as I understand it, is that the build is reproducible, but includes or at least downloads the proprietary Google Play Services library as a side effect of the build.
Trac:
Cc: dcf, arlolra to dcf, arlolra, eighthave
Trac:
Parent: N/A to #19001 (moved)Replying to dcf:
- https://github.com/keroserene/go-webrtc/pull/90 (go-webrtc port to Android)
The current status, as I understand it, is that the build is reproducible, but includes or at least downloads the proprietary Google Play Services library as a side effect of the build.
The android changes are now merged into go-webrtc. We found a workaround for the Google Play services problem.
You can see build instructions for various platforms by looking at the
.gitlab-ci.ymlfile: https://github.com/keroserene/snowflake/blob/master/.gitlab-ci.ymlThese will build the Go code using the libwebrtc binaries included in the git repo.
-
Oops, I guess my
.gitlab-ci.ymlnever got merged upstream, you can see the android build in my fork: https://gitlab.com/eighthave/snowflake/blob/master/.gitlab-ci.yml Replying to gk:
dcf: do you have a script/instructions somewhere on how to build the whole thing which we could then translate into our
tor-browser-buildprocess? If you plan to do that work yourself, you are more than welcome! :). We'll likely put that on our plate otherwise.I didn't do anything beyond making sure that go-webrtc and snowflake would compile for Android, because I don't know how the pluggable transport integration works on Android. eighthave was showing us a
gomobile bindsetup that doesn't use separate executables (requires patching outpackage main) and I didn't know if that was how it was going to be done.I think that go-webrtc will only require minor changes in projects/webrtc/config and projects/snowflake/config. Here you can see the changes we made in the upstream build script: https://github.com/keroserene/go-webrtc/pull/90/files#diff-0b83f9dedf40d7356e5ca147a077acb4 In projects/webrtc/config:
- Set
TARGET_OS=androidTARGET_CPU=arm - Append
target_os = [ 'android' ]to .gclient. I think it's safe to do across all targets, as it only affects what gets downloaded. - In
gclient sync,echo nto not agree to the proprietary Google Play services license (also prevents the Play libraries from downloading). You could alternately patch out this check, etc. - In
gn gen, setsymbol_level=0. This is needed to actually disable debugging symbols; otherwise the libwebrtc is like 100 MB. - Upstream we are using Google's prebuilt arm-linux-androideabi-ar, but it would be better to build your own. In projects/snowflake/config:
- Set
GOOS=androidGOARCH=arm. - You may have to copy webrtc-android-armeabi-v7a.pc over webrtc-linux-arm.pc, as in eighthave's script.
- Set
Replying to dcf:
Replying to gk:
dcf: do you have a script/instructions somewhere on how to build the whole thing which we could then translate into our
tor-browser-buildprocess? If you plan to do that work yourself, you are more than welcome! :). We'll likely put that on our plate otherwise.I didn't do anything beyond making sure that go-webrtc and snowflake would compile for Android, because I don't know how the pluggable transport integration works on Android. eighthave was showing us a
gomobile bindsetup that doesn't use separate executables (requires patching outpackage main) and I didn't know if that was how it was going to be done.I think that go-webrtc will only require minor changes in projects/webrtc/config and projects/snowflake/config. Here you can see the changes we made in the upstream build script: https://github.com/keroserene/go-webrtc/pull/90/files#diff-0b83f9dedf40d7356e5ca147a077acb4 In projects/webrtc/config:
- Set
TARGET_OS=androidTARGET_CPU=arm - Append
target_os = [ 'android' ]to .gclient. I think it's safe to do across all targets, as it only affects what gets downloaded. - In
gclient sync,echo nto not agree to the proprietary Google Play services license (also prevents the Play libraries from downloading). You could alternately patch out this check, etc. - In
gn gen, setsymbol_level=0. This is needed to actually disable debugging symbols; otherwise the libwebrtc is like 100 MB. - Upstream we are using Google's prebuilt arm-linux-androideabi-ar, but it would be better to build your own. In projects/snowflake/config:
- Set
GOOS=androidGOARCH=arm. - You may have to copy webrtc-android-armeabi-v7a.pc over webrtc-linux-arm.pc, as in eighthave's script.
Thanks, that's helpful.
- Set
-
Trac:
Parent: #19001 (moved) to #30318 (moved) 
There is a patch for review in branch
bug_28672_v6: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_28672_v6&id=c18a7c37c9a439205da55830cbecc0c4a89429ecI did not check yet that the build is reproducible.
Trac:
Status: new to needs_review
Keywords: TorBrowserTeam201906 deleted, TorBrowserTeam201906R added-
Replying to boklm:
There is a patch for review in branch
bug_28672_v6: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_28672_v6&id=c18a7c37c9a439205da55830cbecc0c4a89429ecI did not check yet that the build is reproducible.
They are not but it seems we are close. Just
libgojni.sodiffers for me with what seems to be mostly different tmp directories being used during build:--- /dev/fd/63 2019-06-14 18:21:34.285763344 +0200 +++ /dev/fd/62 2019-06-14 18:21:34.285763344 +0200 @@ -20,12 +20,12 @@ 00000130: 0400 0000 52e5 7464 f089 0d00 f099 0d00 ....R.td........ 00000140: f099 0d00 1066 0a00 1066 0a00 0600 0000 .....f...f...... 00000150: 0800 0000 0400 0000 5300 0000 0400 0000 ........S....... -00000160: 476f 0000 7043 6a4a 6c57 3058 4d41 5478 Go..pCjJlW0XMATx -00000170: 5970 6c6f 3775 3452 2f44 6844 5253 6a69 Yplo7u4R/DhDRSji -00000180: 6434 6435 525f 6b79 6d34 715a 462f 2d74 d4d5R_kym4qZF/-t -00000190: 4275 6253 4632 2d6a 344a 714c 3052 6448 BubSF2-j4JqL0RdH -000001a0: 6568 2f69 7a52 4435 5f67 3931 775f 4564 eh/izRD5_g91w_Ed -000001b0: 6d6b 497a 5075 3700 0000 0000 0000 0000 mkIzPu7......... +00000160: 476f 0000 786a 5532 4a43 634c 694f 3272 Go..xjU2JCcLiO2r +00000170: 6b33 5172 4356 586e 2f5f 4d2d 384a 7a59 k3QrCVXn/_M-8JzY +00000180: 3536 4178 354d 6d53 5f5a 784f 612f 5858 56Ax5MmS_ZxOa/XX +00000190: 7337 314b 6d44 5672 4f33 796e 3973 3532 s71KmDVrO3yn9s52 +000001a0: 5076 2f49 4f4c 5378 6b67 6d6e 395f 4665 Pv/IOLSxkgmn9_Fe +000001b0: 722d 7033 2d45 6f00 0000 0000 0000 0000 r-p3-Eo......... 000001c0: 0000 0000 0000 0000 0100 0000 0000 0000 ................ 000001d0: 0000 0000 1200 0000 4300 0000 0000 0000 ........C....... 000001e0: 0000 0000 1200 0000 5000 0000 48d9 0b00 ........P...H... @@ -133,7 +133,7 @@ 00000840: 0000 0000 1000 f1ff 005f 5f63 7861 5f66 .........__cxa_f 00000850: 696e 616c 697a 6500 4c49 4243 006c 6962 inalize.LIBC.lib 00000860: 632e 736f 002f 746d 702f 676f 2d62 7569 c.so./tmp/go-bui -00000870: 6c64 3235 3637 3734 3035 332f 6230 3031 ld256774053/b001 +00000870: 6c64 3739 3430 3230 3539 332f 6230 3031 ld794020593/b001 00000880: 2f65 7865 2f61 2e6f 7574 005f 5f63 7861 /exe/a.out.__cxa 00000890: 5f61 7465 7869 7400 5f63 676f 5f38 3938 _atexit._cgo_898 000008a0: 6432 6164 3234 6165 315f 4366 756e 635f d2ad24ae1_Cfunc_ @@ -330,7 +330,7 @@ 00001490: 0200 0200 0200 0200 0200 0200 0100 0100 ................ 000014a0: 0200 0200 0200 0200 0100 0200 0200 0100 ................ 000014b0: 0100 0000 0100 0100 0100 0000 0100 0100 ................ -000014c0: 0100 0100 045e 8d0d 1400 0000 0000 0000 .....^.......... +000014c0: 0100 0100 f4b7 cd09 1400 0000 0000 0000 ................ 000014d0: 1d00 0000 0000 0000 0100 0100 1500 0000 ................ 000014e0: 1000 0000 0000 0000 630d 0500 0000 0200 ........c....... 000014f0: 1000 0000 0000 0000 f099 0d00 1700 0000 ................ @@ -97548,14 +97548,14 @@ 0017d0b0: b259 0600 8559 0600 5b59 0600 2c59 0600 .Y...Y..[Y..,Y.. 0017d0c0: e158 0600 9058 0600 6058 0600 2858 0600 .X...X..`X..(X.. 0017d0d0: f457 0600 2f74 6d70 2f67 6f6d 6f62 696c .W../tmp/gomobil -0017d0e0: 652d 776f 726b 2d36 3238 3136 3038 3333 e-work-628160833 +0017d0e0: 652d 776f 726b 2d32 3034 3030 3234 3432 e-work-204002442 0017d0f0: 2f73 7263 2f67 6f62 696e 642f 676f 5f6d /src/gobind/go_m 0017d100: 6169 6e2e 676f 0000 2f74 6d70 2f67 6f6d ain.go../tmp/gom -0017d110: 6f62 696c 652d 776f 726b 2d36 3238 3136 obile-work-62816 -0017d120: 3038 3333 2f73 7263 2f67 6f62 696e 642f 0833/src/gobind/ +0017d110: 6f62 696c 652d 776f 726b 2d32 3034 3030 obile-work-20400 +0017d120: 3234 3432 2f73 7263 2f67 6f62 696e 642f 2442/src/gobind/ 0017d130: 7365 715f 616e 6472 6f69 642e 676f 0000 seq_android.go.. 0017d140: 2f74 6d70 2f67 6f6d 6f62 696c 652d 776f /tmp/gomobile-wo -0017d150: 726b 2d36 3238 3136 3038 3333 2f73 7263 rk-628160833/src +0017d150: 726b 2d32 3034 3030 3234 3432 2f73 7263 rk-204002442/src 0017d160: 2f67 6f62 696e 642f 7365 712e 676f 0000 /gobind/seq.go.. 0017d170: 2f76 6172 2f74 6d70 2f64 6973 742f 676f /var/tmp/dist/go 0017d180: 7061 7468 2f73 7263 2f67 6f6c 616e 672e path/src/golang. @@ -113192,14 +113192,14 @@ 001ba270: 0001 0101 0100 0000 0100 005f 6367 6f5f ..........._cgo_ 001ba280: 676f 7479 7065 732e 676f 0000 0000 2f74 gotypes.go..../t 001ba290: 6d70 2f67 6f6d 6f62 696c 652d 776f 726b mp/gomobile-work -001ba2a0: 2d36 3238 3136 3038 3333 2f73 7263 2f67 -628160833/src/g +001ba2a0: 2d32 3034 3030 3234 3432 2f73 7263 2f67 -204002442/src/g 001ba2b0: 6f62 696e 642f 7365 712e 676f 0000 0000 obind/seq.go.... 001ba2c0: 2f74 6d70 2f67 6f6d 6f62 696c 652d 776f /tmp/gomobile-wo -001ba2d0: 726b 2d36 3238 3136 3038 3333 2f73 7263 rk-628160833/src +001ba2d0: 726b 2d32 3034 3030 3234 3432 2f73 7263 rk-204002442/src 001ba2e0: 2f67 6f62 696e 642f 7365 715f 616e 6472 /gobind/seq_andr 001ba2f0: 6f69 642e 676f 0000 0000 2f74 6d70 2f67 oid.go..../tmp/g -001ba300: 6f6d 6f62 696c 652d 776f 726b 2d36 3238 omobile-work-628 -001ba310: 3136 3038 3333 2f73 7263 2f67 6f62 696e 160833/src/gobin +001ba300: 6f6d 6f62 696c 652d 776f 726b 2d32 3034 omobile-work-204 +001ba310: 3030 3234 3432 2f73 7263 2f67 6f62 696e 002442/src/gobin 001ba320: 642f 676f 5f6d 6169 6e2e 676f 0000 0000 d/go_main.go.... 001ba330: 3c61 7574 6f67 656e 6572 6174 6564 3e00 <autogenerated>. 001ba340: 0000 0000 0005 0280 cf0b 0004 0103 f000 ................ @@ -113234,7 +113234,7 @@ 001ba510: 0101 5801 0000 0200 c200 0000 0101 fb0e ..X............. 001ba520: 0d00 0101 0101 0000 0001 0000 012f 746d ............./tm 001ba530: 702f 676f 6d6f 6269 6c65 2d77 6f72 6b2d p/gomobile-work- -001ba540: 3632 3831 3630 3833 332f 7372 632f 676f 628160833/src/go +001ba540: 3230 3430 3032 3434 322f 7372 632f 676f 204002442/src/go 001ba550: 6269 6e64 002f 7661 722f 746d 702f 6469 bind./var/tmp/di 001ba560: 7374 2f67 6f70 6174 682f 706b 672f 676f st/gopath/pkg/go 001ba570: 6d6f 6269 6c65 2f6e 646b 2d74 6f6f 6c63 mobile/ndk-toolc @@ -113260,8 +113260,8 @@ 001ba6b0: 6f6f 6c63 6861 696e 732f 6172 6d2f 6269 oolchains/arm/bi 001ba6c0: 6e2f 2e2e 2f73 7973 726f 6f74 2f75 7372 n/../sysroot/usr 001ba6d0: 2f69 6e63 6c75 6465 002f 746d 702f 676f /include./tmp/go -001ba6e0: 6d6f 6269 6c65 2d77 6f72 6b2d 3632 3831 mobile-work-6281 -001ba6f0: 3630 3833 332f 7372 632f 676f 6269 6e64 60833/src/gobind +001ba6e0: 6d6f 6269 6c65 2d77 6f72 6b2d 3230 3430 mobile-work-2040 +001ba6f0: 3032 3434 322f 7372 632f 676f 6269 6e64 02442/src/gobind 001ba700: 0000 6367 6f2d 6763 632d 7072 6f6c 6f67 ..cgo-gcc-prolog 001ba710: 0000 0000 7374 6469 6e74 2e68 0001 0000 ....stdint.h.... 001ba720: 7365 715f 616e 6472 6f69 642e 6800 0200 seq_android.h... @@ -113293,8 +113293,8 @@ 001ba8c0: 2d74 6f6f 6c63 6861 696e 732f 6172 6d2f -toolchains/arm/ 001ba8d0: 6269 6e2f 2e2e 2f73 7973 726f 6f74 2f75 bin/../sysroot/u 001ba8e0: 7372 2f69 6e63 6c75 6465 002f 746d 702f sr/include./tmp/ -001ba8f0: 676f 6d6f 6269 6c65 2d77 6f72 6b2d 3632 gomobile-work-62 -001ba900: 3831 3630 3833 332f 7372 632f 676f 6269 8160833/src/gobi +001ba8f0: 676f 6d6f 6269 6c65 2d77 6f72 6b2d 3230 gomobile-work-20 +001ba900: 3430 3032 3434 322f 7372 632f 676f 6269 4002442/src/gobi 001ba910: 6e64 002f 7661 722f 746d 702f 6469 7374 nd./var/tmp/dist 001ba920: 2f67 6f70 6174 682f 706b 672f 676f 6d6f /gopath/pkg/gomo 001ba930: 6269 6c65 2f6e 646b 2d74 6f6f 6c63 6861 bile/ndk-toolcha @@ -113471,8 +113471,8 @@ 001bb3e0: 6f6c 6368 6169 6e73 2f61 726d 2f62 696e olchains/arm/bin 001bb3f0: 2f2e 2e2f 7379 7372 6f6f 742f 7573 722f /../sysroot/usr/ 001bb400: 696e 636c 7564 6500 2f74 6d70 2f67 6f6d include./tmp/gom -001bb410: 6f62 696c 652d 776f 726b 2d36 3238 3136 obile-work-62816 -001bb420: 3038 3333 2f73 7263 2f67 6f62 696e 6400 0833/src/gobind. +001bb410: 6f62 696c 652d 776f 726b 2d32 3034 3030 obile-work-20400 +001bb420: 3234 3432 2f73 7263 2f67 6f62 696e 6400 2442/src/gobind. 001bb430: 2f76 6172 2f74 6d70 2f64 6973 742f 676f /var/tmp/dist/go 001bb440: 7061 7468 2f70 6b67 2f67 6f6d 6f62 696c path/pkg/gomobil 001bb450: 652f 6e64 6b2d 746f 6f6c 6368 6169 6e73 e/ndk-toolchains @@ -183567,11 +183567,11 @@ 002cd0e0: 2e63 676f 322e 6300 5f63 676f 5f38 3938 .cgo2.c._cgo_898 002cd0f0: 6432 6164 3234 6165 315f 4366 756e 635f d2ad24ae1_Cfunc_ 002cd100: 6672 6565 002f 746d 702f 676f 6d6f 6269 free./tmp/gomobi -002cd110: 6c65 2d77 6f72 6b2d 3632 3831 3630 3833 le-work-62816083 -002cd120: 332f 7372 632f 676f 6269 6e64 2f73 6571 3/src/gobind/seq +002cd110: 6c65 2d77 6f72 6b2d 3230 3430 3032 3434 le-work-20400244 +002cd120: 322f 7372 632f 676f 6269 6e64 2f73 6571 2/src/gobind/seq 002cd130: 5f61 6e64 726f 6964 2e63 002f 746d 702f _android.c./tmp/ -002cd140: 676f 6d6f 6269 6c65 2d77 6f72 6b2d 3632 gomobile-work-62 -002cd150: 3831 3630 3833 332f 7372 632f 676f 6269 8160833/src/gobi +002cd140: 676f 6d6f 6269 6c65 2d77 6f72 6b2d 3230 gomobile-work-20 +002cd150: 3430 3032 3434 322f 7372 632f 676f 6269 4002442/src/gobi 002cd160: 6e64 006a 766d 0072 6573 6572 7665 6430 nd.jvm.reserved0 002cd170: 0072 6573 6572 7665 6431 0072 6573 6572 .reserved1.reser 002cd180: 7665 6432 0044 6573 7472 6f79 4a61 7661 ved2.DestroyJava @@ -183961,15 +183961,15 @@ 002ce980: 6600 6a6f 626a 0063 6c61 7a7a 0072 6566 f.jobj.clazz.ref 002ce990: 5f63 6c61 7373 0073 6967 006d 006f 626a _class.sig.m.obj 002ce9a0: 0069 7369 6e73 7400 2f74 6d70 2f67 6f6d .isinst./tmp/gom -002ce9b0: 6f62 696c 652d 776f 726b 2d36 3238 3136 obile-work-62816 -002ce9c0: 3038 3333 2f73 7263 2f67 6f62 696e 642f 0833/src/gobind/ +002ce9b0: 6f62 696c 652d 776f 726b 2d32 3034 3030 obile-work-20400 +002ce9c0: 3234 3432 2f73 7263 2f67 6f62 696e 642f 2442/src/gobind/ 002ce9d0: 736e 6f77 666c 616b 6563 6c69 656e 745f snowflakeclient_ 002ce9e0: 616e 6472 6f69 642e 6300 4a61 7661 5f73 android.c.Java_s 002ce9f0: 6e6f 7766 6c61 6b65 636c 6965 6e74 5f53 nowflakeclient_S 002cea00: 6e6f 7766 6c61 6b65 636c 6965 6e74 5f5f nowflakeclient__ 002cea10: 3169 6e69 7400 5f75 6e75 7365 6400 2f74 1init._unused./t 002cea20: 6d70 2f67 6f6d 6f62 696c 652d 776f 726b mp/gomobile-work -002cea30: 2d36 3238 3136 3038 3333 2f73 7263 2f67 -628160833/src/g +002cea30: 2d32 3034 3030 3234 3432 2f73 7263 2f67 -204002442/src/g 002cea40: 6f62 696e 642f 756e 6976 6572 7365 5f61 obind/universe_a 002cea50: 6e64 726f 6964 2e63 0070 726f 7879 5f63 ndroid.c.proxy_c 002cea60: 6c61 7373 5f5f 6572 726f 7200 7072 6f78 lass__error.proxNot sure about
-000014c0: 0100 0100 045e 8d0d 1400 0000 0000 0000 .....^.......... +000014c0: 0100 0100 f4b7 cd09 1400 0000 0000 0000 ................, though. But I guess we'll see how it looks once we ruled the easy issue out.
Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201906R deleted, TorBrowserTeam201906 added
