ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0
Contrary to our testing results the .dmg bundles we produce for 8.0a9 are differing on two machines. And it turns out Linux bundles are affected as well (see: #27937 (moved) for the original Linux bug report).
Activity
Only
XULis different in our case:--- /dev/fd/63 2018-06-24 08:54:28.175516977 +0200 +++ /dev/fd/62 2018-06-24 08:54:28.175516977 +0200 @@ -201,8 +201,8 @@ 00000c80: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c90: 0000 0000 0000 0000 104f 5805 a616 0000 .........OX..... 00000ca0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ -00000cb0: 1b00 0000 1800 0000 ddc1 d65d f3d8 3a65 ...........]..:e -00000cc0: 8465 a64e d247 5b4d 2400 0000 1000 0000 .e.N.G[M$....... +00000cb0: 1b00 0000 1800 0000 18a4 cc7a 95d7 325d ...........z..2] +00000cc0: 96c5 6f08 51ec 0b4e 2400 0000 1000 0000 ..o.Q..N$....... 00000cd0: 0007 0a00 0006 0a00 0c00 0000 5800 0000 ............X... 00000ce0: 1800 0000 0200 0000 0000 1600 0000 0100 ................ 00000cf0: 2f53 7973 7465 6d2f 4c69 6272 6172 792f /System/Library/ @@ -3732284,59 +3732284,59 @@ 038f33b0: ff48 8b07 488b 4f08 4889 8d70 ffff ff48 .H..H.O.H..p...H 038f33c0: 8985 68ff ffff 488d 051e 432a 0048 8985 ..h...H...C*.H.. 038f33d0: 38ff ffff 48c7 8540 ffff ff06 0000 0048 8...H..@.......H -038f33e0: 8d3d 9adf bf01 ff17 8078 2100 0f85 da01 .=.......x!..... +038f33e0: 8d3d 9adf bf01 ff17 8078 2100 0f85 d801 .=.......x!..... 038f33f0: 0000 488d 3d87 dfbf 01ff 1780 7820 0075 ..H.=.......x .u 038f3400: 1f48 8d3d 78df bf01 ff17 4889 c348 8d3d .H.=x.....H..H.= 038f3410: fc95 ffff 4889 dee8 c0c1 0100 c643 2001 ....H........C . -038f3420: 488d 3d59 dfbf 01ff 1748 8338 0174 6748 H.=Y.....H.8.tgH +038f3420: 488d 3d59 dfbf 01ff 1748 8338 0174 6548 H.=Y.....H.8.teH 038f3430: 8d3d 4adf bf01 ff17 660f 6f40 100f 57c9 .=J.....f.o@..W. -038f3440: 0f29 4810 b901 0000 0066 480f 6ec9 488b .)H......fH.n.H. -038f3450: 0866 0f7f 0848 85c9 7429 6648 0f7e c348 .f...H..t)fH.~.H -038f3460: 85db 741f 660f 70c0 4e66 490f 7ec6 4889 ..t.f.p.NfI.~.H. -038f3470: df41 ff16 4983 7e08 0074 0848 89df e8c7 .A..I.~..t.H.... -038f3480: b001 0048 8d3d f6de bf01 ff17 4883 3801 ...H.=......H.8. -038f3490: 0f85 8102 0000 488d 3de3 debf 01ff 1748 ......H.=......H -038f34a0: 8378 0800 7477 e895 f2ff ff48 89c3 4889 .x..tw.....H..H. -038f34b0: 9d48 ffff ff48 8b45 9048 8945 d048 8b45 .H...H.E.H.E.H.E -038f34c0: 8848 8945 c848 8b45 8048 8945 c048 8b85 .H.E.H.E.H.E.H.. -038f34d0: 78ff ffff 4889 45b8 488b 8568 ffff ff48 x...H.E.H..h...H -038f34e0: 8b8d 70ff ffff 4889 4db0 4889 45a8 488d ..p...H.M.H.E.H. -038f34f0: 7dd8 488d b548 ffff ff48 8d55 a8e8 8ef9 }.H..H...H.U.... -038f3500: ffff f048 ff0b 0f85 a100 0000 488d bd48 ...H........H..H -038f3510: ffff ffe8 58c0 ffff e990 0000 0048 8d3d ....X........H.= -038f3520: 5cde bf01 ff17 48c7 4008 ffff ffff 488d \.....H.@.....H. -038f3530: 4808 488d 5010 4889 9548 ffff ff48 898d H.H.P.H..H...H.. -038f3540: 50ff ffff 488b 7010 4885 f60f 84b0 0100 P...H.p.H....... -038f3550: 0048 8d3d 28de bf01 ff17 4889 c348 8b45 .H.=(.....H..H.E -038f3560: 9048 8945 d048 8b43 1848 8b4d 8848 894d .H.E.H.C.H.M.H.M -038f3570: c848 8b4d 8048 894d c048 8b8d 78ff ffff .H.M.H.M.H..x... -038f3580: 4889 4db8 488b 8d68 ffff ff48 8b95 70ff H.M.H..h...H..p. -038f3590: ffff 4889 55b0 4889 4da8 488d 7dd8 488d ..H.U.H.M.H.}.H. -038f35a0: 55a8 ff50 3048 c743 0800 0000 008b 45d9 U..P0H.C......E. -038f35b0: 8945 e88a 45d8 0fb7 4ddd 6689 4dec 8a4d .E..E...M.f.M..M -038f35c0: df88 4dee 3c04 0f85 8000 0000 e86f f1ff ..M.<........o.. -038f35d0: ff48 89c3 4889 9d48 ffff ff48 8b45 9048 .H..H..H...H.E.H -038f35e0: 8945 d048 8b45 8848 8945 c848 8b45 8048 .E.H.E.H.E.H.E.H -038f35f0: 8945 c048 8b85 78ff ffff 4889 45b8 488b .E.H..x...H.E.H. -038f3600: 8568 ffff ff48 8b8d 70ff ffff 4889 4db0 .h...H..p...H.M. -038f3610: 4889 45a8 488d 7d98 488d b548 ffff ff48 H.E.H.}.H..H...H -038f3620: 8d55 a8e8 68f8 ffff f048 ff0b 750c 488d .U..h....H..u.H. -038f3630: bd48 ffff ffe8 36bf ffff 807d 9803 7531 .H....6....}..u1 -038f3640: 4881 c4c0 0000 005b 415e 5dc3 8845 9848 H......[A^]..E.H -038f3650: 8b45 e08b 4de8 894d 990f b74d ec66 894d .E..M..M...M.f.M -038f3660: 9d8a 4dee 884d 9f48 8945 a080 7d98 0374 ..M..M.H.E..}..t -038f3670: cf48 8b45 9848 8b4d a048 894d e048 8945 .H.E.H.M.H.M.H.E -038f3680: d848 8d85 38ff ffff 4889 8548 ffff ff48 .H..8...H..H...H -038f3690: 8d05 dab6 ffff 4889 8550 ffff ff48 8d45 ......H..P...H.E -038f36a0: d848 8985 58ff ffff 488d 0501 99ff ff48 .H..X...H......H -038f36b0: 8985 60ff ffff 488d 052b a7b6 0148 8945 ..`...H..+...H.E -038f36c0: a848 c745 b002 0000 0048 8d05 f863 2a00 .H.E.....H...c*. -038f36d0: 4889 45b8 48c7 45c0 0200 0000 488d 8548 H.E.H.E.....H..H -038f36e0: ffff ff48 8945 c848 c745 d002 0000 0048 ...H.E.H.E.....H -038f36f0: 8d35 12a7 b601 488d 7da8 e881 e5ff ff0f .5....H.}....... -038f3700: 0b48 8d3d 78dc bf01 ff17 48c7 4008 0000 .H.=x.....H.@... -038f3710: 0000 e98f fdff ff48 8d3d eabb b601 e82d .......H.=.....- -038f3720: 8f00 000f 0b66 662e 0f1f 8400 0000 0000 .....ff......... +038f3440: 0f29 4810 b901 0000 0066 480f 6ec9 4883 .)H......fH.n.H. +038f3450: 3800 660f 7f08 7429 6648 0f7e c348 85db 8.f...t)fH.~.H.. +038f3460: 741f 660f 70c0 4e66 490f 7ec6 4889 df41 t.f.p.NfI.~.H..A +038f3470: ff16 4983 7e08 0074 0848 89df e8c9 b001 ..I.~..t.H...... +038f3480: 0048 8d3d f8de bf01 ff17 4883 3801 0f85 .H.=......H.8... +038f3490: 8102 0000 488d 3de5 debf 01ff 1748 8378 ....H.=......H.x +038f34a0: 0800 7477 e897 f2ff ff48 89c3 4889 9d48 ..tw.....H..H..H +038f34b0: ffff ff48 8b45 9048 8945 d048 8b45 8848 ...H.E.H.E.H.E.H +038f34c0: 8945 c848 8b45 8048 8945 c048 8b85 78ff .E.H.E.H.E.H..x. +038f34d0: ffff 4889 45b8 488b 8568 ffff ff48 8b8d ..H.E.H..h...H.. +038f34e0: 70ff ffff 4889 4db0 4889 45a8 488d 7dd8 p...H.M.H.E.H.}. +038f34f0: 488d b548 ffff ff48 8d55 a8e8 90f9 ffff H..H...H.U...... +038f3500: f048 ff0b 0f85 a100 0000 488d bd48 ffff .H........H..H.. +038f3510: ffe8 5ac0 ffff e990 0000 0048 8d3d 5ede ..Z........H.=^. +038f3520: bf01 ff17 48c7 4008 ffff ffff 488d 4808 ....H.@.....H.H. +038f3530: 488d 5010 4889 9548 ffff ff48 898d 50ff H.P.H..H...H..P. +038f3540: ffff 488b 7010 4885 f60f 84b0 0100 0048 ..H.p.H........H +038f3550: 8d3d 2ade bf01 ff17 4889 c348 8b45 9048 .=*.....H..H.E.H +038f3560: 8945 d048 8b43 1848 8b4d 8848 894d c848 .E.H.C.H.M.H.M.H +038f3570: 8b4d 8048 894d c048 8b8d 78ff ffff 4889 .M.H.M.H..x...H. +038f3580: 4db8 488b 8d68 ffff ff48 8b95 70ff ffff M.H..h...H..p... +038f3590: 4889 55b0 4889 4da8 488d 7dd8 488d 55a8 H.U.H.M.H.}.H.U. +038f35a0: ff50 3048 c743 0800 0000 008b 45d9 8945 .P0H.C......E..E +038f35b0: e88a 45d8 0fb7 4ddd 6689 4dec 8a4d df88 ..E...M.f.M..M.. +038f35c0: 4dee 3c04 0f85 8000 0000 e871 f1ff ff48 M.<........q...H +038f35d0: 89c3 4889 9d48 ffff ff48 8b45 9048 8945 ..H..H...H.E.H.E +038f35e0: d048 8b45 8848 8945 c848 8b45 8048 8945 .H.E.H.E.H.E.H.E +038f35f0: c048 8b85 78ff ffff 4889 45b8 488b 8568 .H..x...H.E.H..h +038f3600: ffff ff48 8b8d 70ff ffff 4889 4db0 4889 ...H..p...H.M.H. +038f3610: 45a8 488d 7d98 488d b548 ffff ff48 8d55 E.H.}.H..H...H.U +038f3620: a8e8 6af8 ffff f048 ff0b 750c 488d bd48 ..j....H..u.H..H +038f3630: ffff ffe8 38bf ffff 807d 9803 7531 4881 ....8....}..u1H. +038f3640: c4c0 0000 005b 415e 5dc3 8845 9848 8b45 .....[A^]..E.H.E +038f3650: e08b 4de8 894d 990f b74d ec66 894d 9d8a ..M..M...M.f.M.. +038f3660: 4dee 884d 9f48 8945 a080 7d98 0374 cf48 M..M.H.E..}..t.H +038f3670: 8b45 9848 8b4d a048 894d e048 8945 d848 .E.H.M.H.M.H.E.H +038f3680: 8d85 38ff ffff 4889 8548 ffff ff48 8d05 ..8...H..H...H.. +038f3690: dcb6 ffff 4889 8550 ffff ff48 8d45 d848 ....H..P...H.E.H +038f36a0: 8985 58ff ffff 488d 0503 99ff ff48 8985 ..X...H......H.. +038f36b0: 60ff ffff 488d 052d a7b6 0148 8945 a848 `...H..-...H.E.H +038f36c0: c745 b002 0000 0048 8d05 fa63 2a00 4889 .E.....H...c*.H. +038f36d0: 45b8 48c7 45c0 0200 0000 488d 8548 ffff E.H.E.....H..H.. +038f36e0: ff48 8945 c848 c745 d002 0000 0048 8d35 .H.E.H.E.....H.5 +038f36f0: 14a7 b601 488d 7da8 e883 e5ff ff0f 0b48 ....H.}........H +038f3700: 8d3d 7adc bf01 ff17 48c7 4008 0000 0000 .=z.....H.@..... +038f3710: e98f fdff ff48 8d3d ecbb b601 e82f 8f00 .....H.=...../.. +038f3720: 000f 0b66 6666 662e 0f1f 8400 0000 0000 ...ffff......... 038f3730: 5548 89e5 803f 0074 0e48 8d05 f23f 2a00 UH...?.t.H...?*. 038f3740: ba2b 0000 00eb 0c48 8d05 c83f 2a00 ba1c .+.....H...?*... 038f3750: 0000 0048 89f7 4889 c65d e951 b700 0090 ...H..H..].Q.... @@ -4578681,7 +4578681,7 @@ 045dd780: 2400 0000 fcc0 0200 a85b 31ff ffff ffff $........[1..... 045dd790: 4800 0000 0000 0000 0041 0e10 8602 430d H........A....C. 045dd7a0: 0643 8304 8e03 0000 2400 0000 24c1 0200 .C......$...$... -045dd7b0: d05b 31ff ffff ffff a503 0000 0000 0000 .[1............. +045dd7b0: d05b 31ff ffff ffff a303 0000 0000 0000 .[1............. 045dd7c0: 0041 0e10 8602 430d 064a 8304 8e03 0000 .A....C..J...... 045dd7d0: 2400 0000 4cc1 0200 585f 31ff ffff ffff $...L...X_1..... 045dd7e0: 2f00 0000 0000 0000 0041 0e10 8602 430d /........A....C.
Maybe comparing the
obj-macosdirectory from two builds can help debug the issue.I have started a build with this patch: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26475&id=1ceef284b016441783d276ba32542e9fcd9bfd75
with the command:
$ ./rbm/rbm build firefox --target alpha --target torbrowser-osx-x86_64And will upload the result when it finish.
-
Replying to boklm:
Maybe comparing the
obj-macosdirectory from two builds can help debug the issue.I have started a build with this patch: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26475&id=1ceef284b016441783d276ba32542e9fcd9bfd75
I pushed a fixup commit for that: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26475&id=a5bb5818b3563a1a7a7d13a8068ef0e55943c13d
-
I compared two obj-macos dirs and gkrust seems to be the culprit (as assumed).
diff -u <(xxd tpo/gkrust-d3a9de07b53ab691.gkrust0.rcgu.o) <(xxd uninett/gkrust-d3a9de07b53ab691.gkrust0.rcgu.o)gives me among others
00440e20: ff48 8b07 488b 4f08 4889 8d70 ffff ff48 .H..H.O.H..p...H 00440e30: 8985 68ff ffff 488d 0500 0000 0048 8985 ..h...H......H.. 00440e40: 38ff ffff 48c7 8540 ffff ff06 0000 0048 8...H..@.......H -00440e50: 8b3d 0000 0000 ff17 8078 2100 0f85 da01 .=.......x!..... +00440e50: 8b3d 0000 0000 ff17 8078 2100 0f85 d801 .=.......x!..... 00440e60: 0000 488b 3d00 0000 00ff 1780 7820 0075 ..H.=.......x .u 00440e70: 1f48 8b3d 0000 0000 ff17 4889 c348 8d3d .H.=......H..H.= 00440e80: 0000 0000 4889 dee8 0000 0000 c643 2001 ....H........C . -00440e90: 488b 3d00 0000 00ff 1748 8338 0174 6748 H.=......H.8.tgH +00440e90: 488b 3d00 0000 00ff 1748 8338 0174 6548 H.=......H.8.teH 00440ea0: 8b3d 0000 0000 ff17 660f 6f40 100f 57c9 .=......f.o@..W. -00440eb0: 0f29 4810 b901 0000 0066 480f 6ec9 488b .)H......fH.n.H. -00440ec0: 0866 0f7f 0848 85c9 7429 6648 0f7e c348 .f...H..t)fH.~.H -00440ed0: 85db 741f 660f 70c0 4e66 490f 7ec6 4889 ..t.f.p.NfI.~.H. -00440ee0: df41 ff16 4983 7e08 0074 0848 89df e800 .A..I.~..t.H.... -00440ef0: 0000 0048 8b3d 0000 0000 ff17 4883 3801 ...H.=......H.8. -00440f00: 0f85 8102 0000 488b 3d00 0000 00ff 1748 ......H.=......H -00440f10: 8378 0800 7477 e800 0000 0048 89c3 4889 .x..tw.....H..H. -00440f20: 9d48 ffff ff48 8b45 9048 8945 d048 8b45 .H...H.E.H.E.H.E -00440f30: 8848 8945 c848 8b45 8048 8945 c048 8b85 .H.E.H.E.H.E.H.. -00440f40: 78ff ffff 4889 45b8 488b 8568 ffff ff48 x...H.E.H..h...H -00440f50: 8b8d 70ff ffff 4889 4db0 4889 45a8 488d ..p...H.M.H.E.H. -00440f60: 7dd8 488d b548 ffff ff48 8d55 a8e8 0000 }.H..H...H.U.... -00440f70: 0000 f048 ff0b 0f85 a100 0000 488d bd48 ...H........H..H -00440f80: ffff ffe8 0000 0000 e990 0000 0048 8b3d .............H.= -00440f90: 0000 0000 ff17 48c7 4008 ffff ffff 488d ......H.@.....H. -00440fa0: 4808 488d 5010 4889 9548 ffff ff48 898d H.H.P.H..H...H.. -00440fb0: 50ff ffff 488b 7010 4885 f60f 84b0 0100 P...H.p.H....... -00440fc0: 0048 8b3d 0000 0000 ff17 4889 c348 8b45 .H.=......H..H.E -00440fd0: 9048 8945 d048 8b43 1848 8b4d 8848 894d .H.E.H.C.H.M.H.M -00440fe0: c848 8b4d 8048 894d c048 8b8d 78ff ffff .H.M.H.M.H..x... -00440ff0: 4889 4db8 488b 8d68 ffff ff48 8b95 70ff H.M.H..h...H..p. -00441000: ffff 4889 55b0 4889 4da8 488d 7dd8 488d ..H.U.H.M.H.}.H. -00441010: 55a8 ff50 3048 c743 0800 0000 008b 45d9 U..P0H.C......E. -00441020: 8945 e88a 45d8 0fb7 4ddd 6689 4dec 8a4d .E..E...M.f.M..M -00441030: df88 4dee 3c04 0f85 8000 0000 e800 0000 ..M.<........... -00441040: 0048 89c3 4889 9d48 ffff ff48 8b45 9048 .H..H..H...H.E.H -00441050: 8945 d048 8b45 8848 8945 c848 8b45 8048 .E.H.E.H.E.H.E.H -00441060: 8945 c048 8b85 78ff ffff 4889 45b8 488b .E.H..x...H.E.H. -00441070: 8568 ffff ff48 8b8d 70ff ffff 4889 4db0 .h...H..p...H.M. -00441080: 4889 45a8 488d 7d98 488d b548 ffff ff48 H.E.H.}.H..H...H -00441090: 8d55 a8e8 0000 0000 f048 ff0b 750c 488d .U.......H..u.H. -004410a0: bd48 ffff ffe8 0000 0000 807d 9803 7531 .H.........}..u1 -004410b0: 4881 c4c0 0000 005b 415e 5dc3 8845 9848 H......[A^]..E.H -004410c0: 8b45 e08b 4de8 894d 990f b74d ec66 894d .E..M..M...M.f.M -004410d0: 9d8a 4dee 884d 9f48 8945 a080 7d98 0374 ..M..M.H.E..}..t -004410e0: cf48 8b45 9848 8b4d a048 894d e048 8945 .H.E.H.M.H.M.H.E -004410f0: d848 8d85 38ff ffff 4889 8548 ffff ff48 .H..8...H..H...H -00441100: 8d05 0000 0000 4889 8550 ffff ff48 8d45 ......H..P...H.E -00441110: d848 8985 58ff ffff 488d 0500 0000 0048 .H..X...H......H -00441120: 8985 60ff ffff 488d 0500 0000 0048 8945 ..`...H......H.E -00441130: a848 c745 b002 0000 0048 8d05 0000 0000 .H.E.....H...... -00441140: 4889 45b8 48c7 45c0 0200 0000 488d 8548 H.E.H.E.....H..H -00441150: ffff ff48 8945 c848 c745 d002 0000 0048 ...H.E.H.E.....H -00441160: 8d35 0000 0000 488d 7da8 e800 0000 000f .5....H.}....... -00441170: 0b48 8b3d 0000 0000 ff17 48c7 4008 0000 .H.=......H.@... -00441180: 0000 e98f fdff ff48 8d3d 0000 0000 e800 .......H.=...... -00441190: 0000 000f 0b66 662e 0f1f 8400 0000 0000 .....ff......... +00440eb0: 0f29 4810 b901 0000 0066 480f 6ec9 4883 .)H......fH.n.H. +00440ec0: 3800 660f 7f08 7429 6648 0f7e c348 85db 8.f...t)fH.~.H.. +00440ed0: 741f 660f 70c0 4e66 490f 7ec6 4889 df41 t.f.p.NfI.~.H..A +00440ee0: ff16 4983 7e08 0074 0848 89df e800 0000 ..I.~..t.H...... +00440ef0: 0048 8b3d 0000 0000 ff17 4883 3801 0f85 .H.=......H.8... +00440f00: 8102 0000 488b 3d00 0000 00ff 1748 8378 ....H.=......H.x +00440f10: 0800 7477 e800 0000 0048 89c3 4889 9d48 ..tw.....H..H..H +00440f20: ffff ff48 8b45 9048 8945 d048 8b45 8848 ...H.E.H.E.H.E.H +00440f30: 8945 c848 8b45 8048 8945 c048 8b85 78ff .E.H.E.H.E.H..x. +00440f40: ffff 4889 45b8 488b 8568 ffff ff48 8b8d ..H.E.H..h...H.. +00440f50: 70ff ffff 4889 4db0 4889 45a8 488d 7dd8 p...H.M.H.E.H.}. +00440f60: 488d b548 ffff ff48 8d55 a8e8 0000 0000 H..H...H.U...... +00440f70: f048 ff0b 0f85 a100 0000 488d bd48 ffff .H........H..H.. +00440f80: ffe8 0000 0000 e990 0000 0048 8b3d 0000 ...........H.=.. +00440f90: 0000 ff17 48c7 4008 ffff ffff 488d 4808 ....H.@.....H.H. +00440fa0: 488d 5010 4889 9548 ffff ff48 898d 50ff H.P.H..H...H..P. +00440fb0: ffff 488b 7010 4885 f60f 84b0 0100 0048 ..H.p.H........H +00440fc0: 8b3d 0000 0000 ff17 4889 c348 8b45 9048 .=......H..H.E.H +00440fd0: 8945 d048 8b43 1848 8b4d 8848 894d c848 .E.H.C.H.M.H.M.H +00440fe0: 8b4d 8048 894d c048 8b8d 78ff ffff 4889 .M.H.M.H..x...H. +00440ff0: 4db8 488b 8d68 ffff ff48 8b95 70ff ffff M.H..h...H..p... +00441000: 4889 55b0 4889 4da8 488d 7dd8 488d 55a8 H.U.H.M.H.}.H.U. +00441010: ff50 3048 c743 0800 0000 008b 45d9 8945 .P0H.C......E..E +00441020: e88a 45d8 0fb7 4ddd 6689 4dec 8a4d df88 ..E...M.f.M..M.. +00441030: 4dee 3c04 0f85 8000 0000 e800 0000 0048 M.<............H +00441040: 89c3 4889 9d48 ffff ff48 8b45 9048 8945 ..H..H...H.E.H.E +00441050: d048 8b45 8848 8945 c848 8b45 8048 8945 .H.E.H.E.H.E.H.E +00441060: c048 8b85 78ff ffff 4889 45b8 488b 8568 .H..x...H.E.H..h +00441070: ffff ff48 8b8d 70ff ffff 4889 4db0 4889 ...H..p...H.M.H. +00441080: 45a8 488d 7d98 488d b548 ffff ff48 8d55 E.H.}.H..H...H.U +00441090: a8e8 0000 0000 f048 ff0b 750c 488d bd48 .......H..u.H..H +004410a0: ffff ffe8 0000 0000 807d 9803 7531 4881 .........}..u1H. +004410b0: c4c0 0000 005b 415e 5dc3 8845 9848 8b45 .....[A^]..E.H.E +004410c0: e08b 4de8 894d 990f b74d ec66 894d 9d8a ..M..M...M.f.M.. +004410d0: 4dee 884d 9f48 8945 a080 7d98 0374 cf48 M..M.H.E..}..t.H +004410e0: 8b45 9848 8b4d a048 894d e048 8945 d848 .E.H.M.H.M.H.E.H +004410f0: 8d85 38ff ffff 4889 8548 ffff ff48 8d05 ..8...H..H...H.. +00441100: 0000 0000 4889 8550 ffff ff48 8d45 d848 ....H..P...H.E.H +00441110: 8985 58ff ffff 488d 0500 0000 0048 8985 ..X...H......H.. +00441120: 60ff ffff 488d 0500 0000 0048 8945 a848 `...H......H.E.H +00441130: c745 b002 0000 0048 8d05 0000 0000 4889 .E.....H......H. +00441140: 45b8 48c7 45c0 0200 0000 488d 8548 ffff E.H.E.....H..H.. +00441150: ff48 8945 c848 c745 d002 0000 0048 8d35 .H.E.H.E.....H.5 +00441160: 0000 0000 488d 7da8 e800 0000 000f 0b48 ....H.}........H +00441170: 8b3d 0000 0000 ff17 48c7 4008 0000 0000 .=......H.@..... +00441180: e98f fdff ff48 8d3d 0000 0000 e800 0000 .....H.=........ +00441190: 000f 0b66 6666 662e 0f1f 8400 0000 0000 ...ffff.........which is basically what we see in comment:1.
-
As I expected this is a problem introduced by us switching to 1.26.1. Using version 1.25.0 of the Rust compiler
XULis still reproducible. So, to sum up:Starting with the switch to Rust 1.26.1 Tor Browser for macOS (and only for that platform) is not being built reproducibly anymore, once we enable Stylo. There are small differences visible in
XULstemming fromgkrust(see: comment:1 and comment:4). Using either Rust 1.25.0 or disabling Stylo (what we are doing right now) is "solving" this problem.Alex, Manish: Do you know of any commit that could have caused this? If not, I can bisect I guess.
Trac:
Cc: boklm to boklm, manishearth@gmail.com, acrichton@mozilla.com
Summary: ESR60-based .dmg images are not built reproducibly with Stylo enabled to ESR60-based .dmg images are not built reproducibly with Stylo enabled using rustc > 1.25.0
Status: new to needs_information 
I know of https://github.com/rust-lang/rust/issues/47086 as an macOS-only deterministic compilation problem, but other than that I unfortunately don't know of any macOS-specific deterministic compilation issues as part of the toolchain itself.
Trac:
Username: alexcrichton-
Replying to alexcrichton:
I know of https://github.com/rust-lang/rust/issues/47086 as an macOS-only deterministic compilation problem, but other than that I unfortunately don't know of any macOS-specific deterministic compilation issues as part of the toolchain itself.
Okay, than let's try to track the issue down...
Trac:
Status: needs_information to assigned -
So, I tried for a while to get the bisecting going but it seems non-trivial. Alex: What is the recommended way of doing rust compiler bisecting? What I tried was using
git bisectwith the commit for 1.25.0 and 1.26.1, updating the sub modules and tarring the result up + using it on my build machines, but that breaks for different reasons (first one is no properly vendored crates are included (we compile with --enable-vendor); trying to get crates via crates.io-index fails with an SSL error...).Maybe easier: Are there somewhere nightly source tarballs available (like they are for the official releases) which I could test? I did not find any so far but that might already narrow down the problem sufficiently.
Trac:
Status: assigned to needs_information -
If you're ok downloading prebuilt artifacts I'd recommend bisection via nightlies which rustup can download. Otherwise there's nightly source tarballs as well at urls like https://static.rust-lang.org/dist/2018-06-29/rustc-nightly-src.tar.gz
Trac:
Username: alexcrichton -
There's also this tool, which is kinda like mozregression
https://github.com/rust-lang-nursery/cargo-bisect-rustc
It works off of the tarballs.
I've never tried it with Firefox though.
Trac:
Username: manish.earth -
Okay, I tried the nightlies from 2018-02-25, 2018-03-28, and 2018-05-08 to find a regression range but to my surprise they all are good, meaning I get the same result on different build machines (which does not happen with stable versions > 1.25.0). So, it seems there is something in the stable code but not in the nightlies what is causing this which is confusing to me. We build with
--enable-local-rust --enable-vendor --enable-extended --release-channel=stable --sysconfdir=etc --target=x86_64-apple-darwin --set=target.x86_64-apple-darwin.cc=x86_64-apple-darwin-clangbut it seems that should not make a difference for nightly vs. stable builds. So, hrm...
-
Oh dear that is indeed worrisome! The only difference between nightly and stable builds is
--release-channelpassed to./configureso you should already be emulating our stable builds. It looks like you're cross-compiling rustc from Linux though? For us we compile natively on OSX and I wonder if that causes differences?Trac:
Username: alexcrichton -
Replying to alexcrichton:
Oh dear that is indeed worrisome! The only difference between nightly and stable builds is
--release-channelpassed to./configureso you should already be emulating our stable builds. It looks like you're cross-compiling rustc from Linux though? For us we compile natively on OSX and I wonder if that causes differences?I avoided that by taking the respective nightly sources and compiled them as we compile the stable compiler.
So, the plan right now is to take the nightly sources that are closest to 1.26.1 and remove parts of the diff until I find the problem.
-
Okay, some good news here: I double-checked my setup and it seems I made a mistake which led to no bad nightly being found. I fixed that and the last nightly that is good is the one from March 07. Then there are no nightly sources between 03/08 and 03/14 (inclusive) and the nightly from March 15 is the first bad one.
I fought a bit with using the repo for bisecting (I need to vendor all the crates before creating a tarball, making sure that I really have all the submodules initialized and updated, no
.gitdirs bundled but.gitmodulesfiles available etc.) but it seems to work now. Bisecting...