We tried to upgrade our toolchain during the transition to Tor Browser 5.0 but doing that for Binutils fails for various reasons. We should investigate these and fix them.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Child items
0
Show closed items
No child items are currently assigned. Use child items to break down this issue into smaller parts.
Linked items
0
Link issues together to show that they're related.
Learn more.
One issue that got fixed was timestamps not being zeroed out anymore in .dlls. For a reason why this happend see comment:10:ticket:16351ff. commit 1b5ae9017625081f251c71afb359bf9345c76b03 has a fix for it.
The other one resulted in differences in some Firefox related binaries and fte.cDFA.pyd. The diff of the latter between Mike's and my version is attached.
After reverting commit ed700649d0607e6509d5bbc51f4617bbae13a543 from binutils, firefox builds without error on linux-i686.
gold started to detect errors as ld. Why to revert that?
Maybe, adding static to kLogStrings could help?
Adding static to kLogStrings is indeed fixing the issue.
The next error is:
+ exec /var/tmp/dist/gcc/bin/gcc -B/var/tmp/dist/selfrando/Tools/TorBrowser/tc-wrapper -ffunction-sections -fdata-sections -fPIC -m32 -std=gnu99 -shared -Wl,--gc-sections -Wl,-z,defs -Wl,-soname -Wl,libnssutil3.so -Wl,--version-script,/var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/nssutil.def -o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/libnssutil3.so /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/quickder.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/secdig.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/derdec.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/derenc.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/dersubr.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/dertime.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/errstrs.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/nssb64d.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/nssb64e.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/nssrwlk.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/nssilock.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/oidstring.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/pkcs1sig.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/portreg.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/secalgid.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/secasn1d.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/secasn1e.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/secasn1u.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/secitem.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/secload.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/secoid.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/sectime.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/secport.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/templates.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/utf8.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/utilmod.o /var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/utilpars.o -L/var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/dist/lib -L/var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/dist/bin -lplc4 -lplds4 -lnspr4 -lpthread -ldl -lccollect2: fatal error: ld terminated with signal 6 [Aborted]compilation terminated./var/tmp/dist/binutils/bin/ld.gold.real: warning: wildcard match appears in both version 'NSSUTIL_3.12' and 'NSSUTIL_3.12.3' in script/var/tmp/dist/binutils/bin/ld.gold.real: warning: wildcard match appears in both version 'NSSUTIL_3.12.3' and 'NSSUTIL_3.12.5' in script/var/tmp/dist/binutils/bin/ld.gold.real: warning: wildcard match appears in both version 'NSSUTIL_3.12.5' and 'NSSUTIL_3.12.7' in script/var/tmp/dist/binutils/bin/ld.gold.real: warning: wildcard match appears in both version 'NSSUTIL_3.12.7' and 'NSSUTIL_3.13' in script/var/tmp/dist/binutils/bin/ld.gold.real: warning: wildcard match appears in both version 'NSSUTIL_3.13' and 'NSSUTIL_3.14' in script/var/tmp/dist/binutils/bin/ld.gold.real: warning: wildcard match appears in both version 'NSSUTIL_3.14' and 'NSSUTIL_3.15' in script/var/tmp/dist/binutils/bin/ld.gold.real: warning: wildcard match appears in both version 'NSSUTIL_3.15' and 'NSSUTIL_3.17.1' in script/var/tmp/dist/binutils/bin/ld.gold.real: warning: wildcard match appears in both version 'NSSUTIL_3.17.1' and 'NSSUTIL_3.21' in script/var/tmp/dist/binutils/bin/ld.gold.real: warning: wildcard match appears in both version 'NSSUTIL_3.21' and 'NSSUTIL_3.24' in script/var/tmp/dist/binutils/bin/ld.gold.real: error: /tmp/trapobj-Dtc3WlhCcG: relocation R_386_GOTOFF against preemptible symbol SEC_SequenceOfObjectIDTemplate cannot be used when making a shared object/var/tmp/dist/binutils/bin/ld.gold.real: error: /tmp/trapobj-Dtc3WlhCcG: relocation R_386_GOTOFF against preemptible symbol SEC_PointerToGeneralizedTimeTemplate cannot be used when making a shared object/var/tmp/dist/binutils/bin/ld.gold.real: error: /tmp/trapobj-Dtc3WlhCcG: relocation R_386_GOTOFF against preemptible symbol SEC_PointerToEnumeratedTemplate cannot be used when making a shared object/var/tmp/dist/binutils/bin/ld.gold.real: error: /tmp/trapobj-Dtc3WlhCcG: relocation R_386_GOTOFF against preemptible symbol SEC_SequenceOfAnyTemplate cannot be used when making a shared object/var/tmp/dist/binutils/bin/ld.gold.real: error: /tmp/trapobj-cDg6BKy5jw: relocation R_386_GOTOFF against preemptible symbol sgn_DigestInfoTemplate_Util cannot be used when making a shared object/var/tmp/dist/binutils/bin/ld.gold.real: error: /tmp/trapobj-cDg6BKy5jw: relocation R_386_GOTOFF against preemptible symbol SEC_SetOfAnyTemplate_Util cannot be used when making a shared object/var/tmp/dist/binutils/bin/ld.gold.real: error: /tmp/trapobj-cDg6BKy5jw: relocation R_386_GOTOFF against preemptible symbol SEC_PointerToOctetStringTemplate_Util cannot be used when making a shared object/var/tmp/dist/binutils/bin/ld.gold.real: error: /tmp/trapobj-cDg6BKy5jw: relocation R_386_GOTOFF against preemptible symbol SEC_PointerToAnyTemplate_Util cannot be used when making a shared objectLinker execution failed, status: 1ld: src/Support/posix/Debug.cpp:36: void Error::printf(const char*, ...): Assertion `false' failed.make[7]: *** [/var/tmp/build/firefox-017a7ad9488d/obj-i686-pc-linux-gnu/security/nss/lib/util/libnssutil3.so] Error 1
However, after updating gcc to 6.4.0, it seems those two patches are not needed anymore (and make the build fail). So I think we should update both binutils and gcc at the same time.
However, after updating gcc to 6.4.0, it seems those two patches are not needed anymore (and make the build fail). So I think we should update both binutils and gcc at the same time.
Agreed, bumping both at the same time sounds like a good idea, especially if that means we don't need those additional patches.
Have you checked about the deterministic output of the Windows bundles after the switch to the newer binutils version? That has been the main reason for why an update did not happen earlier (see: comment:1).
However, after updating gcc to 6.4.0, it seems those two patches are not needed anymore (and make the build fail). So I think we should update both binutils and gcc at the same time.
It seems I was wrong on that, I had been trying an x86_64 build instead of an i686 one, and on i686 the build still fails with gcc 6.4.0 with the same error as in comment:9.
Hm. Is that caused by Selfrando? I can't otherwise imagine we are the only ones hitting that while compiling Firefox given that binutils 2.27+ and GCC 6.4.0 is not an esoteric compiler/linker combo.
Hm. Is that caused by Selfrando? I can't otherwise imagine we are the only ones hitting that while compiling Firefox given that binutils 2.27+ and GCC 6.4.0 is not an esoteric compiler/linker combo.
Yes, it seems to be caused by Selfrando. I started a build after disabling Selfrando, and the build did not fail in this case.
Hm. Is that caused by Selfrando? I can't otherwise imagine we are the only ones hitting that while compiling Firefox given that binutils 2.27+ and GCC 6.4.0 is not an esoteric compiler/linker combo.
Yes, it seems to be caused by Selfrando. I started a build after disabling Selfrando, and the build did not fail in this case.
Okay, let me get back to the selfrando folks with that.
boklm: We are using an older tag for selfrando. Could you double-check that the issue is still there with the latest code on master? That said I think it is fine dropping selfrando in a separate patch for this bug if that's the only issue blocking the binutils uprade, at least until it is fixed upstream (i.e. on selfrando's side).
boklm: We are using an older tag for selfrando. Could you double-check that the issue is still there with the latest code on master? That said I think it is fine dropping selfrando in a separate patch for this bug if that's the only issue blocking the binutils uprade, at least until it is fixed upstream (i.e. on selfrando's side).
I tried a build using selfrando commit de95c99364214d631df01364e0e87e6b6c307eba (current master), and it also fails with the error from comment:9.
Have you checked about the deterministic output of the Windows bundles after the switch to the newer binutils version? That has been the main reason for why an update did not happen earlier (see: comment:1).
It seems it is not deterministic. I did two builds using branch boklm/bug_16472_v3, and have some differences in the Windows builds. I will attach the output of diffoscope on tor-win32-0.3.3.2-alpha.zip.
boklm: could you test the selfrando commit d2951ac1d3747a4d8a9ef222d38d2d13e5ae2ba9 and report back whether that solved our problems? ahomescu said we should be good with that one.
boklm: could you test the selfrando commit d2951ac1d3747a4d8a9ef222d38d2d13e5ae2ba9 and report back whether that solved our problems? ahomescu said we should be good with that one.
It seems it fixed the previous error, however we now have a different error:
Executing /var/tmp/build/firefox-8ee6fdadea2a/obj-i686-pc-linux-gnu/dist/bin/shlibsign -v -o ../../dist/firefox/libsoftokn3.chk -i ../../dist/firefox/libsoftokn3.soTraceback (most recent call last): File "/var/tmp/build/firefox-8ee6fdadea2a/toolkit/mozapps/installer/packager.py", line 341, in <module> main() File "/var/tmp/build/firefox-8ee6fdadea2a/toolkit/mozapps/installer/packager.py", line 337, in main copier.copy(args.destination) File "/var/tmp/build/firefox-8ee6fdadea2a/python/mozbuild/mozpack/copier.py", line 399, in copy copy_results.append((destfile, f.copy(destfile, skip_if_older))) File "/var/tmp/build/firefox-8ee6fdadea2a/toolkit/mozapps/installer/packager.py", line 124, in copy errors.fatal('Error while signing %s' % self.path) File "/var/tmp/build/firefox-8ee6fdadea2a/python/mozbuild/mozpack/errors.py", line 103, in fatal self._handle(self.FATAL, msg) File "/var/tmp/build/firefox-8ee6fdadea2a/python/mozbuild/mozpack/errors.py", line 98, in _handle raise ErrorMessage(msg)mozpack.errors.ErrorMessage: Error: Error while signing ../../dist/firefox/libsoftokn3.somake[3]: *** [stage-package] Error 1make[3]: Leaving directory `/var/tmp/build/firefox-8ee6fdadea2a/obj-i686-pc-linux-gnu/browser/installer'
Good stuff! I am a bit confused about why the upgrade to binutils 2.30 suddenly needs all those nsis related build changes. I mean the hardening flags are not really changing just with the binutils update... So, what's up with that?
Good stuff! I am a bit confused about why the upgrade to binutils 2.30 suddenly needs all those nsis related build changes. I mean the hardening flags are not really changing just with the binutils update... So, what's up with that?
The reason is that binutils 2.25 added this change:
* PE binaries now once again contain real timestamps by default. To disable the inclusion of a timestamp in a PE binary, use the --no-insert-timestamp command line option.
So we need to add the --no-insert-timestamp flag to make the build reproducible, which was not necessary with binutils 2.24.
Alternatively, we could patch ld/emultempl/pe.em and change this line to make it false by default:
Isn't it better just to upgrade nsis to 3.x version which Tom uses?
Hardening on Windows is harmless for all programs, except ancient crap. But if nsis has problems with ASLR, why do you add --enable-reloc-section? As you removed SSP, why -lssp?
+# the nsis build system expect all toolchain binaries to be in the same
*expects
Good stuff! I am a bit confused about why the upgrade to binutils 2.30 suddenly needs all those nsis related build changes. I mean the hardening flags are not really changing just with the binutils update... So, what's up with that?
The reason is that binutils 2.25 added this change:
{{{
PE binaries now once again contain real timestamps by default. To disable
the inclusion of a timestamp in a PE binary, use the --no-insert-timestamp
command line option.
}}}
So we need to add the --no-insert-timestamp flag to make the build reproducible, which was not necessary with binutils 2.24.
I understand that part and that's not the thing that confuses me. Before the patch we had
"# remove hardening wrappers" but now we have "# Some of the hardening flags are causing the build to fail, so we overwrite the helpers with only the flags required to make the build reproducible." So, why do we have suddenly the need for the hardening option -Wl,--enable-reloc-section? Just because we need to deal with --no-insert-timestamp?
Alternatively, we could patch ld/emultempl/pe.em and change this line to make it false by default:
{{{
static bfd_boolean insert_timestamp = TRUE;
}}}
I understand that part and that's not the thing that confuses me. Before the patch we had
"# remove hardening wrappers" but now we have "# Some of the hardening flags are causing the build to fail, so we overwrite the helpers with only the flags required to make the build reproducible." So, why do we have suddenly the need for the hardening option -Wl,--enable-reloc-section? Just because we need to deal with --no-insert-timestamp?
Ah, I think we only need --no-insert-timestamp and not the other options. I will make a new revision of the patch keeping only --no-insert-timestamp.
I made test builds with your patch but both the 32bit and the 64bit bundles are crashing for me right on start. In the debugger I see tons of messages like
warning: 0f64:0944 @ 01846240 - LdrpSnapThunk - WARNING: Hint index 0x119 for procedure "NtEnumerateKey" in DLL "ntdll.dll" is invalidwarning: 0f64:0944 @ 01846256 - LdrpSnapThunk - WARNING: Hint index 0x3bb for procedure "RtlIntegerToUnicodeString" in DLL "ntdll.dll" is invalidwarning: 0f64:0944 @ 01846287 - LdrpSnapThunk - WARNING: Hint index 0x26e for procedure "RtlAppendUnicodeStringToString" in DLL "ntdll.dll" is invalidwarning: 0f64:0944 @ 01846303 - LdrpSnapThunk - WARNING: Hint index 0x4a7 for procedure "RtlStringFromGUID" in DLL "ntdll.dll" is invalid```.**Trac**: **Status**: needs_review **to** needs_revision **Keywords**: TorBrowserTeam201803R **deleted**, TorBrowserTeam201803 **added**
A Windows-i686 build (I did not try x86_64 yet) using binutils 2.29 is also working fine. So it seems the crash is caused by some change between binutils 2.29 and 2.30.
I made test builds with your patch but both the 32bit and the 64bit bundles are crashing for me right on start. In the debugger I see tons of messages like
{{{
warning: 0f64:0944 @ 01846240 - LdrpSnapThunk - WARNING: Hint index 0x119 for pr
ocedure "NtEnumerateKey" in DLL "ntdll.dll" is invalid
warning: 0f64:0944 @ 01846256 - LdrpSnapThunk - WARNING: Hint index 0x3bb for pr
ocedure "RtlIntegerToUnicodeString" in DLL "ntdll.dll" is invalid
warning: 0f64:0944 @ 01846287 - LdrpSnapThunk - WARNING: Hint index 0x26e for pr
ocedure "RtlAppendUnicodeStringToString" in DLL "ntdll.dll" is invalid
warning: 0f64:0944 @ 01846303 - LdrpSnapThunk - WARNING: Hint index 0x4a7 for pr
ocedure "RtlStringFromGUID" in DLL "ntdll.dll" is invalid
}}}.
bump mingw-w64 ;)
Building firefox with master from mingw-w64 fails with:
{{{
i686-w64-mingw32-widl: /var/tmp/build/mingw-w64-b633824ecafd/mingw-w64-tools/widl/src/typetree.h:274: type_alias_get_aliasee: Assertion `type_is_alias(type)' failed.
Aborted
Makefile:109: recipe for target 'typelib_done' failed
}}}
Yeah, we should not do that right now, at least not until we found the issue why the crash is happening (fwiw I notified Jacek about it, see the stylo bug on Mozilla's bug tracker as a patch from him is causing that breakage). I think picking the latest stable that works and filing an upstream bug (making sure it is still unfixed) seems like a good idea to me.
Building firefox with master from mingw-w64 fails with:
{{{
i686-w64-mingw32-widl: /var/tmp/build/mingw-w64-b633824ecafd/mingw-w64-tools/widl/src/typetree.h:274: type_alias_get_aliasee: Assertion type_is_alias(type)' failed. Aborted Makefile:109: recipe for target 'typelib_done' failed }}} It can be something recent and close to binutils 2.30 release date, not necessary master`.
Yeah, we should not do that right now,
What about Nightlies?
at least not until we found the issue why the crash is happening
Do you mean https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=9d9c67b06c1bf4c4550e3de0eb575c2bfbe96df9 ?
(fwiw I notified Jacek about it, see the stylo bug on Mozilla's bug tracker as a patch from him is causing that breakage).
Where?
I think picking the latest stable that works
2.29.1 ?
and filing an upstream bug (making sure it is still unfixed) seems like a good idea to me.
Hrm...