Opened 6 years ago

Closed 5 years ago

#10153 closed defect (fixed)

Gitian is often hanging while building Firefox on Mac OS X

Reported by: gk Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Keywords: gitian, tbb-gitian
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

To keep our own sanity and that of people that like to help us building TBB the Gitian way we should make sure the Firefox build for Mac OS X is not hanging anymore (so often). The last output in the build log before the freeze is usually (at least for me) something like:

/home/ubuntu/build/x-tools/x86_64-apple-darwin10/bin/x86_64-apple-darwin10-strip -x -S firefox
/home/ubuntu/build/tor-browser/obj-macos/config/nsinstall -t -m 755 "firefox" "../../dist/bin"
cp -p firefox ../../dist/bin/firefox-bin
/home/ubuntu/build/tor-browser/obj-macos/config/nsinstall -t -m 644 /home/ubuntu/build/tor-browser/browser/app/profile/prefs.js ../../dist/bin/browser/defaults/profile
/home/ubuntu/build/tor-browser/obj-macos/config/nsinstall -D ../../dist/bin/defaults/pref
/home/ubuntu/build/tor-browser/obj-macos/_virtualenv/bin/python /home/ubuntu/build/tor-browser/config/Preprocessor.py  -DCROSS_COMPILE=1 -DX_DISPLAY_MISSING=1 -DJSGC_INCREMENTAL=1 -DMOZ_ENABLE_PROFILER_SPS=1 -DMOZILLA_VERSION=\"24.1.0\" -DMOZILLA_VERSION_U=24.1.0 -DMOZILLA_UAVERSION=\"24.0\" -DXP_MACOSX=1 -DXP_DARWIN=1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_STDCALL=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_UINT=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_CPUID_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_NETINET_IN_H=1 -DHAVE_SIN_LEN=1 -DHAVE_SCONN_LEN=1 -DHAVE_SIN6_LEN=1 -DHAVE_SA_LEN=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLADDR=1 -DNO_X11=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_STAT64=1 -DHAVE_LSTAT64=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_LANGINFO_CODESET=1 -DVA_COPY=va_copy -DHAVE_VA_COPY=1 -DMALLOC_H=\<malloc/malloc.h\> -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_LOCALECONV=1 -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DRELEASE_BUILD=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_GOOGLE_API_KEY=no-google-api-key -DEARLY_BETA_OR_EARLIER=1 -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_WIDGET_COCOA=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_OFFICIAL_BRANDING=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DACCESSIBILITY=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_WEBSPEECH=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DMOZ_MEDIA_NAVIGATOR=1 -DMOZ_VP8=1 -DVPX_X86_ASM=1 -DMOZ_WAVE=1 -DMOZ_SPEEX_RESAMPLER=1 -DMOZ_SOUNDTOUCH=1 -DMOZ_CUBEB=1 -DMOZ_VORBIS=1 -DMOZ_OPUS=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DMOZ_GAMEPAD=1 -DBREAKPAD_CUSTOM_STDINT_H=\"mozilla/StandardInteger.h\" -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DLIBJPEG_TURBO_X86_ASM=1 -DMOZ_WEBAPP_RUNTIME=1 -DMOZ_FEEDS=1 -DMOZ_SAFE_BROWSING=1 -DMOZ_URL_CLASSIFIER=1 -DGL_PROVIDER_=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DJSGC_INCREMENTAL=1 -DHAVE___CXA_DEMANGLE=1 -DHAVE__UNWIND_BACKTRACE=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\"Mozilla\" -DMOZ_STATIC_JS=1 -DMOZ_TREE_PIXMAN=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_ENABLE_SKIA=1 -DUSE_SKIA=1 -DUSE_SKIA_GPU=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SOCIAL=1 -DMOZ_SERVICES_COMMON=1 -DMOZ_SERVICES_CRYPTO=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_SERVICES_METRICS=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_MACBUNDLE_ID=org.mozilla.torbrowser -DMOZ_B2G_VERSION=\"1.0.0\" -DMOZ_B2G_OS_NAME=\"\" -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"24.1.0\" -DFIREFOX_VERSION=24.1.0 -DMOZ_UA_BUILDID=\"20100101\" -DMOZ_TELEMETRY_DISPLAY_REV=2 -DMOZ_DATA_REPORTING=1 -DMOZ_DLL_SUFFIX=\".dylib\" -DXP_UNIX=1 /home/ubuntu/build/tor-browser/browser/app/profile/channel-prefs.js > ../../dist/bin/defaults/pref/channel-prefs.js
/home/ubuntu/build/tor-browser/obj-macos/config/nsinstall -t -m 644 /home/ubuntu/build/tor-browser/browser/app/blocklist.xml ../../dist/bin/browser
mkdir -p ../../dist/TorBrowser.app/Contents/MacOS
mkdir -p ../../dist/TorBrowser.app/Contents/Resources/en.lproj
rsync -a --exclude "*.in" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents ../../dist/TorBrowser.app --exclude English.lproj
rsync -a --exclude "*.in" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents/Resources/English.lproj/ ../../dist/TorBrowser.app/Contents/Resources/en.lproj
sed -e "s/%APP_VERSION%/24.1.0/" -e "s/%MAC_APP_NAME%/TorBrowser/" -e "s/%MOZ_MACBUNDLE_ID%/org.mozilla.torbrowser/" -e "s/%MAC_BUNDLE_VERSION%/2400.1.1/" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents/Info.plist.in > ../../dist/TorBrowser.app/Contents/Info.plist
sed -e "s/%MAC_APP_NAME%/TorBrowser/" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents/Resources/English.lproj/InfoPlist.strings.in | iconv -f UTF-8 -t UTF-16 > ../../dist/TorBrowser.app/Contents/Resources/en.lproj/InfoPlist.strings
rsync -a ../../dist/bin/ ../../dist/TorBrowser.app/Contents/MacOS
rm -f ../../dist/TorBrowser.app/Contents/MacOS/firefox
rsync -aL firefox ../../dist/TorBrowser.app/Contents/MacOS

Child Tickets

Attachments (5)

lsof12934 (2.1 KB) - added by gk 6 years ago.
lsof12935 (1.9 KB) - added by gk 6 years ago.
netstatrsync (284 bytes) - added by gk 6 years ago.
strace12934 (651 bytes) - added by gk 6 years ago.
strace12935 (750 bytes) - added by gk 6 years ago.

Download all attachments as: .zip

Change History (15)

comment:1 Changed 6 years ago by gk

Here is some debugging information about both rsync processes that were frozen...

Last edited 6 years ago by gk (previous) (diff)

Changed 6 years ago by gk

Attachment: lsof12934 added

Changed 6 years ago by gk

Attachment: lsof12935 added

Changed 6 years ago by gk

Attachment: netstatrsync added

Changed 6 years ago by gk

Attachment: strace12934 added

Changed 6 years ago by gk

Attachment: strace12935 added

comment:2 Changed 5 years ago by dcf

I got this too today for the first time, while building a branch off of tbb-3.5.2.1-build2.

/home/ubuntu/build/tor-browser/obj-macos/config/nsinstall -t -m 644 /home/ubuntu/build/tor-browser/browser/app/blocklist.xml ../../dist/bin/browser
mkdir -p ../../dist/TorBrowser.app/Contents/MacOS
mkdir -p ../../dist/TorBrowser.app/Contents/Resources/en.lproj
rsync -a --exclude "*.in" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents ../../dist/TorBrowser.app --exclude English.lproj
rsync -a --exclude "*.in" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents/Resources/English.lproj/ ../../dist/TorBrowser.app/Contents/Resources/en.lproj

comment:3 Changed 5 years ago by ln5

I'm at the exact same spot as dcf. gdb confirmes gk's observation that we're in select():

(gdb) bt
#0  0x00002b8cfe300743 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000419ca4 in ?? ()
#2  0x000000000041cf5a in ?? ()
#3  0x000000000041dbbb in ?? ()
#4  0x0000000000404253 in ?? ()
#5  0x00002b8cfe23476d in __libc_start_main ()
   from /lib/x86_64-linux-gnu/libc.so.6
#6  0x00000000004047d5 in ?? ()
#7  0x00007fff0e375118 in ?? ()
#8  0x000000000000001c in ?? ()
#9  0x0000000000000008 in ?? ()
#10 0x00007fff0e375c5c in ?? ()
#11 0x00007fff0e375c62 in ?? ()
#12 0x00007fff0e375c65 in ?? ()
#13 0x00007fff0e375c6f in ?? ()
#14 0x00007fff0e375c74 in ?? ()
#15 0x00007fff0e375cb1 in ?? ()
#16 0x00007fff0e375ccb in ?? ()
#17 0x00007fff0e375cd5 in ?? ()
#18 0x0000000000000000 in ?? ()

comment:4 Changed 5 years ago by erinn

It's also worth noting that rsync -a --exclude is only being run in the OSX build for Firefox and doesn't appear elsewhere in the source (at least based on a simple grep I did). In the end it might be easier to just patch Firefox to copy those files a different way.

comment:5 in reply to:  4 Changed 5 years ago by gk

Replying to erinn:

It's also worth noting that rsync -a --exclude is only being run in the OSX build for Firefox and doesn't appear elsewhere in the source (at least based on a simple grep I did). In the end it might be easier to just patch Firefox to copy those files a different way.

Well, yes, maybe. I am not so happy patching Firefox for this issue, though, as we want to get all patches merged upstream and this is just a bug in our toolchain for which there is no big incentive on the Mozilla side to care about (let alone that this is strictly speaking no Firefox bug). Maybe this is a problem already fixed in a newer rsync version? I gonna try spending some cycles next week to ask for help on the rsync mailing list as a first step.

comment:6 Changed 5 years ago by dcf

I hit this again today, building a branch off of 3.6.1-build1. It's a slightly different place (third rsync rather than second).

mkdir -p ../../dist/TorBrowser.app/Contents/MacOS
mkdir -p ../../dist/TorBrowser.app/Contents/Resources/en.lproj
rsync -a --exclude "*.in" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents ../../dist/TorBrowser.app --exclude English.lproj
rsync -a --exclude "*.in" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents/Resources/English.lproj/ ../../dist/TorBrowser.app/Contents/Resources/en.lproj
sed -e "s/%APP_VERSION%/24.5.0/" -e "s/%MAC_APP_NAME%/TorBrowser/" -e "s/%MOZ_MACBUNDLE_ID%/org.mozilla.torbrowser/" -e "s/%MAC_BUNDLE_VERSION%/2400.1.1/" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents/Info.plist.in > ../../dist/TorBrowser.app/Contents/Info.plist
sed -e "s/%MAC_APP_NAME%/TorBrowser/" /home/ubuntu/build/tor-browser/browser/app/macbuild/Contents/Resources/English.lproj/InfoPlist.strings.in | iconv -f UTF-8 -t UTF-16 > ../../dist/TorBrowser.app/Contents/Resources/en.lproj/InfoPlist.strings
rsync -a ../../dist/bin/ ../../dist/TorBrowser.app/Contents/MacOS

comment:7 Changed 5 years ago by erinn

Component: Tor bundles/installationTor Browser
Keywords: tbb-gitian added
Owner: changed from erinn to tbb-team

comment:8 Changed 5 years ago by gk

rsync master does not fix the problem.

comment:9 Changed 5 years ago by gk

This will be fixed by #12812 which makes it necessary to deploy a newer libfaketime version and exclude rsync properly from it.

comment:10 Changed 5 years ago by gk

Resolution: fixed
Status: newclosed

Fixed by commit 82153228c675451dc5b37d8302beff0a93e981e4 in the tor-browser-bundle repo.

Note: See TracTickets for help on using tickets.