Add option for firefox incremental builds
In Tor Browser builds, when we need to rebuild firefox, we do a full firefox rebuild. During the development of firefox patches it would be useful to be able to do incremental rebuilds of firefox.
To do that, we could add a new makefile target for this type of build and an option in rbm.local.conf pointing to the firefox sources directory. We would then mount that directory in the build container and use it instead of the sources from git.
Activity
Branch
bug_27045_v2has a patch adding an option to specify a directory containing the firefox sources to use: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27045_v2&id=0db721026831589e87634ff104920b2afd5b1839You should define
var/ffdev_dirinrbm.local.confto select your firefox sources directory. You can then runmake testbuild-ffdev-windows-i686to do a windows-i686 build.pospeselr: could you check if that works for you?
Trac:
Cc: N/A to pospeselr-
Branch
bug_27045_v3is adding some information about this option inREADME.HACKINGandREADME.MAKEFILE: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27045_v3&id=c4ef2e932320ca961a0bab8b454dae9c631e25a8Trac:
Status: new to needs_review
Keywords: N/A deleted, TorBrowserTeam201808R added 
So whenever I try to do an ffdev build (tried both windows i686 and linux x86_64 so far) I get the following error:
Use of uninitialized value $fname in -f at rbm/lib/RBM.pm line 724. Use of uninitialized value $fname in -l at rbm/lib/RBM.pm line 725. Error: Template Error: undef error - Path::Tiny paths require defined, positive-length parts at rbm/lib/RBM.pm line 726. Makefile:113: recipe for target 'testbuild-ffdev-linux-x86_64' failed make: *** [testbuild-ffdev-linux-x86_64] Error 1I have had successful builds in the past, so I'm going to start fresh and see if I somehow managed to mess things up on my end. Will report back later tonight.
-
Confirmed on a fresh clone of your bug_27045_v3 branch. Fails right after binutils.
Did a make testbuild-ffdev-windows-i686 with my local rbm conf setting ffdev to my tor-browser dev directory.
Here's the contents of my tor-browser-build/out directory:
$ tree out out ├── binutils ├── container-image │ ├── container-image_jessie-i386-74dfa60d3344.tar.gz │ └── container-image_jessie-i386-ae006590ff88.tar.gz ├── debootstrap-image │ ├── container-image_jessie-i386.tar.gz │ └── container-image_ubuntu-base-16.04.3-base-amd64.tar.gz ├── firefox │ └── binutils-2.26.1.tar.bz2 ├── mingw-w64 └── tor-browserMy logs dir only has container-image-windows-i686.log and debootstrap-image.log and neither of them have any indication of errors in them.
The host OS is Ubuntu 18.04.
-
Trac:
logs.zip -
Replying to boklm:
Can you post the logs from the console too (not the log files)?
What is needed is the lines before the error, such as for example:
Building project firefox - firefox-a0efd2fcd6e9-linux-i686-fc70a8 Tag tor-browser-60.1.0esr-8.0-1-build2 is signed with key 35CD74C24A9B15A19E1A81A194373AA94B7C3223 Created /home/boklm/tor-browser-build/tmp/rbm-ecVkd/firefox-a0efd2fcd6e9.tar.gz Using file /home/boklm/tor-browser-build/out/container-image/container-image_wheezy-i386-279bd3a261cd.tar.gz Using file /home/boklm/tor-browser-build/out/gcc/gcc-6.4.0-5d1aa7.tar.gz Using file /home/boklm/tor-browser-build/projects/firefox/get-moz-build-date Using file /home/boklm/tor-browser-build/projects/firefox/mozconfig-linux-i686 Using file /home/boklm/tor-browser-build/out/binutils/binutils-2.26.1-914171.tar.gz Using file /home/boklm/tor-browser-build/out/rust/rust-1.26.1-linux-i686-0ce4aa.tar.gz Using file /home/boklm/tor-browser-build/out/llvm/llvm-3.9.1-93f4ef.tar.gz Using file /home/boklm/tor-browser-build/projects/firefox/abicheck.ccThis should show where exactly the error is happening (not knowing where exactly the error is happening makes it difficult to debug).
Trac:
Status: needs_review to needs_information
Keywords: TorBrowserTeam201808R deleted, TorBrowserTeam201808 added -
So building nightly-windows-i686 builds without issue. And having done so, testbuild-ffdev-windows-i686 builds, though rbm will claim the build ultimately fails and dumps me in the build console for firefox:
$ make testbuild-ffdev-windows-i686 git submodule update --init rm -rf out/firefox/firefox-ffdev-windows-i686 rm -rf out/tor-browser/tor-browser-ffdev-windows-i686 ./rbm/rbm build release --target testbuild-ffdev --target torbrowser-windows-i686 Building project tor-browser - tor-browser-ffdev-windows-i686 Using file /media/ssd/Code/Tor/tor-browser-build/out/container-image/container-image_jessie-i386-74dfa60d3344.tar.gz Building project firefox - firefox-ffdev-windows-i686 Using file /media/ssd/Code/Tor/tor-browser-build/out/container-image/container-image_jessie-i386-ae006590ff88.tar.gz Using file /media/ssd/Code/Tor/tor-browser-build/out/mingw-w64/mingw-w64-ee9fc3d0b8c8-54287c.tar.gz Using file /media/ssd/Code/Tor/tor-browser-build/projects/firefox/get-moz-build-date Using file /media/ssd/Code/Tor/tor-browser-build/projects/firefox/mozconfig-windows-i686 Using file /media/ssd/Code/Tor/tor-browser-build/out/firefox/msvcr100.dll Using file /media/ssd/Code/Tor/tor-browser-build/out/gcc/gcc-6.4.0-cc852c.tar.gz Using file /media/ssd/Code/Tor/tor-browser-build/projects/firefox/nsis-uninstall.patch Using file /media/ssd/Code/Tor/tor-browser-build/projects/firefox/1464084.patch Using file /media/ssd/Code/Tor/tor-browser-build/projects/firefox/1467041.patch Using file /media/ssd/Code/Tor/tor-browser-build/out/rust/rust-1.26.1-windows-i686-f18863.tar.gz Using file /media/ssd/Code/Tor/tor-browser-build/out/fxc2/fxc2-079f5c7b9f6e-windows-i686-3304dc.tar.gz [sudo] password for pospeselr: Build log: /media/ssd/Code/Tor/tor-browser-build/logs/firefox-windows-i686.log Error running build Opening debug shell Warning: build files will be removed when you exit this shell. Container directory: /media/ssd/Code/Tor/tor-browser-build/tmp/rbm-4jtGSe/rbm-containers/a7a1eaebd395f44f50017949382a5dbe4c159dcabd19bd71104b24b648b1ab23 /bin/sh: 0: can't access tty; job control turned off debug-firefox$ ls 1464084.patch get-moz-build-date 1467041.patch mingw-w64-ee9fc3d0b8c8-54287c.tar.gz build msvcr100.dll container-image_jessie-i386-ae006590ff88.tar.gz nsis-uninstall.patch fxc2-079f5c7b9f6e-windows-i686-3304dc.tar.gz rust-1.26.1-windows-i686-f18863.tar.gz gcc-6.4.0-cc852c.tar.gzHowever if I open up firefox-windows-i686.log it claims success!
If I completely remove the out directory and try to do an ffdev build:
git submodule update --init rm -rf out/firefox/firefox-ffdev-windows-i686 rm -rf out/tor-browser/tor-browser-ffdev-windows-i686 ./rbm/rbm build release --target testbuild-ffdev --target torbrowser-windows-i686 Building project tor-browser - tor-browser-ffdev-windows-i686 Building project container-image - container-image_jessie-i386-74dfa60d3344.tar.gz Building project debootstrap-image - container-image_jessie-i386.tar.gz --2018-08-21 16:55:19-- http://cdimage.ubuntu.com/ubuntu-base/releases/16.04.3/release/ubuntu-base-16.04.3-base-amd64.tar.gz Resolving cdimage.ubuntu.com (cdimage.ubuntu.com)... 91.189.88.39, 2001:67c:1360:8001::28 Connecting to cdimage.ubuntu.com (cdimage.ubuntu.com)|91.189.88.39|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 46119172 (44M) [application/x-gzip] Saving to: '/media/ssd/Code/Tor/tor-browser-build/out/debootstrap-image/container-image_ubuntu-base-16.04.3-base-amd64.tar.gz' /media/ssd/Code/Tor/tor-brows 100%[===============================================>] 43.98M 10.3MB/s in 5.1s 2018-08-21 16:55:24 (8.66 MB/s) - '/media/ssd/Code/Tor/tor-browser-build/out/debootstrap-image/container-image_ubuntu-base-16.04.3-base-amd64.tar.gz' saved [46119172/46119172] Using file /media/ssd/Code/Tor/tor-browser-build/out/debootstrap-image/container-image_ubuntu-base-16.04.3-base-amd64.tar.gz Build log: /media/ssd/Code/Tor/tor-browser-build/logs/debootstrap-image.log Finished build of project debootstrap-image - container-image_jessie-i386.tar.gz Using file /media/ssd/Code/Tor/tor-browser-build/out/debootstrap-image/container-image_jessie-i386.tar.gz Build log: /media/ssd/Code/Tor/tor-browser-build/logs/container-image-windows-i686.log Finished build of project container-image - container-image_jessie-i386-74dfa60d3344.tar.gz Using file /media/ssd/Code/Tor/tor-browser-build/out/container-image/container-image_jessie-i386-74dfa60d3344.tar.gz Building project firefox - firefox-ffdev-windows-i686 Building project container-image - container-image_jessie-i386-ae006590ff88.tar.gz Using file /media/ssd/Code/Tor/tor-browser-build/out/debootstrap-image/container-image_jessie-i386.tar.gz Build log: /media/ssd/Code/Tor/tor-browser-build/logs/container-image-windows-i686.log tar: ./run/dbus/system_bus_socket: socket ignored Finished build of project container-image - container-image_jessie-i386-ae006590ff88.tar.gz Using file /media/ssd/Code/Tor/tor-browser-build/out/container-image/container-image_jessie-i386-ae006590ff88.tar.gz --2018-08-21 17:01:00-- https://ftp.gnu.org/gnu/binutils/binutils-2.26.1.tar.bz2 Resolving ftp.gnu.org (ftp.gnu.org)... 208.118.235.20, 2001:4830:134:3::b Connecting to ftp.gnu.org (ftp.gnu.org)|208.118.235.20|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 25595243 (24M) [application/x-bzip2] Saving to: '/media/ssd/Code/Tor/tor-browser-build/out/firefox/binutils-2.26.1.tar.bz2' /media/ssd/Code/Tor/tor-brows 100%[===============================================>] 24.41M 1.68MB/s in 18s 2018-08-21 17:01:19 (1.34 MB/s) - '/media/ssd/Code/Tor/tor-browser-build/out/firefox/binutils-2.26.1.tar.bz2' saved [25595243/25595243] Use of uninitialized value $fname in -f at rbm/lib/RBM.pm line 724. Use of uninitialized value $fname in -l at rbm/lib/RBM.pm line 725. Error: Template Error: undef error - Path::Tiny paths require defined, positive-length parts at rbm/lib/RBM.pm line 726. Makefile:133: recipe for target 'testbuild-ffdev-windows-i686' failed make: *** [testbuild-ffdev-windows-i686] Error 1And a subsequent attempt after the error:
git submodule update --init rm -rf out/firefox/firefox-ffdev-windows-i686 rm -rf out/tor-browser/tor-browser-ffdev-windows-i686 ./rbm/rbm build release --target testbuild-ffdev --target torbrowser-windows-i686 Building project tor-browser - tor-browser-ffdev-windows-i686 Using file /media/ssd/Code/Tor/tor-browser-build/out/container-image/container-image_jessie-i386-74dfa60d3344.tar.gz Building project firefox - firefox-ffdev-windows-i686 Using file /media/ssd/Code/Tor/tor-browser-build/out/container-image/container-image_jessie-i386-ae006590ff88.tar.gz --2018-08-21 17:01:41-- https://ftp.gnu.org/gnu/binutils/binutils-2.26.1.tar.bz2 Resolving ftp.gnu.org (ftp.gnu.org)... 208.118.235.20, 2001:4830:134:3::b Connecting to ftp.gnu.org (ftp.gnu.org)|208.118.235.20|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 25595243 (24M) [application/x-bzip2] Saving to: '/media/ssd/Code/Tor/tor-browser-build/out/firefox/binutils-2.26.1.tar.bz2' /media/ssd/Code/Tor/tor-brows 100%[===============================================>] 24.41M 2.55MB/s in 43s 2018-08-21 17:02:25 (580 KB/s) - '/media/ssd/Code/Tor/tor-browser-build/out/firefox/binutils-2.26.1.tar.bz2' saved [25595243/25595243] Use of uninitialized value $fname in -f at rbm/lib/RBM.pm line 724. Use of uninitialized value $fname in -l at rbm/lib/RBM.pm line 725. Error: Template Error: undef error - Path::Tiny paths require defined, positive-length parts at rbm/lib/RBM.pm line 726. Makefile:133: recipe for target 'testbuild-ffdev-windows-i686' failed make: *** [testbuild-ffdev-windows-i686] Error 1 -
Replying to pospeselr:
If I completely remove the out directory and try to do an ffdev build:
Thanks. I can now reproduce the issue, with an empty
outdirectory. It seems to be caused by a bug in rbm, where in some cases we download a file in the wrong directory (in this case we download binutils inout/firefoxinstead ofout/binutils). I am now testing a fix for this. -
I opened #27265 (moved) for the rbm issue.
-
In branch
bug_27045_v5the patch is rebased onbug_27265, which should fix the issue that happened when doing an ffdev build from scratch: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27045_v5&id=187437efa799b7fa61ef878538ed9863c82872adTrac:
Keywords: TorBrowserTeam201811 deleted, TorBrowserTeam201811R added 
Okay, I am finally getting to review this. Sorry for the long delay. I think it's a nice feature to have.
I used my
bug_27045_rebase2branch for testing. Apart from failing to buildStarting build: Mon Dec 16 10:07:49 2019 /var/tmp/dist/gcc/bin ~ ~ mv: cannot create regular file `.mozconfig': Permission deniedhere a couple of issues we might want to address:
a) the
ffdevversion should be available for mobile targets, too, now b) "The patch that may be applied in projects/firefox/build are not applied" s/patch/patches c) we should not hardcodecopyright_yeard)+ "options": [ + "rbind" + ]it seems
]is wrongly indented.Trac:
Keywords: TorBrowserTeam201912R deleted, TorBrowserTeam201912 added
Status: needs_review to needs_revisionmentioned in issue #27265 (moved)
mentioned in issue tpo/applications/rbm#27265 (closed)
