System upgrade on a Jessie VM in Gitian is randomly failing

added TorBrowserTeam201708 component::applications/tor browser owner::tbb-team priority::medium resolution::fixed severity::normal status::closed tbb-gitian type::defect labels

dcf did some debugging and traced the down to:

If I set the -x flag in target-bin/upgrade-system.sh to get some debugging output, I see an error relating to a missing /usr/sbin/update-grub:

$ ssh -o NoHostAuthenticationForLocalhost=yes root@127.0.0.1 -p 2223 -i var/id_rsa < target-bin/upgrade-system.sh
Pseudo-terminal will not be allocated because stdin is not a terminal.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
stdin: is not a tty
+ mkdir -p /var/cache/gitian
+ apt-get -q -y purge grub
+ true
+ DEBIAN_FRONTEND=noninteractive
+ apt-get -y dist-upgrade
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  bcron fgetty grub-common lib32ncurses5 lib32tinfo5 libbg1 libbg1-doc
  libc6-i386 libfuse2 os-prober runit ucspi-unix
Use 'apt-get autoremove' to remove them.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up linux-image-3.16.0-4-amd64 (3.16.43-2) ...
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
Running /usr/sbin/update-grub.
Can't exec "/usr/sbin/update-grub": No such file or directory at /var/lib/dpkg/info/linux-image-3.16.0-4-amd64.postinst line 604.
User postinst hook script [/usr/sbin/update-grub] failed to execute: No such file or directory
dpkg: error processing package linux-image-3.16.0-4-amd64 (--configure):
 subprocess installed post-installation script returned error exit status 255
Errors were encountered while processing:
 linux-image-3.16.0-4-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

(originally reported in #22398 (moved))

Trac:
Cc: dcf to dcf, boklm

I encountered this problem again today during Building sandbox-linux for jessie amd64. I tried applying the following workaround and it seems to work. The problem seems to be related to linux-image-xxxx trying to upgrade itself during dist-upgrade while grub is not installed, which causes the post-installation script to fail. My idea was to ensure that linux-image-xxxx is upgraded before removing grub and doing dist-upgrade.

diff --git a/target-bin/upgrade-system.sh b/target-bin/upgrade-system.sh
index 9384229..795c3b9 100644
--- a/target-bin/upgrade-system.sh
+++ b/target-bin/upgrade-system.sh
@@ -6,6 +6,9 @@ set -e
 
 mkdir -p /var/cache/gitian
 
+DEBIAN_FRONTEND=noninteractive apt-get -y install grub
+DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-$(uname -r)
+
 # remove obsolete grub, it causes package dependency issues
 apt-get -q -y purge grub > /dev/null 2>&1 || true

Thanks, adding josephbisch who did work on the Debian related Gitian parts. josephbisch: Do you think that's a good way forward, especially regarding upstreaming? If not, do you have better ideas we should try?

(FWIW: dcf's change fixed the problem for me as well)

Trac:
Keywords: N/A deleted, TorBrowserTeam201707 added
Status: new to needs_information
Cc: dcf, boklm to dcf, boklm, josephbisch

dcf: If you want to make a patch for your workaround, feel free to do so. I'd take it for the build this week. If you don't then I'll do it myself later this week to get us unblocked on this issue.

Moving our Tickets to August.

Trac:
Keywords: TorBrowserTeam201707 deleted, TorBrowserTeam201708 added

Okay, I went ahead and put the workaround into a proper patch and committed it to tor-browser-builder-4 (commit cbcaacd2152ca43ebc51874dcbb074de14dcc392). Thanks dcf.

Trac:
Resolution: N/A to fixed
Status: needs_information to closed

closed

moved to tpo/applications/tor-browser#22467 (closed)

System upgrade on a Jessie VM in Gitian is randomly failing

Child items 0

Activity