Opened 3 years ago

Closed 2 years ago

#22467 closed defect (fixed)

System upgrade on a Jessie VM in Gitian is randomly failing

Reported by: gk Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-gitian, TorBrowserTeam201708
Cc: dcf, boklm, josephbisch Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Sometimes upgrades on Jessie VM's in Gitian are failing and the only workaround I've found so far is recreating the base VM or copying one over that is still working.

Child Tickets

Change History (6)

comment:1 Changed 3 years ago by gk

Cc: boklm added

dcf did some debugging and traced the down to:

If I set the -x flag in target-bin/upgrade-system.sh to get some debugging output, I see an error relating to a missing /usr/sbin/update-grub:

$ ssh -o NoHostAuthenticationForLocalhost=yes root@127.0.0.1 -p 2223 -i var/id_rsa < target-bin/upgrade-system.sh
Pseudo-terminal will not be allocated because stdin is not a terminal.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
stdin: is not a tty
+ mkdir -p /var/cache/gitian
+ apt-get -q -y purge grub
+ true
+ DEBIAN_FRONTEND=noninteractive
+ apt-get -y dist-upgrade
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  bcron fgetty grub-common lib32ncurses5 lib32tinfo5 libbg1 libbg1-doc
  libc6-i386 libfuse2 os-prober runit ucspi-unix
Use 'apt-get autoremove' to remove them.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up linux-image-3.16.0-4-amd64 (3.16.43-2) ...
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
Running /usr/sbin/update-grub.
Can't exec "/usr/sbin/update-grub": No such file or directory at /var/lib/dpkg/info/linux-image-3.16.0-4-amd64.postinst line 604.
User postinst hook script [/usr/sbin/update-grub] failed to execute: No such file or directory
dpkg: error processing package linux-image-3.16.0-4-amd64 (--configure):
 subprocess installed post-installation script returned error exit status 255
Errors were encountered while processing:
 linux-image-3.16.0-4-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

(originally reported in #22398)

comment:2 Changed 3 years ago by dcf

I encountered this problem again today during Building sandbox-linux for jessie amd64. I tried applying the following workaround and it seems to work. The problem seems to be related to linux-image-xxxx trying to upgrade itself during dist-upgrade while grub is not installed, which causes the post-installation script to fail. My idea was to ensure that linux-image-xxxx is upgraded before removing grub and doing dist-upgrade.

diff --git a/target-bin/upgrade-system.sh b/target-bin/upgrade-system.sh
index 9384229..795c3b9 100644
--- a/target-bin/upgrade-system.sh
+++ b/target-bin/upgrade-system.sh
@@ -6,6 +6,9 @@ set -e
 
 mkdir -p /var/cache/gitian
 
+DEBIAN_FRONTEND=noninteractive apt-get -y install grub
+DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-$(uname -r)
+
 # remove obsolete grub, it causes package dependency issues
 apt-get -q -y purge grub > /dev/null 2>&1 || true
 

comment:3 Changed 3 years ago by gk

Cc: josephbisch added
Keywords: TorBrowserTeam201707 added
Status: newneeds_information

Thanks, adding josephbisch who did work on the Debian related Gitian parts. josephbisch: Do you think that's a good way forward, especially regarding upstreaming? If not, do you have better ideas we should try?

(FWIW: dcf's change fixed the problem for me as well)

comment:4 Changed 2 years ago by gk

dcf: If you want to make a patch for your workaround, feel free to do so. I'd take it for the build this week. If you don't then I'll do it myself later this week to get us unblocked on this issue.

comment:5 Changed 2 years ago by gk

Keywords: TorBrowserTeam201708 added; TorBrowserTeam201707 removed

Moving our Tickets to August.

comment:6 Changed 2 years ago by gk

Resolution: fixed
Status: needs_informationclosed

Okay, I went ahead and put the workaround into a proper patch and committed it to tor-browser-builder-4 (commit cbcaacd2152ca43ebc51874dcbb074de14dcc392). Thanks dcf.

Note: See TracTickets for help on using tickets.