Opened 3 years ago

Closed 2 years ago

Last modified 2 years ago

#26193 closed defect (fixed)

Tarballs are not compressed in a run following an aborted run

Reported by: karsten Owned by: karsten
Priority: Medium Milestone: CollecTor 1.7.0
Component: Metrics/CollecTor Version:
Severity: Normal Keywords:
Cc: metrics-team Actual Points:
Parent ID: Points:
Reviewer: irl Sponsor:


Today I wondered why some of our 2018-05 tarballs are over 3 days old even though we're creating new tarballs every 2 days. And I wondered even more after finding a seemingly completed run from yesterday. Here's what happened:

  • The May 19 run succeeded without issues.
  • The May 21 run was interrupted, probably due to a host reboot. Apparently, it finished compressing tarballs except for bridge extra-info descriptors, but it did not move any of them in place.
  • The May 23 run went through, but it did not compress any tarballs except for bridge extra-info descriptors.

I think this is related to line 102 in the script:

for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
  echo `date` "Creating" ${TARBALLS[$i]}'.tar'
  tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
  if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then           # <- this one
    echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
    xz -9e ${TARBALLS[$i]}.tar

Explanation: there were still compressed files in the working directory from May 21, and as a result we did not attempt to compress new tarballs from May 23.

Suggested fix: remove that if statement and compress the tarball regardless of whether a previous compressed tarball exists in the working directory using xz -9e -f:

diff --git a/src/main/resources/ b/src/main/resources/
index cd16b2dc..d247c520 100755
--- a/src/main/resources/
+++ b/src/main/resources/
@@ -99,10 +99,8 @@ done
 for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
   echo `date` "Creating" ${TARBALLS[$i]}'.tar'
   tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
-  if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then
-    echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
-    xz -9e ${TARBALLS[$i]}.tar
-  fi
+  echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
+  xz -9e -f ${TARBALLS[$i]}.tar
 cd $OUTDIR/webstats/

Child Tickets

Change History (6)

comment:1 Changed 3 years ago by karsten

Status: assignedneeds_review

comment:2 Changed 3 years ago by iwakeh

Is it lighter on memory when doing two separate steps? I.e. replace the tar and xz command by tar -I "xz -9e" -chf ${TARBALLS[$i]}.tar.xz?

But the solution seems ok in general, no matter if the tar&compress step is on or two lines.

comment:3 Changed 2 years ago by irl

Reviewer: irl

Will start review this week.

comment:4 Changed 2 years ago by irl

Status: needs_reviewmerge_ready

Looks good to me.

I don't think there is a need to combine the tar and xz steps into a single command, it's more readable as separate steps. There would probably be a speed up due to less disk I/O to be had if we really needed it.

comment:5 Changed 2 years ago by karsten

Resolution: fixed
Status: merge_readyclosed

Great! Fixed in master. Closing. Thanks!

comment:6 Changed 2 years ago by karsten

Milestone: CollecTor 1.7.0
Note: See TracTickets for help on using tickets.