#25531 closed defect (fixed)

Add locks in archive.tpo rsync scripts

Reported by: boklm Owned by: boklm
Priority: Medium Milestone:
Component: Internal Services Version:
Severity: Normal Keywords: boklm201803, AffectsTails
Cc: intrigeri Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We are using cron to run rsync every hour to update archive.tpo. It works fine, except when the rsync takes more than one hour, which cause multiple rsyncs to be running at the same time.

We should avoid that by using flock/lckdo.

Child Tickets

Change History (3)

comment:1 Changed 17 months ago by intrigeri

Cc: intrigeri added

comment:2 Changed 17 months ago by intrigeri

Keywords: AffectsTails added

comment:3 Changed 17 months ago by boklm

Resolution: fixed
Status: assignedclosed

The content of the crontab file was:

# mirror tails livesystem hourly
0 * * * * rsync -rltgoDq --delete --password-file=/home/torarchive/amnesia-rsync-pword.txt rsync://torarchive@rsync.tails.boum.org/tails/ /srv/archive.torproject.org/htdocs/amnesia.boum.org/ 2>&1 | mail -s "rsync on arlgirdense.torproject.org" -E tails-sysadmins@boum.org lunar@torproject.org boklm@torproject.org

# sync tpo/dist to tor package archive hourly.
0 * * * * rsync -aq /srv/static.torproject.org/mirrors/dist.torproject.org/cur/ /srv/archive.torproject.org/htdocs/tor-package-archive/ 2>&1 | mail -s "rsync on arlgirdense.torproject.org" -E boklm@torproject.org

# sync deb.tpo/torproject.org/pool to tor package archive hourly.
0 * * * * rsync -aq /srv/static.torproject.org/mirrors/deb.torproject.org/cur/torproject.org/pool/ /srv/archive.torproject.org/htdocs/tor-package-archive/debs/ 2>&1 | mail -s "rsync on arlgirdense.torproject.org" -E boklm@torproject.org

I have now updated it to:

# mirror tails livesystem hourly
0 * * * * flock -n -E 0 /home/torarchive/sync-tails.lock rsync -rltgoDq --delete --password-file=/home/torarchive/amnesia-rsync-pword.txt rsync://torarchive@rsync.tails.boum.org/tails/ /srv/archive.torproject.org/htdocs/amnesia.boum.org/ 2>&1 | mail -s "rsync on arlgirdense.torproject.org" -E tails-sysadmins@boum.org lunar@torproject.org boklm@torproject.org

# sync tpo/dist to tor package archive hourly.
0 * * * * flock -n -E 0 /home/torarchive/sync-dist.lock rsync -aq /srv/static.torproject.org/mirrors/dist.torproject.org/cur/ /srv/archive.torproject.org/htdocs/tor-package-archive/ 2>&1 | mail -s "rsync on arlgirdense.torproject.org" -E boklm@torproject.org

# sync deb.tpo/torproject.org/pool to tor package archive hourly.
0 * * * * flock -n -E 0 /home/torarchive/sync-deb.lock rsync -aq /srv/static.torproject.org/mirrors/deb.torproject.org/cur/torproject.org/pool/ /srv/archive.torproject.org/htdocs/tor-package-archive/debs/ 2>&1 | mail -s "rsync on arlgirdense.torproject.org" -E boklm@torproject.org
Note: See TracTickets for help on using tickets.