#31051 closed defect (fixed)

bungei running out of space

Reported by: anarcat Owned by: anarcat
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

we got this warning from nagios this weekend:

DISK WARNING - free space: /srv/backups/bacula 1886723 MB (9% inode=99%)

indeed, the main storage is full:

Sys. de fichiers                    Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/vg_bulk-backups--bacula    20T     18T  1,6T  93% /srv/backups/bacula

There's plenty of free space on the VG, however:

  VG      #PV #LV #SN Attr   VSize  VFree 
  vg_bulk   1   2   0 wz--n- 72,60t 51,60t

So this could quickly be resolved by just resizing that partition. But it's interesting to note that the vast majority of storage in use comes from the static content. I suspect there's a lot of duplication around those and we could save space in backups by being a little smarter about this:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help                                                                                                                                   
--- /srv/backups/bacula -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
    4,3 TiB [ 25,0% ##########] /arlgirdense.torproject.org                                                                                                                                    
    4,3 TiB [ 24,5% ######### ] /staticiforme.torproject.org
    2,1 TiB [ 11,9% ####      ] /archive-01.torproject.org
    1,1 TiB [  6,2% ##        ] /perdulce.torproject.org
  871,0 GiB [  4,9% #         ] /colchicifolium.torproject.org
  806,1 GiB [  4,5% #         ] /Catalog
  554,4 GiB [  3,1% #         ] /corsicum.torproject.org
  375,1 GiB [  2,1%           ] /gillii.torproject.org
  237,5 GiB [  1,3%           ] /eugeni.torproject.org
  209,0 GiB [  1,2%           ] /hetzner-nbg1-01.torproject.org

Above is the top-ten in disk usage. It's interesting to note that archive01 is only half of its partners. We should also note that arlgirdense is likely going to go away eventually, because archive-01 will replace it.

Child Tickets

Change History (7)

comment:1 Changed 16 months ago by weasel

manually, and behind bacula's back, removed one of arlgirdense's old fulls. We can reconsider growing things once we have migrated properly to the new archive host.

comment:2 Changed 15 months ago by anarcat

i don't quite understand how to delete old entries in the database and/or the storage. not "behind bacula's back" as that sounds ominous... but i think the rest of arlgirdense can be removed now that archive-01 is up and running. i just don't know how.

see also #30880 for that documentation.

comment:3 Changed 13 months ago by anarcat

Resolution: fixed
Status: newclosed

i think this is done. the files can be removed directly from the filesystem, and stuff in the database gets pruned semi-automatically.

comment:4 Changed 11 months ago by anarcat

Resolution: fixed
Status: closedreopened

this is a problem again today.

root@bungei:~# df -h 
Sys. de fichiers                    Taille Utilisé Dispo Uti% Monté sur
udev                                   63G       0   63G   0% /dev
tmpfs                                  13G    1,3G   12G  11% /run
/dev/mapper/croot                      20G    2,9G   16G  16% /
tmpfs                                  63G     72K   63G   1% /dev/shm
tmpfs                                 5,0M       0  5,0M   0% /run/lock
tmpfs                                  63G       0   63G   0% /sys/fs/cgroup
tmpfs                                 4,0G       0  4,0G   0% /tmp
/dev/md127                            488M    140M  319M  31% /boot
/dev/mapper/vg_bulk-backups--pg      1007G    854G  103G  90% /srv/backups/pg
/dev/mapper/vg_bulk-backups--bacula    26T     13T   13T  50% /srv/backups/bacula
tmpfs                                  13G       0   13G   0% /run/user/0

performing an online resize to add 10TB now.

comment:5 Changed 11 months ago by anarcat

Owner: changed from tpa to anarcat
Status: reopenedaccepted

comment:6 Changed 11 months ago by anarcat

i actually grew the *wrong* filesystem (/srv/backups/bacula instead of .../pg), so i'll have to go at it again.

root@bungei:~# df -h 
Sys. de fichiers                    Taille Utilisé Dispo Uti% Monté sur
udev                                   63G       0   63G   0% /dev
tmpfs                                  13G    1,3G   12G  11% /run
/dev/mapper/croot                      20G    2,9G   16G  16% /
tmpfs                                  63G     72K   63G   1% /dev/shm
tmpfs                                 5,0M       0  5,0M   0% /run/lock
tmpfs                                  63G       0   63G   0% /sys/fs/cgroup
tmpfs                                 4,0G       0  4,0G   0% /tmp
/dev/md127                            488M    140M  319M  31% /boot
/dev/mapper/vg_bulk-backups--pg      1007G    854G  103G  90% /srv/backups/pg
/dev/mapper/vg_bulk-backups--bacula    30T     13T   17T  43% /srv/backups/bacula
tmpfs                                  13G       0   13G   0% /run/user/0
tmpfs                                  13G       0   13G   0% /run/user/1529

i don't think it's a problem that bacula is bigger, that said... shrinking it seems unnecessary. i'll just give a few more TBs to pg already (say 2TB).

comment:7 Changed 11 months ago by anarcat

Resolution: fixed
Status: acceptedclosed

all good now:

root@bungei:~# lvextend vg_bulk/backups-pg -L +2T
  Size of logical volume vg_bulk/backups-pg changed from 1,00 TiB (262144 extents) to 3,00 TiB (786432 extents).
  Logical volume vg_bulk/backups-pg successfully resized.
root@bungei:~# resize2fs /dev/mapper/vg_bulk-backups--pg
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/mapper/vg_bulk-backups--pg is mounted on /srv/backups/pg; on-line resizing required
old_desc_blocks = 128, new_desc_blocks = 384
The filesystem on /dev/mapper/vg_bulk-backups--pg is now 805306368 (4k) blocks long.

root@bungei:~# df -h
Sys. de fichiers                    Taille Utilisé Dispo Uti% Monté sur
udev                                   63G       0   63G   0% /dev
tmpfs                                  13G    1,3G   12G  11% /run
/dev/mapper/croot                      20G    2,6G   17G  14% /
tmpfs                                  63G     72K   63G   1% /dev/shm
tmpfs                                 5,0M       0  5,0M   0% /run/lock
tmpfs                                  63G       0   63G   0% /sys/fs/cgroup
tmpfs                                 4,0G       0  4,0G   0% /tmp
/dev/md127                            488M    140M  319M  31% /boot
/dev/mapper/vg_bulk-backups--pg       3,0T    854G  2,0T  30% /srv/backups/pg
/dev/mapper/vg_bulk-backups--bacula    30T     13T   17T  43% /srv/backups/bacula
tmpfs                                  13G       0   13G   0% /run/user/0
Note: See TracTickets for help on using tickets.