Opened 4 months ago

Closed 2 weeks ago

#31051 closed defect (fixed)

bungei running out of space

Reported by: anarcat Owned by: tpa
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:


we got this warning from nagios this weekend:

DISK WARNING - free space: /srv/backups/bacula 1886723 MB (9% inode=99%)

indeed, the main storage is full:

Sys. de fichiers                    Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/vg_bulk-backups--bacula    20T     18T  1,6T  93% /srv/backups/bacula

There's plenty of free space on the VG, however:

  VG      #PV #LV #SN Attr   VSize  VFree 
  vg_bulk   1   2   0 wz--n- 72,60t 51,60t

So this could quickly be resolved by just resizing that partition. But it's interesting to note that the vast majority of storage in use comes from the static content. I suspect there's a lot of duplication around those and we could save space in backups by being a little smarter about this:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help                                                                                                                                   
--- /srv/backups/bacula -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
    4,3 TiB [ 25,0% ##########] /                                                                                                                                    
    4,3 TiB [ 24,5% ######### ] /
    2,1 TiB [ 11,9% ####      ] /
    1,1 TiB [  6,2% ##        ] /
  871,0 GiB [  4,9% #         ] /
  806,1 GiB [  4,5% #         ] /Catalog
  554,4 GiB [  3,1% #         ] /
  375,1 GiB [  2,1%           ] /
  237,5 GiB [  1,3%           ] /
  209,0 GiB [  1,2%           ] /

Above is the top-ten in disk usage. It's interesting to note that archive01 is only half of its partners. We should also note that arlgirdense is likely going to go away eventually, because archive-01 will replace it.

Child Tickets

Change History (3)

comment:1 Changed 4 months ago by weasel

manually, and behind bacula's back, removed one of arlgirdense's old fulls. We can reconsider growing things once we have migrated properly to the new archive host.

comment:2 Changed 2 months ago by anarcat

i don't quite understand how to delete old entries in the database and/or the storage. not "behind bacula's back" as that sounds ominous... but i think the rest of arlgirdense can be removed now that archive-01 is up and running. i just don't know how.

see also #30880 for that documentation.

comment:3 Changed 2 weeks ago by anarcat

Resolution: fixed
Status: newclosed

i think this is done. the files can be removed directly from the filesystem, and stuff in the database gets pruned semi-automatically.

Note: See TracTickets for help on using tickets.