Opened 8 weeks ago

Last modified 7 days ago

#31051 new defect

bungei running out of space

Reported by: anarcat Owned by: tpa
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

we got this warning from nagios this weekend:

DISK WARNING - free space: /srv/backups/bacula 1886723 MB (9% inode=99%)

indeed, the main storage is full:

Sys. de fichiers                    Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/vg_bulk-backups--bacula    20T     18T  1,6T  93% /srv/backups/bacula

There's plenty of free space on the VG, however:

  VG      #PV #LV #SN Attr   VSize  VFree 
  vg_bulk   1   2   0 wz--n- 72,60t 51,60t

So this could quickly be resolved by just resizing that partition. But it's interesting to note that the vast majority of storage in use comes from the static content. I suspect there's a lot of duplication around those and we could save space in backups by being a little smarter about this:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help                                                                                                                                   
--- /srv/backups/bacula -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
    4,3 TiB [ 25,0% ##########] /arlgirdense.torproject.org                                                                                                                                    
    4,3 TiB [ 24,5% ######### ] /staticiforme.torproject.org
    2,1 TiB [ 11,9% ####      ] /archive-01.torproject.org
    1,1 TiB [  6,2% ##        ] /perdulce.torproject.org
  871,0 GiB [  4,9% #         ] /colchicifolium.torproject.org
  806,1 GiB [  4,5% #         ] /Catalog
  554,4 GiB [  3,1% #         ] /corsicum.torproject.org
  375,1 GiB [  2,1%           ] /gillii.torproject.org
  237,5 GiB [  1,3%           ] /eugeni.torproject.org
  209,0 GiB [  1,2%           ] /hetzner-nbg1-01.torproject.org

Above is the top-ten in disk usage. It's interesting to note that archive01 is only half of its partners. We should also note that arlgirdense is likely going to go away eventually, because archive-01 will replace it.

Child Tickets

Change History (2)

comment:1 Changed 7 weeks ago by weasel

manually, and behind bacula's back, removed one of arlgirdense's old fulls. We can reconsider growing things once we have migrated properly to the new archive host.

comment:2 Changed 7 days ago by anarcat

i don't quite understand how to delete old entries in the database and/or the storage. not "behind bacula's back" as that sounds ominous... but i think the rest of arlgirdense can be removed now that archive-01 is up and running. i just don't know how.

see also #30880 for that documentation.

Note: See TracTickets for help on using tickets.