Opened 4 months ago

Closed 4 months ago

#29841 closed defect (fixed)

ipsec VPN generates gigantic logs

Reported by: anarcat Owned by: weasel
Priority: Low Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Serious yak shaving night...

To try to silence this seemingly innocuous warning:

/etc/cron.daily/logrotate:
error: Compressing program wrote following message to stderr when compressing log /var/log/syslog.1:
gzip: stdin: file size changed while zipping

... I have looked at the logrotate configuration deployed through Puppet, and it seems slightly out of date compared to the one available in stretch. This is the configuration left over from the stretch upgrade on eugeni, for example:

/var/log/syslog
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		invoke-rc.d syslog-ng reload > /dev/null
	endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
/var/log/error
{
	rotate 4
	weekly
	missingok
	notifempty
	compress
	delaycompress
	sharedscripts
	postrotate
		invoke-rc.d syslog-ng reload > /dev/null
	endscript
}

Out of those, we're not doing the syslog-ng reload, the delaycompress, notifempty and each logfile is in a separate block which makes it harder to read. So I looked at doing the postrotate action, but then I realized it was happening on the syslog logfile which *is* correctly reloaded. so then i figured the delaycompress might be the bit missing.

but before enabling that blindly, I figured I would check if this would blow up the disk space on a server. how to do that you ask? well with our shiny new Cumin tool of course:

anarcat@curie:~(master)$ cumin -p 0 '*' 'for log in /var/log/*.log ; do if [ `du -b "$log" | cut -f1` -gt 1000000000 ] ; then echo "logfile $log larger than 1GB"; exit 1 ; fi; done'
74 hosts will be targeted:
alberti.torproject.org,arlgirdense.torproject.org,bracteata.torproject.org,brulloi.torproject.org,build-arm-[01-03].torproject.org,build-x86-[05-09].torproject.org,bungei.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,chamaemoly.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crispum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,dictyotum.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,gayi.torproject.org,getulum.torproject.org,gitlab-01.torproject.org,henryi.torproject.org,hetzner-hel1-[01-03].torproject.org,hetzner-nbg1-01.torproject.org,hyalinum.torproject.org,iranicum.torproject.org,kvm[4-5].torproject.org,listera.torproject.org,macrum.torproject.org,majus.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nova.torproject.org,nutans.torproject.org,omeiense.torproject.org,oo-hetzner-03.torproject.org,opacum.torproject.org,orestis.torproject.org,oschaninii.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,savii.torproject.org,saxatile.torproject.org,scw-arm-ams-01.torproject.org,scw-arm-par-01.torproject.org,staticiforme.torproject.org,subnotabile.torproject.org,textile.torproject.org,togashii.torproject.org,troodi.torproject.org,unifolium.torproject.org,vineale.torproject.org,web-cymru-01.torproject.org,web-hetzner-01.torproject.org
Confirm to continue [y/n]? y
                                                                                                                                                                            |██████████████▌                                                                                                         |  12% (9/74) [00:47<08:25,  7.78s/hosts]
===== NODE GROUP =====                                                                                                                                                                                                                                                                                                                             |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
(3) build-arm-[01-03].torproject.org                                                                                                                                                                                                                                                                                                               |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' -----                                                                                                                                                                                                                                                                                        |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
Connection timed out during banner exchange                                                                                                                                                                                                                                                                                                        |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
===== NODE GROUP =====                                                                                                                                                                                                                                                                                                                             |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
(5) hetzner-hel1-01.torproject.org,kvm4.torproject.org,macrum.torproject.org,textile.torproject.org,unifolium.torproject.org                                                                                                                                                                                                                       |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' -----                                                                                                                                                                                                                                                                                        |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
logfile /var/log/daemon.log larger than 1GB                                                                                                                                                                                                                                                                                                        |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
===== NODE GROUP =====                                                                                                                                                                                                                                                                                                                             |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
(1) hyalinum.torproject.org                                                                                                                                                                                                                                                                                                                        |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' -----                                                                                                                                                                                                                                                                                        |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
ssh: Could not resolve hostname hyalinum.torproject.org: No address associated with hostname                                                                                                                                                                                                                                                       |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
================                                                                                                                                                       PASS |████████████████████████████████████████████████████████████████████████████████████████████████████████▌              |  88% (65/74) [00:52<00:07,  1.23hosts/s]
FAIL |██████████████▌                                                                                                         |  12% (9/74) [00:52<08:25,  7.78s/hosts]
12.2% (9/74) of nodes failed to execute command 'for log in /var/...xit 1 ; fi; done': build-arm-[01-03].torproject.org,hetzner-hel1-01.torproject.org,hyalinum.torproject.org,kvm4.torproject.org,macrum.torproject.org,textile.torproject.org,unifolium.torproject.org
87.8% (65/74) success ratio (>= 0.0% threshold) for command: 'for log in /var/...xit 1 ; fi; done'.: alberti.torproject.org,arlgirdense.torproject.org,bracteata.torproject.org,brulloi.torproject.org,build-x86-[05-09].torproject.org,bungei.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,chamaemoly.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crispum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,dictyotum.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,gayi.torproject.org,getulum.torproject.org,gitlab-01.torproject.org,henryi.torproject.org,hetzner-hel1-[02-03].torproject.org,hetzner-nbg1-01.torproject.org,iranicum.torproject.org,kvm5.torproject.org,listera.torproject.org,majus.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nova.torproject.org,nutans.torproject.org,omeiense.torproject.org,oo-hetzner-03.torproject.org,opacum.torproject.org,orestis.torproject.org,oschaninii.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,savii.torproject.org,saxatile.torproject.org,scw-arm-ams-01.torproject.org,scw-arm-par-01.torproject.org,staticiforme.torproject.org,subnotabile.torproject.org,togashii.torproject.org,troodi.torproject.org,vineale.torproject.org,web-cymru-01.torproject.org,web-hetzner-01.torproject.org
87.8% (65/74) success ratio (>= 0.0% threshold) of nodes successfully executed all commands.: alberti.torproject.org,arlgirdense.torproject.org,bracteata.torproject.org,brulloi.torproject.org,build-x86-[05-09].torproject.org,bungei.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,chamaemoly.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crispum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,dictyotum.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,gayi.torproject.org,getulum.torproject.org,gitlab-01.torproject.org,henryi.torproject.org,hetzner-hel1-[02-03].torproject.org,hetzner-nbg1-01.torproject.org,iranicum.torproject.org,kvm5.torproject.org,listera.torproject.org,majus.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nova.torproject.org,nutans.torproject.org,omeiense.torproject.org,oo-hetzner-03.torproject.org,opacum.torproject.org,orestis.torproject.org,oschaninii.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,savii.torproject.org,saxatile.torproject.org,scw-arm-ams-01.torproject.org,scw-arm-par-01.torproject.org,staticiforme.torproject.org,subnotabile.torproject.org,togashii.torproject.org,troodi.torproject.org,vineale.torproject.org,web-cymru-01.torproject.org,web-hetzner-01.torproject.org

This might not be very easy to read, but the important bit is this:

(5) hetzner-hel1-01.torproject.org,kvm4.torproject.org,macrum.torproject.org,textile.torproject.org,unifolium.torproject.org
----- OUTPUT of 'for log in /var/...xit 1 ; fi; done' -----
|logfile /var/log/daemon.log larger than 1GB

So I looked at the first one of those (hetzner-hel1-01) and lo and behold, the daemon.log is gigantic:

1,4G	/var/log/daemon.log

I looked into the file briefly and it looks like a *lot* of information from ipsec. But before I start shaving another yak, I figured I would just file this as a ticket to document how far I went and let this one rest for a while.

(I did end up setting delaycompress after doing more investigations in Prometheus about free disk space, but that's documented in the tor-puppet commit 44f86c7d and previous.)

Child Tickets

Change History (3)

comment:1 Changed 4 months ago by anarcat

weasel looked into this a while back and found that ticket:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=640067

we need to tweak the log levels, there should be a fairly simple way for that, we just didn't have time to get around to doing so.

comment:2 Changed 4 months ago by anarcat

Owner: changed from tpa to weasel
Status: newassigned

i believe weasel fixed this this morning, in puppet commit af31efa4

comment:3 Changed 4 months ago by anarcat

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.