haykranen.nl

I got a typo in my logrotate config which pretty much killed my VPS.

My Apache server runs a lot of virtual hosts, i didn’t want to type them all out for logrotate, so i created something like this to handle it:

/var/www/*/logs/*.lo* {
        daily
        missingok
        rotate 7
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
                        /etc/init.d/apache2 reload > /dev/null
                fi
        endscript
}

Note the ending * at the first line. I didn’t know what got into me, but this caused logrotate to rotate the rotated logfiles, hence 100% CPU and hence, a broken system.

I fixed it (by replacing the ‘*’ with a ‘g’), but logrotate still didn’t work. CPU stayed at 100%, logs didn’t rotate. So i ran a strace -f logrotate and got lots of these:

stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2917, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2917, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2917, ...}) = 0

A little bit of Googling turned up this blogpost: apparently lograte‘s own log got corrupted. A simple rm /var/lib/logrotate/status did the trick and everything worked perfectly again.