||[Tuesday 27th January 2009 at 9:35 pm]
Today's discovery was a full /var/log/wtmp file.
Those of you who know what that is are probably staring at this going "WTF?". For those that don't know (i.e. non-die-hard-linux-geeks), this file tracks all logins and logouts. Every time someone (or something) logs in or out, an entry gets added to this. And, following the Unix philosophy, no program ever expects that this file might become full. Because, of course, such a thing could never possibly happen. Ever.
Ha. Ha. Ha.
It turned out that the ftpd variant we were using wrote to this file on login/out (oh yes - on Linux it's the responsibility of each individual program to log account usage, not the operating system), and this particular system had a 2GB file size limit. Why, I don't know - even FAT could handle files larger than that. Anyway, given that this is a load box it was quite easy to hit the 2GB limit, and when this happened rather than return an error code Linux's default behaviour is apparently to send a SIGXFSZ signal. And the default behaviour for *that* is to terminate the process.