Out of disk space, what's the source?

Solution 1:

I think you have something writing to a file that has been deleted from the drive but not yet closed by the application/server, so the space remains allocated on disk but can't be seen by du since the file was removed from the filesystem. The lsof program lists processes that have files open. If you had more filesystems mounted and the number didn't fluctuate so much, then I would have suggested that you had a filesystem mounted on top of a directory that wasn't empty (though you could try umount /var/lib/ureadahead/debugfs and make sure that directory is empty and there isn't a bunch of junk written to the directory hiding under that mountpoint).

If this is the case, then you should easily find these with sudo lsof | grep deleted. lsof includes (deleted) in the last column if a file has been deleted while a process still has it open. The first column is the name of the command, the second column is the PID. You can get a more detailed look at the command using ps for instance ps auxww | grep PID, or ps auxwwf | less -S to see the process list in "forest" mode so you can see what process that PID came from. Once you've tracked down the process(es) that are holding open giant files, you can stop it to free the drive space, then figure out how to fix it to close the file properly. The usual cause of this is a logrotate script that renames/deletes log files but doesn't notify the application that it has done so (either through an appropriate signal with kill or by restarting the application), so the application continues to hold the old log file open.

Solution 2:

Run

du -h --max-depth=1 /

And it should give a clearer picture. If its coming and going it sounds like temp files being created and then not deleted once done with, until whichever process is causing it crashes. What OS is this server runnng and is it running anything in particular?

Solution 3:

It looks like the problem is /var/lib/ureadahead/debugfs. It appears this is a known problem, here is a link to ubuntuforums with more information http://ubuntuguide.net/howto-fix-ureadahead-problem-after-upgrading-to-ubuntu-10-04. The tl;dr seems to be update and upgrade, sudo mv /etc/init.d/ureadahead.conf /etc/init.d/ureadahead.conf.disabled, then reboot. Of course, I am assuming you are running 10.04.