Unexpected space consumption

I have a problem. I run out of disk space all the time. I don't know exactly what files consume all the space. It seems that there are bunch of little files, so it's hard to find with du command. Every time I find something to get rid of to gain some free space (mails, logs, old archives and rpms), disk is full again after some time. Space is stolen from system partition (/). Results of df command:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              4061540   3848404      3492 100% /
/dev/sda3              4061572   2441348   1410576  64% /home
tmpfs                   517636         0    517636   0% /dev/shm
/dev/sdb1             10317828   1894904   7898808  20% /var/lib/mysql

What would you recommend in this situation?

It would be great if you can also enumerate a list of cases when you encounter excessive or unexpected space consumption. It might help to resolve problem

OS: CentOS 5.

Solution 1:

Here's my standard "find what's eating the space" regime:

1. du -hx --max-depth=1 / -- look for what's eating the space. Examine the largest subdirectory (say du -hx --max-depth=1 /var) until you find some space hogs. Logs (in /var/log) are common culprits (which you should deal with using logrotate), as are the cruft that yum likes to keep around (repackaged RPMs). If you can identify any repeat offenders, work out a way to automate whatever maintenance work you're doing to clean up repeatedly.
2. lsof -n |grep deleted -- look for large files that have been deleted. Identify which process is holding them open, and kill it or tell it to recycle it's handles (if they're log files, kill -HUP <pid> will often do the trick) if you can.
3. My third line of attack is to resize partitions or allocate new partitions to various mounts, although in your case, since you're not using LVM, that's tricky. Consider taking a maintenance to turn /dev/sdb1 into a VG and allocate from there.
4. Upgrade. Disk space is so cheap, that if you spend an hour of your time hunting down what's causing space consumption, you just "spent" the time required to pay for a new hard drive, so buy new drives rather than spend too much time hunting down problems.

Solution 2:

One addition to womble's answer - if you find you're getting a lot of large log files, consider using logrotate (which is installed by default on most distros) to keep them down to size. If there is one particula log that's taking up most of the space, you should also check it for repeated errors and make sure that the service which produces the log isn't set to too high a verbosity level.