ubuntu server slowly filling up

(This is a Linux focused answer. Other UNIX variants may be different.)

There are two pieces of information relevant to your problem: (1) which files are filling up your file system, and (2) which processes are writing to those files.

Notes

Below, when I put the $ character in commands, that's probably a place holder where you need to substitute a real value. Hopefully, it's obvious where to do that and where not to.

Which Files?

Be aware that there are really two resources in most file system types that can be used up by individual files: meta-data (e.g. inodes), and real data. You can see the number of inodes (search Google for a definition, but they're "pointers" to the structures that make up your files) with a command like:

df -i

... and as you already know, something like this will show the space being used by real data:

df -h

Also, be aware that file system space can be taken up by files that don't exist on disk. These files are still in the open state by some process, but have been removed (we'll cover that below).

Once you've identified the full file system(s), then you need to start looking for lots of little files, a few big files, or both. Running out of meta-data resources is usually caused by having lots of little files, whereas running out of real data resources is usually caused by a few big files. I like to use this command to help find the big files:

sudo find $file_system -mount -ls | awk '{print $7, $11}' | sort -rn > $output

... and this command to help find directories with lots of little files (Update:: added null termination to improve file name handling):

sudo find . -mount -print0 | xargs -0n 1 dirname | sort | uniq -c | sort -rn > $output

... be aware that these commands may take a while to run, and do a lot of I/O, depending. Once run, you can read through $output to find the offending files or directories. The name and location of each may give you a hint about where the data's coming from, but requires some Linux experience.

Once you've identified the offenders, you can rm $file to get rid of the problem.

Which Processes?

The most straight forward way to find the processes potentially filling up your file system is to run a command like:

fuser -c $file_system 2>/dev/null

... which will tell you the PID of processes that have open file descriptors (files and network sockets) for the given file system (the 2>/dev/null part gets rid of some information you don't need). You might be able to deduce just from these PIDs which process is filling up your file system. Search for the processes with:

ps -ef | grep $pid

You can also try running this command which will give you even more detail (and help identify open files with no corresponding file name on the disk -- I mentioned this above):

sudo lsof $file_system | grep $directory_filling_up

... and if you've identified a suspect PID from the fuser command, you can do this:

sudo lsof -p $pid

The problem with fuser and lsof is that they only give you a snapshot of the system at the time your run the command. If the offending process doesn't happen to be writing when you run them, you're out of luck. You can counter this by repeatedly running them over time and saving the output. This will require reading through the output to find patterns, or writing a program to do it for you. An alternative is to use a tool like SystemTap. SystemTap allows you to trap all kinds of useful information, and is "programmable." It even comes with some sample source files that would allow you see what processes are writing to what files over some span of time. It would be perfect, but it's an advanced tool and requires a lot of Linux knowledge.

Once you've identified the offending process(es), you can kill (and maybe restart them). If the process is associated with the operating system or some well packaged software, there will probably be a mechanism for restarting them, but it will depend on your Linux distro (I think Ubuntu will allow you to run something like /etc/init.d/$init_script restart, but you'll have to check your distro's documentation). Otherwise, you can kill it with kill $pid or kill -9 $pid if it's not behaving. Be careful to note how the process was running (e.g. what were it's arguments shown in ps -ef) in case you need to restart it (you may need to refer to that software's documentation).

Use du to track down the directory that contains the file(s) that are filling the disk.

cd /
du -h --max-depth 1

will show you which directory in / uses the most space. Traverse the filesystem running the du command to find the culprit.

e.g.

cd /
du -h --max-depth 1

shows /usr is usong 2.3G of the 3.5G used on the system.

cd /usr
du -h --max-depth 1

shows /usr/lib uses 1.1G of the 2.3 in /usr ...

This may also be caused by an open file that has been deleted.

You can use lsof to find files that are open but unlinked (deleted)

lsof +L1

should do the trick. As the man page states:

A specification of the form +L1 will select open files that have been unlinked. A specification of the form +L1 <file_system> will select unlinked open files on the specified file system.