df says disk is full, but it is not

On a virtualized server running Ubuntu 10.04, df reports the following:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             7.4G  7.0G     0 100% /
none                  498M  160K  498M   1% /dev
none                  500M     0  500M   0% /dev/shm
none                  500M   92K  500M   1% /var/run
none                  500M     0  500M   0% /var/lock
none                  500M     0  500M   0% /lib/init/rw
/dev/sda3             917G  305G  566G  36% /home

This is puzzling me for two reasons: 1.) df says that /dev/sda1, mounted at /, has a 7.4 gigabyte capacity, of which only 7.0 gigabytes are in use, yet it reports / being 100 percent full; and 2.) I can create files on / so it clearly does have space left.

Possibly relevant is that the directory /www is a symbolic link to /home/www, which is on a different partition (/dev/sda3, mounted at /home).

Can anyone offer suggestions on what might be going on here? The server appears to be working without issue, but I want to make sure there's not a problem with the partition table, file systems or something else which might result in implosion (or explosion) later.

It's possible that a process has opened a large file which has since been deleted. You'll have to kill that process to free up the space. You may be able to identify the process by using lsof. On Linux deleted yet open files are known to lsof and marked as (deleted) in lsof's output.

You can check this with sudo lsof +L1

5% (by default) of the filesystem is reserved for cases where the filesystem fills up to prevent serious problems. Your filesystem is full. Nothing catastrophic is happening because of the 5% buffer -- root is permitted to use that safety buffer and, in your setup, non-root users have no reason to write into that filesystem.

If you have daemons that run as a non-root user but that need to manage files in that filesystem, things will break. One common such daemon is named. Another is ntpd.

You may be out of inodes. Check inode usage with this command:

df -i

Most Linux filesystems (ext3, ext4) reserve 5% space for use only the root user.

You can see this with e.g

dumpe2fs /dev/sda1 | grep -i reserved

You can change the reserved amount using :

tune2fs -m 0 /dev/sda1

0 in this command stands for percent of disk size so maybe you would want to leave at least 1%.

In most cases the server will appear to continue working fine - assuming all processes are being run as 'root'.

In addition to already suggested causes, in some cases it could be also following:

• a different disk is mounted "over" the existing folder which is full of data
• du will calculate the size spent of mounted disk and df will show really spent
• solution: (when possible) unmount all non-root disks and check the size with du -md 1 again. Fix situation by moving hidden folder to some other place or mount on different place.