"No space left on device", df shows discrepancy

A few hours ago my root partition filled up, I moved files away from it and df reports:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             183G  174G     0 100% /

So there should be 9GB free, but avail reports 0 and Use is still at 100%.

I tested as root, e.g.

# echo test >a ; cat a
test

it works as expected; however as a normal user, I still get the error:

$ echo test >a ; cat a
bash: echo: write error: No space left on device

The root home directory where I conducted the positive test and my home directory are on the same partition.The fstab entry is:

/dev/hda1 / ext3 noatime,defaults,errors=remount-ro 0 1

Solution 1:

Most filing systems reserve a certain percentage for root, so you can still log in as root and solve out of diskspace issues. Usually this is 5%. 9GB is roughly 5% of 183GB, so this would make sense. You can see how much is reserved using tune2fs:

# tune2fs -l /dev/sda1 | grep -i reserved
Reserved block count:     936488
Reserved GDT blocks:      1019
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)

You can modify it using

# tune2fs -m 3 /dev/sda1
tune2fs 1.41.9 (22-Aug-2009)
Setting reserved blocks percentage to 3% (561893 blocks)

On modern large drives 5% is probably a little excessive, and you probably want to set it lower. You don't want to set it to zero.

Solution 2:

im with bob, try df -i, if you have a bunch of ill output crontabs for instance, your /var/spool/clientmqueue/ directory can get filled up

Solution 3:

Take a look at INODES too. In a "vanilla" installation, if you have to many small files they can consume inodes, but not space. You'll see that you have available space, but since your inodes are full, you'll not be able to use this space.