df reports an ext4 partition is full, but there is no data

Solution 1:

It could be that some process still has deleted files open. If this is the case then they will not appear in the du output but would still be counted in the df output.

One quick way to check for these is to list /proc as user root (hint sudo su should get you a root shell). Any open, but deleted files will have (deleted) at the end of the symbolic link target name.

ls -l /proc/*/fd/* | grep deleted | grep /home

should give you a list of any files open. Once you have that then an ls -lL of the specific file should give you the size of the file.

As an example (using /tmp on my system because there are no examples on /home here) I see a few files owned by user mysql.

richm@viking:/$ sudo su
root@viking:/# ls -l /proc/*/fd/* | grep deleted | grep /tmp
lrwx------ 1 root     root     64 Oct 13 06:30 /proc/1489/fd/11 -> /tmp/ibwmCqpg (deleted)
lrwx------ 1 root     root     64 Oct 13 06:30 /proc/1491/fd/12 -> /tmp/ib9MTMQi (deleted)
root@viking:/# ls -lL /proc/1489/fd/11
-rw------- 0 mysql2 mysql2 0 Aug 24 14:09 /proc/1489/fd/11
root@viking:/# ls -lL /proc/1491/fd/12
-rw------- 0 mysql mysql 1320 Oct 15 13:40 /proc/1491/fd/12

If you have any processes with large deleted files open then stoping the process should be enough to reclaim the disk space. Alternatively a reboot should do the same thing.

Solution 2:

Each file system only has a certain amount of inodes and blocks that can be stored on it. Even in case there is enough space, you can't go further.

Check your settings with

dumpe2fs /dev/sda5

(only the top 50 lines are important here).

If you have a lot of small files that are smaller than the blocksize, a lot of space is wasted.