df shows all space taken, but du doesn't add up

I'm having a problem with Ubuntu 12.04 LTS. This is the second time I've encountered this problem in the last 3 weeks. The first time is described in this closed question on StackOverflow. The TL;DR version is I managed to use all the inodes on a 450G ext4 system compiling and building the Android stack less than 20 times.

I thought I was solving the problem by reformatting the disk as XFS so the inode storage could grow.

This morning after doing a build overnight, I'm down to less than 1GB of free space. There is nothing on this machine other than what is needed to build Android. I've done 5 total builds on the platform sources. The build creates a bunch of files, then I delete them shortly after with make clean. I'm not really less than 1GB but the tools are reporting it that way. I deleted a bunch of temporary files and had about 40GB "freed". A couple hours later, just idling, I was back to less than 1GB free.

Running Ubuntu from a flash drive returns the following for the partition...

$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda5      468521456 468255460    265996 100% /media/f71c77eb-b4cc-

$ df -i
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda5      1691760 624214 1067546   37% /media/f71c77eb-b4cc-

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       447G  447G  260M 100% /media/f71c77eb-b4cc-

Here's is the evidence that something is amiss. When I run du (with and without --apparent-size) or the visual Disk Usage Analyzer, I show that I'm really only using about 35GB or so. 98.7% of the used space is in /home/eric, but du doesn't add up to that. The discrepancy is between /home/eric and /home/eric/android

enter image description here

I've read through the relevant questions here and on SO and they usually suggest it's deleted files held by open processes. I rebooted into a flash drive to run this test, so it shouldn't be open files. FWIW, /tmp is empty.

Is there a tool I can install on a flash drive to recover the 'lost' space? I can try to free up memory on the system and run it there, but I'm assuming it's better to do this from a flash drive.

Should I be configuring this system in a different way? I would rather not do another wipe and install, but I need a sustainable Android build system.

FOLLOWUP - I had to nuke the installation last week and re-installed 12.04 in order to get work done. As I go through Android builds again this week, I will keep a watchful eye over the disk usage and give information here as I learn more.

Thanks


On Oracle Linux it happens when you have (a lot of/large) files which are deleted but still open by a running process. Then stop of the processes or reboot of the machine helps.


I recently ran into this, and in my case a fsck needed to be run.

I did touch /forcefsck && reboot and after a few minutes, the server was back online and suddenly my missing 6 GB freed up.


Before you go to far.....take the system down to single user mode and do a FULL fsck (I really mean full fsck -f /dev/sda5) of the filesystem and see what it shows. You might find the space as portions of problem areas on your disk or mismatch between what is allocated and what is present on the disk.