df shows all space taken, but du doesn't add up
I'm having a problem with Ubuntu 12.04 LTS. This is the second time I've encountered this problem in the last 3 weeks. The first time is described in this closed question on StackOverflow. The TL;DR version is I managed to use all the inodes on a 450G ext4 system compiling and building the Android stack less than 20 times.
I thought I was solving the problem by reformatting the disk as XFS so the inode storage could grow.
This morning after doing a build overnight, I'm down to less than 1GB of free space. There is nothing on this machine other than what is needed to build Android. I've done 5 total builds on the platform sources. The build creates a bunch of files, then I delete them shortly after with make clean
. I'm not really less than 1GB but the tools are reporting it that way. I deleted a bunch of temporary files and had about 40GB "freed". A couple hours later, just idling, I was back to less than 1GB free.
Running Ubuntu from a flash drive returns the following for the partition...
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda5 468521456 468255460 265996 100% /media/f71c77eb-b4cc-
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda5 1691760 624214 1067546 37% /media/f71c77eb-b4cc-
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 447G 447G 260M 100% /media/f71c77eb-b4cc-
Here's is the evidence that something is amiss. When I run du
(with and without --apparent-size) or the visual Disk Usage Analyzer, I show that I'm really only using about
35GB or so. 98.7% of the used space is in /home/eric
, but du
doesn't add up to that. The discrepancy is between /home/eric
and /home/eric/android
I've read through the relevant questions here and on SO and they usually suggest it's deleted files held by open processes. I rebooted into a flash drive to run this test, so it shouldn't be open files. FWIW, /tmp is empty.
Is there a tool I can install on a flash drive to recover the 'lost' space? I can try to free up memory on the system and run it there, but I'm assuming it's better to do this from a flash drive.
Should I be configuring this system in a different way? I would rather not do another wipe and install, but I need a sustainable Android build system.
FOLLOWUP - I had to nuke the installation last week and re-installed 12.04 in order to get work done. As I go through Android builds again this week, I will keep a watchful eye over the disk usage and give information here as I learn more.
Thanks
On Oracle Linux it happens when you have (a lot of/large) files which are deleted but still open by a running process. Then stop of the processes or reboot of the machine helps.
I recently ran into this, and in my case a fsck
needed to be run.
I did touch /forcefsck && reboot
and after a few minutes, the server was back online and suddenly my missing 6 GB freed up.
Before you go to far.....take the system down to single user mode and do a FULL fsck (I really mean full fsck -f /dev/sda5
) of the filesystem and see what it shows. You might find the space as portions of problem areas on your disk or mismatch between what is allocated and what is present on the disk.