Ubuntu "No space left on device" for /home, df shows 100% full, ds shows much, much less [duplicate]
Possible Duplicate:
Linux different size for df and du for root partition
On an Ubuntu 12.04 server, normal users can no longer create or add to files in /home, encountering a "No space left on device" error.
The /home directory has a capacity of 1.7 terabytes and as far as I can tell is nowhere near full in terms of actual data stored or inodes used.
df -h shows:
Filesystem Size Used Avail Use% Mounted on
/dev/md2 1.0T 18G 955G 2% /
udev 7.7G 4.0K 7.7G 1% /dev
tmpfs 3.1G 320K 3.1G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.7G 0 7.7G 0% /run/shm
cgroup 7.7G 0 7.7G 0% /sys/fs/cgroup
/dev/md3 1.7T 1.7T 0 100% /home
/dev/md1 496M 45M 426M 10% /boot
/home indeed looks rather full.
du -hs /home suggests otherwise:
1.4G /home
There appears no inode issue - df -i:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md2 67108864 75334 67033530 1% /
udev 2013497 527 2012970 1% /dev
tmpfs 2015816 440 2015376 1% /run
none 2015816 2 2015814 1% /run/lock
none 2015816 1 2015815 1% /run/shm
cgroup 2015816 9 2015807 1% /sys/fs/cgroup
/dev/md3 113909760 105981 113803779 1% /home
/dev/md1 131072 239 130833 1% /boot
I recently deleted a many gigabytes of application cache and log data from /home, however this was in the tens of gigabytes at best and nowhere near the capcity of /home.
Update 1:
du -hs --apparent-size /home
1.2G /home
du -hs /home
1.4G /home
What might be going on here?
Solution 1:
Check the files are still opened in write mode:
lsof | awk '/deleted/ && $4 ~ /[0-9]+w/ { print $0 }'
you will see something like this:
ossec-mai 1111 ossecm 4w REG 3,3 ... (deleted)
then find the process that still uses the file:
cd /proc/1111/fd
ls -l 4
You can free up the space without restarting the service by getting the size down to zero:
> /proc/1111/fd/4
Give it a try.