I've tried to deal with this for days now with no luck. In this forum and also in other ones I can see a lot of discussion about this but none of the solutions have worked for me.

My current situation is that I've got some GBs "missing" that I can't free or find anywhere. I guess this is some kind of problem like memory leaks that only get fixed when rebooting, but I just would like to ask if someone has a better idea because there are currently a few customers on that server.

Here are some of the commands I've seen in other threads and my particular outputs:

df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              19G   16G  1.8G  90% /
tmpfs                 384M     0  384M   0% /lib/init/rw
udev                   10M  1.2M  8.9M  12% /dev
tmpfs                 384M     0  384M   0% /dev/shm

du -Pshx /* 2>/dev/null

4.8M    /bin
27M /boot
0   /cdrom
1.2M    /dev
8.8M    /emul
504M    /etc
4.0K    /grubconf
88K /home
60K /images
0   /initrd.img
147M    /lib
0   /lib32
0   /lib64
16K /lost+found
12K /media
4.0K    /mnt
4.4M    /opt
777M    /proc
57M /root
4.7M    /sbin
4.0K    /selinux
4.0K    /srv
0   /sys
4.0K    /tmp
332M    /usr
471M    /var
0   /vmlinuz

lsof +L1

(no output)

lsof | grep -i delete

(no output)

find / -size +50000 -exec ls -lah {} +;

-r-------- 1 root    root 777M 2015-07-21 06:19 /proc/kcore
-rw------- 1 root    root  64M 2015-07-09 03:31 /sys/devices/pci0000:00/0000:00:06.0/0000:01:04.0/resource0
-rw-r----- 1 root    adm   49M 2015-07-21 06:19 /var/log/auth.log
-rw-r----- 1 root    adm  174M 2015-07-19 06:25 /var/log/auth.log.1
-rw-rw---- 1 control mail 100M 2015-07-21 06:00 /var/mail/control

If you make the maths from du, for example, you get a few GBs, but nothing close to 16GB.

/ is ext3.

OS is Debian 6.0

is there anything am I missing here?

99.9% of the time the reason this happens is because one or more files have been deleted, but there is a process which is still writing to the old file handle.

When a program wants to perform I/O operations on a file, it asks the kernel "hey kernel, I would like to access the file /bla.txt and I want to be able to read and write to it". The kernel then returns a "file handle" which is a reference to the file that can be used to perform read and write operations.

As long as the program keeps the file handle open, it can keep writing to it, even if the file is subsequently deleted.

This typically becomes evident when you do something to the file, and most of the time it'll be something like rotating log files. What happens then is that df reports actual usage on disk, which can include space taken up by open files that are still being written to. du on the other hand goes through all the known filenames and takes their space into account. This can be less because a deleted file no longer has a filename, thus du does not take it into account.

To see if this is your problem, run this command: lsof +L1, and if you see any filenames that say '(deleted)' then you will need to stop the process named in the "COMMAND" column of the output, and then figure out why it holds deleted files open. Once you figure that out you can take steps to stop it from happening.