How do I recover free space on deleted files without restarting the referencing processes?

When big files are deleted on a server, the files might still be referenced by processes, so the file system doesn't have more free space.

I tried to use lsof, but it seems it didn't list the deleted files. fuser -c did better work, but the list of processes is just too long to check it out for each process, especially since each process is an Oracle process.

bash-3.2# fuser -c /var
/var:      105o   29999o   20444c    3528c   27258o    7715o    3864o    3862o    2494o   18205o   17450co   17445co   14912co   14824co   14818co   14816o   14814o    8532c    8530c    7633com    7118o    6958o    6790c    6784co    6734o    6693o    6689o    6684o    6675o    6635o    6594c    6548o    6547o    6546o    6545o    6544o    6543o    6542o    6541o    6540o    6537o    6535o    6456o    6128co    6113o     335o     245co     229o     161o       8o
bash-3.2# du -hs /proc
 139T   /proc

It happens sometimes that a file gets deleted by an application or a user, e.g. a logfile and that this file is still being referenced by a process that cannot be restarted.

Are there goods methods to reclaim disk space on deleted files without restarting the process that has a reference to this deleted file?

Solution 1:

find /proc/*/fd -ls 2> /dev/null | grep '(deleted)'

Find all opened file descriptors.

Grep deleted.

StdError to /dev/null

Output:

160448715    0 lrwx------   1 user      user            64 Nov 29 15:34 /proc/28680/fd/113 -> /tmp/vteT3FWPX\ (deleted)

Or you can use awk

find /proc/*/fd -ls 2> /dev/null | awk '/deleted/ {print $11}';

awk output(tested in bash Ubuntu 12.04):

/proc/28680/fd/113

Find and truncate all deleted files(tested in bash Ubuntu 12.04):

(DON'T DO THIS IF YOU DON'T KNOW WHAT YOU DO)

find /proc/*/fd -ls 2> /dev/null | awk '/deleted/ {print $11}' | xargs -p -n 1 truncate -s 0

-p prompt before execute truncate

Better way is manual truncate

Manual truncate:

: > /proc/28680/fd/113

or:

> /proc/28680/fd/113

or:

truncate -s 0 /proc/28680/fd/113

Enjoy ;)

Solution 2:

Here is a simple example with less:

Let's assume we have a file, my10MBfile:

$ dd if=/dev/zero of=/tmp/my10MBfile bs=1M count=10
10+0 enregistrements lus
10+0 enregistrements écrits
10485760 octets (10 MB) copiés, 0,0454491 s, 231 MB/s

$ ls -l /tmp/my10MBfile
-rw-r--r-- 1 max max 10485760 avril 23 22:49 /tmp/my10MBfile

$ df -m /tmp
/dev/disk/by-uuid/6835b2fd-971d-420c-ba18-3c729ec2e8a0     14637  9225       4662  67% /

Now I open that file with less (yes, it is a binary file... never mind)

$ less /tmp/my10MBfile &

$ lsof -p $(pidof less) | grep 10MBfile
less    29351  max    4r   REG    8,3 10485760 521464 /tmp/my10MBfile

Then I remove that file

$ rm /tmp/my10MBfile

$ lsof -p $(pidof less) | grep 10MBfile
less    29351  max    4r   REG    8,3 10485760 521464 /tmp/my10MBfile (deleted)

$ df -m /tmp
/dev/disk/by-uuid/6835b2fd-971d-420c-ba18-3c729ec2e8a0     14637  9225       4662  67% /

It is still there, but deleted. Look at the 4th column of the lsof output: File Descriptor number 4 open for Reading (4r)

Let's run GDB!

$ gdb -p $(pidof less)

GNU gdb (GDB) 7.4.1-debian
....
Attaching to process 29351
....

(gdb) p close(4)
$1 = 0
(gdb) q

That's it!

$ df -m /tmp
/dev/disk/by-uuid/6835b2fd-971d-420c-ba18-3c729ec2e8a0     14637  9215       4672  67% /

Our 10 MB are welcome back :)

$ ls /proc/29351/fd
0  1  2  3

$ ps 29351
29351 pts/0    S+     0:00 less /tmp/my10MBfile

The process is still running.

Solution 3:

This command will show all deleted files still open on a Solaris system:

find /proc/*/fd -type f -links 0

You can truncate the ones you are sure you want with this command:

:> /proc/p/fd/x

with p being the process id and x the file descriptor returned by the first command.

Don't worry if with some programs the size reported by ls is restored to the size before truncation after a while, the actual size used on disk will be much smaller as the file is now a sparse one.