After deleting lots of large files, free space increases with a large delay

Yesterday, I have deleted 71 GB of files on my home/media server.
Free Space before: 117 GB
Free space after: 126 GB

So, instead of having 71 GB of additional free space, I only had 9 GB. I have double-checked that no files were open and I really have deleted 71 GB and free space really increased only 9 GB.

I also tried to sync, but no effect.

This is not the first time it happens. Indeed, I have seen this behaviour since years every now and then. First on ext3, now on ext4.
When this happens, I can reclaim the free space by unmounting and then remounting the filesystem. In these cases, unmounting takes up to 2 minutes instead of nearly no time.

Nowadays, I can not easily unmount and remount the filesystem, because it is permanently busy from my video recorder software, the owncloud server for my family, and a few more services I did not have so far. And I don't want to get up at 3 o'clock in the night just to unmount and remount.

No, the 'at' utility won't do because one of the services performs non resumable long running tasks and thus needs a manual state check to find a good moment when it can be shut down, i.e. a task was just finished.

But this morning, I noticed that the space has been freed overnight. Looks to me as if there was some kind of cleanup, and this may be the same thing that takes the additional time when unmounting.

So far, I only noticed this behaviour when deleting a large amount of data. In the other hand, I am not sure if it happens regularly and the difference is just too small to notice.

The filesystem was created with 0% reserved for root (mkfs -m 0). According to fsck -f (I'm doing this always between unmount and remount), the filesystem is not corrupted, and according to S.M.A.R.T. diagnostics extended test, the hardware is also OK.

[EDIT]

tune2fs 1.42 (29-Nov-2011)  
Filesystem volume name:   bigdata  
Last mounted on:          /bigdata  
Filesystem UUID:          6aebd17a-e064-41dc-9c68-c9a3acbe4f66  
Filesystem magic number:  0xEF53  
Filesystem revision #:    1 (dynamic)  
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize  
Filesystem flags:         signed_directory_hash   
Default mount options:    user_xattr acl  
Filesystem state:         clean  
Errors behavior:          Continue  
Filesystem OS type:       Linux  
Inode count:              121413632  
Block count:              485645568  
Reserved block count:     0  
Free blocks:              29081276  
Free inodes:              121382378  
First block:              0  
Block size:               4096  
Fragment size:            4096  
Reserved GDT blocks:      908  
Blocks per group:         32768  
Fragments per group:      32768  
Inodes per group:         8192  
Inode blocks per group:   512  
Flex block group size:    16  
Filesystem created:       Tue Dec 25 23:42:35 2012  
Last mount time:          Fri Jan  3 17:37:36 2014  
Last write time:          Fri Jan  3 17:37:36 2014  
Mount count:              37  
Maximum mount count:      -1  
Last checked:             Thu Apr 18 17:03:40 2013  
Check interval:           0 (<none>)  
Lifetime writes:          14 TB  
Reserved blocks uid:      0 (user root)  
Reserved blocks gid:      0 (group root)  
First inode:              11  
Inode size:           256  
Required extra isize:     28  
Desired extra isize:      28  
Journal inode:            8  
Default directory hash:   half_md4  
Directory Hash Seed:      be2b977e-5127-4843-9123-fe33b6d7b573  
Journal backup:           inode blocks  

[/EDIT]

So here are my 2 questions:

  1. What is happening here? Why is space freed at umount or with a delay instead of immediately at delete?
  2. Is there something I can do to trigger freeing the space right now without unmounting and remounting?

There are two factors that may be interacting.

  • Unlike Windows you can delete files that are open. If you delete a movie that is being streamed, it will be removed from the directory, but will still exist as a file until the streaming program closes it. Once the streaming software closes it, the space will be released. The fuser -m command can be used to find the process id of any processes with open files. Some programs may not immmediately close files when they are done with them.

  • These are both journaled file systems. Changes are written to a journal and then committed. It can take a while for the changes to commit. The operating system will often cache disk changes and only commit changes to disk periodically. Running the sync command should flush any pending changes to disk. Mounting the disk with the sync option will improve the speed to disk, but works the disk harder.