Is there any faster way to remove a directory than "rm -rf"?
I have a folder that has many files and "rm -rf" takes a lot of time to complete. Is there any faster way to remove a directory and it's contents (subdirs, etc)?
Solution 1:
You could try unlinking the inode for the directory but that would leave you with a whole load of orphan files that fsck will flip out about.
rm is as good as it gets.
A few people are mentioning edge cases where some things are faster than others. But let's make sure we're comparing the best versions of the same things.
If you want to delete a directory and everything in it, I'm suggesting you:
rm -rf path/to/directory
rm will internally list the files and directories it's going to delete. And that's all in compiled C. It's those two reasons it's fastest.
This is very pointedly not the same thing as rm -rf path/to/directory/* which will expand at shell level and pass a load of arguments into rm. Then rm has to parse those and then recurse from each. That's much slower.
Just as a "benchmark" that compares find path/to/directory -exec {} \; is nonsense. That runs rm once per file it finds. So slow. Find can xargs-style build commands arguments with -exec rm {} + but that's just as slow as expansion. You can call -delete which uses an internal unlink call to the kernel (like rm does) but that'll only work for files at first.
So to repeat, unless you throw the disk into liquid hot magma, rm is king.
On a related note, different filesystems delete things at different rates because of how they're structured. If you're doing this on a regular basis you might want to store these files in a partition formatted in XFS which tends to handle deletions pretty fast.
Or use a faster disk. If you have tons of RAM, using /dev/shm (a RAM disk) could be an idea.
Solution 2:
If you don't need the free space the quickest way is delay the deletion and do that in the background:
- mkdir .delete_me
- mv big-directory-that-i-want-gone .delete_me
Then have a crontab that does it in the background, at a quiet time, with a low I/O proiority:
3 3 * * * root ionice -c 3 nice find /path/to/.delete_me -maxdepth 1 ! -name \. -exec echo rm -rf "{}" +
Notes:
- check your output before removing the echo in the crontab!
- the .delete_me directory has to be in the same filesystem - in case it's not obvious to everyone.
Update: I found a neat trick to run multiple rm in parallel - this will help if you have a large disk array:
ionice -c 3 nice find target_directory -depth -maxdepth 3 | xargs -d \n -P 5 -n 5 rm -rf
-depth to do a depth-first traversal.
-maxdepth to limit the depth of the directory traversal so we don't end up listening individual files.
-d \n to handle spaces in filenames.
-P and -n handles the degree of parallelism (check manpage).
ref: http://blog.liw.fi/posts/rm-is-too-slow/#comment-3e028c69183a348ee748d904a7474019
Update 2 (2018): With ZFS shipped with Ubuntu 18.04 I use it for everything and I will create a new dataset for any big project. If you plan ahead and do this beforehand you can simply "zfs destroy" a filesystem when you are done. ;-)
I used the instructions from the zfsonlinux wiki to install Ubuntu to ZFS natively: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS