Deleting large amount of files in Windows is slow

I have a Windows XP box with an NTFS disk and deleting large amounts of files is extremely slow. If I select a folder that contains a large number of files in a tree of folders and delete (using shift-del to save the recycle bin) it takes time that seems to be directly proportional to the number of files within the folder before it even pops up the confirmation box. It then takes an even longer time to delete each file in the folder.

Is there a way to delete a folder in Windows and not having the time taken proportional to the number of files within it?

Solution 1:

Is there a way to delete a folder in Windows and not having the time taken proportional to the number of files within it?

I don't think so, but some methods are clearly much quicker than others.

The worst way is to send to Recycle Bin: you still need to delete them. Next worst is shift+delete with Windows Explorer: it wastes loads of time checking the contents before starting deleting anything.

Next best is to use rmdir /s/q foldername from the command line. del /f/s/q foldername is good too, but it leaves behind the directory structure.

The best I've found is a two line batch file with a first pass to delete files and outputs to nul to avoid the overhead of writing to screen for every singe file. A second pass then cleans up the remaining directory structure:

del /f/s/q foldername > nul
rmdir /s/q foldername

This is nearly three times faster than a single rmdir, based on time tests with a Windows XP encrypted disk, deleting ~30GB/1,000,000 files/15,000 folders: rmdir takes ~2.5 hours, del+rmdir takes ~53 minutes. More info here.

This is a regular task for me, so I usually move the stuff I need to delete to C:\stufftodelete and have those del+rmdir commands in a deletestuff.bat batch file. This is scheduled to run at night, but sometimes I need to run it during the day so the quicker the better.

Solution 2:

Is there a way to delete a folder in Windows and not having the time taken proportional to the number of files within it?

Well, yes, format the partition. I'm a bit surprised nobody suggested that in the previous 9 years.

It's pretty radical, but if you anticipate doing this frequently for a specific folder, it might be worthwhile creating a separate partition for it.

If that's too radical, the other answers are your only hope. There is a good explanation why on serverfault. It's for linux and XFS filesystems, but the same logic applies here. You can't improve much on build-in OS functions.

However, if you know the paths to all the files you wish to delete, then you can save on calls that list the directory contents and call remove directly, saving some overhead. Still proportional to the number of files though.

Personally, I like some from of progress report to ensure myself that the program didn't die. So I like to delete stuff via python. For example, if all the files are in one directory without sub-directories:

import tqdm
import sys
import os

location = sys.argv[1]
directory = os.fsencode(location)

with os.scandir(directory) as it:
    for dir_entry in tqdm.tqdm(it):
        try:
            os.remove(dir_entry.path)
        except OSError:
            pass  # was not a file

This deletes about 250 files/s on my 12 year old SEAGATE ST3250620NS. I'd assume it will be much faster on your drive.

However, at this point it's just micro-optimization, so won't do very much unless you have millions of files in one directory. (like me, lol, what have I done D:)

Solution 3:

Install gnutools for windows and run:

find YOURFOLDER -type d -maxdepth 3 | xargs rm -Rf