Can file system performance decrease if there is a very large number of files in a single directory (NTFS)?

Solution 1:

I answer my own question : Yes, it's definitely slower.

I wrote a C# Console Application that creates many empty files in a folder and then randomly access them. Here is the results :

10 files in a folder        : ~26000 operation/sec
1.000.000 files a in folder : ~6000 operation/sec

Here is source code :

List<string> files = new List<string>();

Console.WriteLine("creating files...");
for (int i = 0; i < 1000 * 1000; i++)
{
    string filename = @"C:\test\" + Guid.NewGuid().ToString();
    using (File.Create(filename));
    files.Add(filename);
}

Console.WriteLine("benchmark...");            
Random r = new Random();
Stopwatch sw = new Stopwatch();
sw.Start();

int count = 0;
while (sw.ElapsedMilliseconds < 5000)
{
    string filename = files[r.Next(files.Count)];
    string text = System.IO.File.ReadAllText(filename);
    count++;
}
Console.WriteLine("{0} operation/sec ", count / 5);

Solution 2:

If you read this, then you should get a pretty good understanding of how NTFS works indexing files and folders.

Locally it shouldn't be much of a hazel indexing files and folders, if you follow the guidelines in the link above, but it will need alot of maintenance with that many files.
On a network it will be another story. It will be slow, this is from my own expirience at work, where we have folders with thousand of folders and that takes some time to index over a network.

Another thing to probably increase with that many files is to disable short-naming:, which will stop windows from creating a second file directory entry which will follow the 8.3 convention (MS-DOS file-naming convention) and decrease the time for folders to enumerate, because it doesn't have to look up the short-names associated with their long-names when enumerating.

• Go to Run in the Start menu
• Type cmd and when you see the command-prompt, then right-click on it and select Run as administrator
• When in the Command prompt type fsutil behavior set disable8dot3 1 to disable short-naming
• Reboot

If you want to enable it again, type fsutil behavior set disable8dot3 0