What has changed Finder list view in Lion to make "Calculate all sizes" exponentially faster?

Since way before Mac OS X came on the scene, we have been able to ask the Finder to Calculate all sizes in order to determine the sum total of readable file size space contained in each folder for the Finder window in question.

I have tested List view sizing of folders on several Macs where it doesn't matter if an SSD is present or not, but Lion is so fast at calculating sizes I am curious if there is some new caching data structure or if Finder is using the metadata information from Spotlight or a similar database to speed up this calculation immensely.

I have not observed Lion to be faster at calculating folder (and package/bundle) sizes during the first time it calculates sizes in a folder. However, subsequent calculations in the same folder do seem to be much faster.

Part of the perceived quickness may be that Finder will immediately show the previously calculated sizes in gray text while it recalculates folder sizes, rather than showing "--" until it's been calculated. After a folder's size has been recalculated, the number will update (if the size has changed) and turn black.

Because the Finder is observably caching previously calculated folder sizes, it's possible that it's only recalculating sizes for folders that have changed since the last time it was calculated.

Before Lion, the File Size column in Finder.app would display the size each file requires on the hard disk, not the exact file size. For example, 1 byte files were displayed as 4 KB because they do in fact take up 4 KB of space on a HFS-formatted system. There was no easy way to see the actual file size of 1 byte, other than opening File › Get Info (or using a different app, like Terminal.app and then using ls -lsa, or a Finder.app replacement like TotalFinder.app).

(Back in the day, I reported this as bug 8926275 on bugreport.apple.com.)

As of Lion, this behavior has been corrected, and the File Size column will now show the exact file size for each file rather than the size it allocates on the hard disk (which depends on the file system anyway).

Since these sizes are the same numbers you’d get from the ls binary in Terminal, they’re much more efficient to calculate.

Starting with OS X Lion, Apple has added a SQLite database that the OS uses for file tracking in system features such as Spotlight. Querying from a SQLite database rather than inspecting the file system every time is more than likely the cause for the performance improvement. John Siracusa's OS X Lion review explains in depth the changes to the file system in Lion. Particularly, here you will find an explanation on the new SQLite database.

Hope this helps.

I wouldn't be surprised if they were using Spotlight metadata to cache file sizes. If you're already using FSEvents to track all changes in the file system and (potentially) Time Machine to back all those changes up, the added cost of calculating and storing aggregate file sizes is negligible.