Is "All My Files" slow because of my git and svn repos?
Solution 1:
It would seem that have too many files in Spotlight does indeed slow down "All My Files" in finder.
It doesn't really matter that the files are in code repos, although repos do tend to have lots of files.
Upon removing my ~/Sites folder from spotlight which contains only 15GB of data, but more than half of the 500,000 files Spotlight was indexing "All my Files" came up nearly instantly again.