git is very very slow when tracking large binary files

Do you garbage collect?

git gc

This makes a significant difference in speed, even for small repos.

Explanation

Git is really good at huge histories of small text files because it can store them and their changes efficiently. At the same time, git is very bad at binary files, and will naïvely store separate copies of the file (by default, at least). The repository gets huge, and then it gets slow, as you've observed.

This is a common problem among DVCS's, exacerbated by the fact that you download every version of every file ("the whole repository") every time you clone. The guys at Kiln are working on a plugin to treat these large files more like Subversion, which only downloads historical versions on-demand.

Solution

This command will list all files under the current directory of size >= 5MB.

find . -size +5000000c 2>/dev/null -exec ls -l {} \;

If you want to remove the files from the entire history of the repository, you can use this idea with git filter-branch to walk the history and get rid of all traces of large files. After doing this, all new clones of the repository will be leaner. If you want to lean-up a repository without cloning, you'll find directions on the man page (see "Checklist for Shrinking a Repository").

git filter-branch --index-filter \
    'find . -size +5000000c 2>/dev/null -exec git rm --cached --ignore-unmatch {} \;'

A word of warning: this will make your repository incompatible with other clones, because the trees and indices have different files checked in; you won't be able to push or pull from them anymore.

Here is a censored revision intended to be less negative and inflammatory:

Git has a well-known weakness when it comes to files that are not line-by-line text files. There is currently no solution, and no plans announced by the core git team to address this. There are workarounds if your project is small, say, 100 MB or so. There exist branches of the git project to address this scalability issue, but these branches are not mature at this time. Some other revision control systems do not have this specific issue. You should consider this issue as just one of many factors when deciding whether to select git as your revision control system.

There is nothing specific about binary files and the way git is handling them. When you add a file to a git repository, a header is added and the file is compressed with zlib and renamed after the SHA1 hash. This is exactly the same regardless of file type. There is nothing in zlib compression that makes it problematic for binary files.

But at some points (pushing, gc) Git start to look at the possibility to delta compress content. If git find files that are similar (filename etc) it is putting them in RAM and starting to compress them together. If you have 100 files and each of them arr say 50Mb it will try to put 5GB in the memory at the same time. To this you have to add some more to make things work. You computer may not have this amount of RAM and it starts to swap. The process takes time.

You can limit the depth of the delta compression so that the process doesn't use that much memory but the result is less efficient compression. (core.bigFileThreshold, delta attribute, pack.window, pack.depth, pack.windowMemory etc)

So there are lots of thinks you can do to make git work very well with large files.

One way of speeding things up is to use the --depth 1 flag. See the man page for details. I am not a great git guru but I believe this says do the equivalent of a p4 get or an svn get, that is it give you only the latest files only instead of "give me all of the revisions of all the files through out all time" which is what git clone does.