How best to compare huge directory trees

How can I compare directory trees of huge size?

I am thinking a free tool to make a snapshot of the filesystem structure (listing of files and directories, their size & timestamps) would be ideal so I could compare the snapshot to another one made later.

Treecomp would be great for that but with a huge tree (I mean really huge!) it crashes because it tries to keep it in memory (4GB of memory are not enough)...

I worked around the problem by splitting the snapshots into pieces and compare these pieces. But that's tedious, and the problem can for sure be solved better.

Is there another free (best also open source) tool that I can try out? Or is there another way to do this that I am overlooking?

OS can be Linux or Windows.

Solution 1:

I'll try to expand a bit on how to do it with Total Commander (I hope I understood what you want to do).

• install DiskDir packer plugin (I put a direct link to plugin, if you prefer you can go to plugins page and look for DiskDir plugin
• after the plugin is installed "pack" the directory you want to track changes of with Alt+F5 and select "lst" from the drop down list in Packer part of the dialog box; this will create a "package" that you can enter by pressing enter, like you would enter a directory and it will show complete contents of the directory
• when comparing results go to the original directory on the left pane and enter desired snapshot on right pane
• use "Synchronize Dirs" function, located in Commands menu
• in Synchronize directories window uncheck compare by contents, check Subdirs and Ignore date (or not if changed date is important) and run comparison
• window will show you files that are equal (in this case not by contents, only by size), files that are different and files missing on left/right side

Since the snapshot is a plain text file and you are not comparing by contents it should be fast but I never used it for a really huge directory.

This is useful if you are not making backups but only wish to make a snapshot of what contents of the directory was at some point. If you do make backups you can use same tool (Synchronize dirs) to also compare by contents.

There is also an extended version of DiskDir plugin, download link is in the first post. This version enables you to have packages (like zip, 7z...) show as directories in the snapshot. This would of course increase time to make a snapshot.

Solution 2:

you can just use in the terminal

du -a

This will return all the files in all sub folder including there sizes, then just compare the files

To save the data to a text file

du -a > dump.txt

Then you can just use something like diff to compare the files

This is for linux :D