Find duplicate files and folders and move these to a different folder [duplicate]

Solution 1:

fdupes

No GUI but fdupes / sudo apt-get install fdupes is very fast and reliable. It uses sizes and modification dates for a preliminary analysis, then compares md5 hashes of the files and then does a bit compare if necessary. It's also dead easy to use. I strongly recommend it.

Typical usage:

fdupes -d -r /path/to/directory/

-r for walking subdirectories as opposed to walking just the contents of the specified dir.

-d to prompt the user about which file to delete (without this fdupes just compiles a list of duplicated)

-N deletes without prompt

-H normally, when two or more files point to the same disk area they are treated as non-duplicates; this option will change this behavior

-L hardlink duplicate files to the first file in each set of duplicates without prompting the user (this option was rolled back in some versions as it was found to be buggy and unsafe in rare cases. It might be reintroduced in future versions).

Edit: The hardlink options was removed as buggy for now. It might return some day. For now you have to use hardlink / sudo apt-get install hardlink

fslint

If you insist on a graphical user interface you might want to have a look at fslint / sudo apt-get install fslint (see website for description). It is more feature rich but also more complicated and less reliable.

Solution 2:

According to the fdupes --help command, -H does not create hardlinks:

 -H --hardlinks         normally, when two or more files point to the same
                                disk area they are treated as non-duplicates; this
                                option will change this behavior

Instead, -L seems to do this:

 -L --linkhard          hardlink duplicate files to the first file in
                              each set of duplicates without prompting the user