backup files with many duplicated files

Solution 1:

I also just went through this too.

If you compress your files into a Tar Ball, 7z's LZMA compression may or may not recognise the duplicates if they are separated too far in the Tar Ball (it's a function of Dictionary Size and a few other things).

7z has a WIM format which collects duplicates, then you can use normal LZMA compression for there. Windows Command Line Example:

7z a -twim "Example.wim" *
7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on -mhc=on "Example.7z" "Example.wim"
del "Example.wim"

It works well, give it a go.

Solution 2:

I suggest 3 options that I've tried (in Windows):

1. 7zip LZMA2 compression with dictionary size of 1536Mb
2. WinRar "solid" file
3. 7zip WIM file

I had 10 folders with different versions of a web site (with files such as .php, .html, .js, .css, .jpeg, .sql, etc.) with a total size of 1Gb (100Mb average per folder). While standard 7zip or WinRar compression gave me a file of about 400/500Mb, these options gave me a file of (1) 80Mb, (2) 100Mb & (3) 170Mb respectively.