Losslessly compressing similar images?

I need to reduce the size of my photo library so I naturally want to compress them. Many of them are not quite identical, but still very similar (subsequent shots of the same scene). Is there any compression algorithm that takes advantage of this fact to effectively compress these images? 7zip (LZMA) is useless.

Solution 1:

You might try Paq 8 (fp8_v2.zip). I just tried it myself on 1440 similar PNG images and then again on 111 similar JPG images. Here are the results.

• 1440 PNG Files, 28,631,615 bytes => 2,058,653 bytes compressed
• 111 JPG Files, 15,003,820 bytes => 489,096 bytes compressed

Compression of the PNG files took about 8 minutes and 550 MB of memory when using:

fp8_v2.exe -7 images *.png

Compression of the JPG files took about 5 minutes and 125 MB of memory when using:

fp8_v2.exe -5 images image12*.jpg

See also: jpg lossless image compression test

Solution 2:

I would imagine that that the burrows-wheeler transform with an arithmetic coder would be ideal for this given a large enough window. What happens if you configure BZIP2 to use a block size equal to a small run of photos? It'll be slower and take more memory but the compression ratio should skyrocket. And have you tried LZMA with larger block sizes yet?

Solution 3:

Here's a simple solution which doesn't work for photos but may work if one has several images with large pixel-by-pixel identical areas: save the images in an unpacked format like BMP (not PNG or GIF) and then TAR them and compress with a decent compressor like XZ, e. g. on Linux with something like

tar -c myDirectory | xz -9 >myDirectory.tar.xz

Instead of TAR and XZ, one may use 7-Zip with the “solid archive” option to get roughly the same performance. This way I could compress 16 similar screenshots, that took about 900 KB each when saved as separate PNG files, into a 2 MB archive. The benefit of this solution is that it uses common file formats, so it works without installing new software. (Unfortunately the older and even more common programs GZIP and BZIP2 didn't do a good job for me — maybe because the block size of BZIP2 cannot be configured to be larger than 900 KB.)