Faster zlib alternatives [closed]
I wonder, if there are any faster builds of zlib around with more advanced optimizations?
If it's possible to optimize it using SSE instructions or Intel C++ compiller, or some trick which were patented earlier (I know patents were a serious limitation during gzip/zlib development), have anyone bothered to implement that?
I am especially interested in compression speed, which have a direct impact on high-performance web-services serving static & dynamic content.
Solution 1:
Several years after this question was first asked some faster x86_64 zlib's appeared using the type of optimization BarsMonster suggested:
- Cloudflare zlib
- Intel zlib
- Intel's igzip
- Google's c-zlib
- libdeflate
The authors of samtools (a suite of tools for interacting with high-throughput sequencing data) produced a comparison of zlib speeds.
zlib-ng collects zlib optimizations not in mainline zlib but it may not be as stable as mainline zlib. Its issue tracker is also instructive as a reference to other known zlib speed ups.
Recently a pull request that claims to "speed up inflate_fast by up to 1.5x on x86_64" (by filling and copying 8 bytes at a time) has been made but not accepted into mainline zlib. The patch providing this work is also having a bumpy ride being accepted into Chromium (see this Chromium bug and the Chromium review) but hopefully the submitter will take a break to step away and recharge because from a distance it looks like progress was being very slowly made in a complex situation...
Even more recently, libdeflate is a heavily optimised rewrite that produces/confumes zlib compatible output/input but is does not provide a zlib compatible API. At the time of writing this has the zlib speed crown.
Solution 2:
Another alternative is the miniz library (public domain, unlicense.org), which implements most of the zlib API in a single C source file and reads/writes zlib-compatible compressed data streams. At compression level 1 it uses a real-time compressor which is extremely fast (a little slower than minilzo, but with higher compression).