How to properly handle a gzipped page when using curl?
Solution 1:
curl
will automatically decompress the response if you set the --compressed
flag:
curl --compressed "http://example.com"
--compressed (HTTP) Request a compressed response using one of the algorithms libcurl supports, and save the uncompressed document. If this option is used and the server sends an unsupported encoding, curl will report an error.
gzip is most likely supported, but you can check this by running curl -V
and looking for libz somewhere in the "Features" line:
$ curl -V
...
Protocols: ...
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
Note that it's really the website in question that is at fault here. If curl
did not pass an Accept-Encoding: gzip
request header, the server should not have sent a compressed response.