Block an IP-address which tries to download entire website

I need to block an IP-address which tries to download my entire website. Currently I've blocked it using iptables.

Is there any way to block an IP automatically if it's trying to request to many pages, i.e. downloading an entire website?

P.S. I'm using Nginx as frontend to Apache.

Thank you!

Solution 1:

Another possible option would be to use fail2ban. Under Debian this is part of the standard pacakges, for other distros you might have to get the original source and install/configure manually.

This is a package that can monitor all sorts of things and trigger an (IPtables based) block on the requesting IP address. You might have to experiment a little for this particular thing, but I would be surprised if it couldn't be made to work.

I use it on most of my servers, mostly to keep those irritating ssh login attempts from kiddy hackers under control, and it works like a charm.

Solution 2:

Read the Serverfault blog: http://blog.serverfault.com/post/1016491873/better-rate-limiting-for-all-with-haproxy