Somebody eating all our bandwidth, what should I do?

Don't block the robot completely, but slow it down with QoS rules (in linux, check out the "tc" command). Also, read this. This is how the people running this very website solved the problems using Haproxy.


I would like to suggest PHREL. Very small and simple tool that makes dynamically blocking abusive IPs a breeze. I deployed it on some caching nameservers with immediate results. Test it out a bit and figure out what your thresholds should be and let it rip!

http://www.digitalgenesis.com/software/phrel/


Playing whack-a-mole is usually a fruitless effort, but it may be the only option here.

You could possibly automate the identification of your bandwidth hogs. I would look at monitoring the web logs and identifying anomalous behavior such as the same IP looking at the same page more than once a second over a defined period of time, or some other similar metric. You don't mention which platform this is, but it would be a bit easier to implement in Linux than Windows.