Linux: prevent outgoing TCP flood

I run several hundred webservers behind loadbalancers, hosting many different sites with a plethora of applications (of which I have no control). About once every month, one of the sites gets hacked and a flood script is uploaded to attack some bank or political institution. In the past, these were always UDP floods which were effectively resolved by blocking outgoing UDP traffic on the individual webserver. Yesterday they started flooding a large US bank from our servers using many TCP connections to port 80. As these type of connections are perfectly valid for our applications, just blocking them is not an acceptable solution.

I am considering the following alternatives. Which one would you recommend? Have you implemented these, and how?

• Limit on the webserver (iptables) outgoing TCP packets with source port != 80
• Same but with queueing (tc)
• Rate limit outgoing traffic per user per server. Quite an administrative burden, as there are potentially 1000's of different users per application server. Maybe this: how can I limit per user bandwidth?
• Anything else?

Naturally, I'm also looking into ways to minimize the chance of hackers getting into one of our hosted sites, but as that mechanism will never be 100% waterproof, I want to severely limit the impact of an intrusion.

Update: I'm currently testing with these rules, that would have prevented this specific attack. How would you propose to make them more generic? Am I missing a known TCP DoS attack when I only rate limit on SYN packets?

iptables -A OUTPUT -p tcp --syn -m limit --limit 100/min -j ACCEPT
iptables -A OUTPUT -p tcp --syn -m limit --limit 1000/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
iptables -A OUTPUT -p tcp --syn -j REJECT --reject-with tcp-reset

Cheers!

Solution 1:

The best possible solutions for my opinion and the one that has worked very well for me is to limit the number of connections/packets for destination ip. Setting the limit to a reasonable rate will prevent the attacker to send a big amount of connections to the target. Setting the port and protocol is not a good idea because if the attacker is sending http flood today, tomorrow he will use different type of attack. so limiting connections per ip as general would be a solution for your problem.

I hope it helps :)