Limit incoming connections using iptables per IP

I think that this is what you need to limit each source IP to a specified rate.

-m hashlimit --hashlimit-mode srcip --hashlimit-upto 5/min

Detailed examples are given in the URL's below.

http://www.mauromascia.com/blog/limiting-concurrent-connections-per-ip/?lang=it

http://etel.wiki.oreilly.com/wiki/index.php/SIP_DoS/DDoS_Mitigation

http://ipset.netfilter.org/iptables-extensions.man.html