Limit incoming connections using iptables per IP
I think that this is what you need to limit each source IP to a specified rate.
-m hashlimit --hashlimit-mode srcip --hashlimit-upto 5/min
Detailed examples are given in the URL's below.
http://www.mauromascia.com/blog/limiting-concurrent-connections-per-ip/?lang=it
http://etel.wiki.oreilly.com/wiki/index.php/SIP_DoS/DDoS_Mitigation
http://ipset.netfilter.org/iptables-extensions.man.html