How can I rate limit SSH connections with iptables?

I'm trying to limit the number of possible SSH connections to my server but it just seems to lock me out every time. I'm not very familiar with iptables but I've been reading up on the rules that I need to apply to rate limit the connections but no success. Here's my iptables config file:

:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [28130:3096101]
:RH-Firewall-1-INPUT - [0:0]
:WebServices - [0:0]
-A INPUT -p tcp --dport 2020 -m state --state NEW -m recent --set --name SSH
-A INPUT -p tcp --dport 2020 -m state --state NEW -m recent --update --seconds 120 --hitcount 8 --rttl --name SSH -j DROP
-A INPUT -j WebServices
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p esp -j ACCEPT
-A RH-Firewall-1-INPUT -p ah -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
-A WebServices -p tcp -m tcp --dport 8088 -j ACCEPT
-A WebServices -p tcp -m tcp --dport 443 -j ACCEPT
-A WebServices -p tcp -m tcp --dport 80 -j ACCEPT
COMMIT

The top 2 rules I believe should prevent any new connections being established from a host if they exceed 8 connections in 2 minutes but it just locks down the port all together. What am I doing wrong?

Solution 1:

Just add the following:

-A INPUT -p tcp --dport 2020 -m state --state NEW -j ACCEPT

... right after these two lines:

-A INPUT -p tcp --dport 2020 -m state --state NEW -m recent --set --name SSH
-A INPUT -p tcp --dport 2020 -m state --state NEW -m recent --update --seconds 120 --hitcount 8 --rttl --name SSH -j DROP

Also, you should think about a cron task that will clean your /proc/net/ipt_recent/SSH (ipt_recent may be xt_recent on newer platforms) from time to time, in case you get locked out.

EDIT: Your rule ordering looks a bit odd to me, if I were you I would go with something like this:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]

# Loopback interface
-A INPUT -i lo -j ACCEPT

# ICMP traffic
-A INPUT -p icmp --icmp-type any -j ACCEPT

# Already established connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# SSH
-A INPUT -p tcp --dport 2020 -m state --state NEW -m recent --set --name SSH
-A INPUT -p tcp --dport 2020 -m state --state NEW -m recent --update --seconds 120 --hitcount 8 --rttl --name SSH -j DROP
-A INPUT -p tcp --dport 2020 -m state --state NEW -j ACCEPT

# Web services
-A INPUT -p tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -p tcp --dport 443 -m state --state NEW -j ACCEPT
-A INPUT -p tcp --dport 8088 -m state --state NEW -j ACCEPT

# Reject everything else
-A INPUT -j REJECT --reject-with icmp-host-prohibited

COMMIT