Grokking sophisticated dictionary attack

It appears that one of my servers is undergoing a sophisticated dictionary attack on ssh, in that I am seeing a bunch of usernames being tried on on of my servers alphabetically with password failures.

The unusual things about this are:

• There is only one attempt every 2 minutes
• every attempt comes from a different IP address

Can you help me understand how this can be done from different IP addresses, can it be effectively blocked? and does the slow persistence of the attack (probably will take several months to go through the alphabet) mean anything specific?

I'd also be interested to know if anyone else is currently seeing similar activity on their public ssh servers (ie. are we a specific target, or is this attacker blanketing thousands of ssh servers with this attack?)

Also, is there any way for me to reveal what passwords (or even hashes) are being tried? I notice that each name is only tried once or twice. I assume they are trying "password" or the user's username - but can i verify this?

The easy answer to this is don't use passwords!!!

You can:

• use ssh keys. Encrypt them for extra security
• use an one time password generator like Mobile-OTP
• use kerberos

If you must use passwords, allow them only from subnets you trust, not the whole internet. Also, it helps to run your ssh daemon on some port other than 22. Normally I don't suggest such things, but if it reduces the volume of attacks you see, so much the better.

In answers to your specific concerns:

• I've seen these sorts of attacks on any host I run that has a public facing ssh server.
• I suspect they're using a dictionary of user names / passwords developed from past experience with what works. For instance, my logs have
  • 18 attempts against user "ftp"
  • 23 attempts against user "forum"
  • 3 for user "bernard"
  • 1 for user "bret"
• I've also seen this sort of attack where they used actual site-specific logins, but those were probably culled from that site's wide-open ldap server (universities, gotta love 'em!)
• These attacks are probably coming from zombie computers that have some impressive centralized control.
• these attacks are coming at a rate of one every 1 to 5 minutes, all day, every day
• I just tried trussing sshd to see if it is trivial to collect the password the attacker is trying, and it looks like you'd have to patch the code.
  • these people did just that and found that the passwords were often permutations of the username or a well-known password such as sql or admin.

Here's some discussion and analysis of a "slow brutes" botnet attack that started in 2008: Slow Brutes Final Roundup

Other discussion from the same time period at this site suggests

Fail2ban is but one tool to improve server security. Of course the first step is to avoid use of passwords and use serverkey authentication instead. In addition limit allowed traffic to certain IPs, disallow IP ranges, use IP tables, use blacklists such as rsync-mirrors.uceprotect.net to update your iptables for known SSH hacker origins and origin networks. For details on above 3 levels of blacklists, see uceprotect.net.

Other items I've seen suggested related to this kind of attempt recommend monitoring for the failed attempts and blocking the source IPs - eventually the same IP will be used for another probe, so by detecting and blacklisting after attempts to log in with a nonexistent account you gradually reduce the threat at the cost of increased firewall rule processing. Also, it might be best to not block after only a single connection attempt - typos do happen.