fail2ban does not seem to ban an IP after repeated failed ssh login attempts

The following issue can be useful for more information: https://github.com/fail2ban/fail2ban/issues/2765, although it relates to more recent versions of fail2ban than the one mentioned in the question.

By default, the sshd filter doesn't ban repeated failed passkey login attempts for an existing/valid user.

On more recent versions (although not yet available in 0.11.1), the sshd filter has a publickey parameter than can be set to any in order to capture these failed attempts:

[sshd]
publickey = any
...

Otherwise, there is the option to use ddos (equivalent to your older ssh-dos filter?) or aggressive mode for the sshd filter (which would catch the 'connection closed' log entries followed by unsuccessful login attempts):

[sshd]
mode = aggressive
...

Finally (and this should be useful on any fail2ban version not having the publickey parameter in the sshd filter), you could add the following regex to your filter as suggested here:

[sshd]
failregex = %(known/failregex)s
            ^Failed publickey for <F-USER>.+</F-USER> from <HOST>

This worked for me under debian 10.

[sshd]
failregex = %(known/failregex)s
            ^.*Connection closed by authenticating user [a-z][-a-z0-9_]* <HOST> port \d+ \[preauth\]