How to detect SMTP AUTH attempts in Fail2Ban?

Your idea of adding -v to smtpd helped me solve this problem. I'm on Debian.

First I created /etc/fail2ban/jail.local:

[postfix]
enabled = true
logpath = /var/log/mail.log
bantime = 86400
findtime = 86400
maxretry = 2

This tells fail2ban to watch /var/log/mail.log with a maximum of 2 attempts during 24 hours, which would lead to a ban of 24 hours as well.

Then I copied /etc/fail2ban/filter.d/postfix.conf to /etc/fail2ban/filter.d/postfix.local, and added this line:

failregex = ^%(__prefix_line)s> \S+\[<HOST>\]: 503 5\.5\.1 .*$

This would trigger upon:

postfix/smtpd[23185]: > unknown[x.x.x.x]: 503 5.5.1 Error: authentication not enabled

UPDATE (October 2021):

Above works nicely with fail2ban 0.9.6. However, fail2ban 0.10.2 changed the layout of postfix.conf a bit.

To fix, again, copy the new /etc/fail2ban/filter.d/postfix.conf to /etc/fail2ban/filter.d/postfix.local. In this .local file, edit the normal section as follows:

mdpr-normal = (?:NOQUEUE: reject:|improper command pipelining after \S+|>)
mdre-normal=^RCPT from [^[]*\[<HOST>\]%(_port)s: 55[04] 5\.7\.1\s
            ^RCPT from [^[]*\[<HOST>\]%(_port)s: 45[04] 4\.7\.1 (?:Service unavailable\b|Client host rejected: cannot find your (reverse )?hostname\b)
            ^RCPT from [^[]*\[<HOST>\]%(_port)s: 450 4\.7\.1 (<[^>]*>)?: Helo command rejected: Host not found\b
            ^EHLO from [^[]*\[<HOST>\]%(_port)s: 504 5\.5\.2 (<[^>]*>)?: Helo command rejected: need fully-qualified hostname\b
            ^VRFY from [^[]*\[<HOST>\]%(_port)s: 550 5\.1\.1\s
            ^RCPT from [^[]*\[<HOST>\]%(_port)s: 450 4\.1\.8 (<[^>]*>)?: Sender address rejected: Domain not found\b
            ^from [^[]*\[<HOST>\]%(_port)s:?
            ^[^[]*\[<HOST>\]: 503 5\.5\.1 Error: authentication not enabled\b

Note the addition of |> to mdpr-normal.

In /etc/fail2ban/jail.local, add mode = normal to the [postfix] section.

And do not forget to add -v to the smtp directive in /etc/postfix/master.cf:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (no)    (never) (100)
# ==========================================================================
smtp      inet  n       -       y       -       -       smtpd -v

Restart your services and you should be good to go.