fail2ban "expired bantime" after unban
What does this expired bantime exactly mean?
This means that the ban if it would take place will be immediately expired due to your config, so unban would be called immediately after ban, what would make the banning process unnecessary.
The end of ban is calculated using formula:
end_of_ban = time_of_last_known_failure_causing_ban + bantime
This is happening because related to your configuration end of ban of lot of messages is probably smaller than now, so fail2ban filter ignoring this and is simply searching for further messages.
Another possibility could be that your banning action does not really work so the banned "intruder" (or rather simulated stress tester) is still able to produce messages, so this does overflooding the log.
Or fail2ban is simply unable to scroll over the log faster than you generates the messages.
It seems that fail2ban will keep on processing stuff even though stress testing with ab has stopped a long time ago
Well this signaling indeed that something is not correct in your config: slow regex's, wrong action etc or even too many messages.
Please take a look at fail2ban wiki :: Best practice describing certain optimizations.
Is there a possibility that there is a buffer of some sort on fail2ban that I could reduce
Not really. But you can probably reduce parasite log traffic if you would switch a procedure:
- don't monitor access log with too many messages or use optimized regexs (and better appropriated logging format) to increase speed of the filter (line matching);
- try to use some modules of web-server (like limit-req of nginx and fail2ban's filters like nginx-limit-req) instead of scanning of the access log;
- or use better suitable methods in order to generate fewer log messages, e. g. like logging http-packets of IP using iptables together with limit-burst or another similar things.