How to protect server against brute-forcing of http authentication?

This is exactly the kind of thing that fail2ban is good at.

Take a look at your error_log and notice the messages that are generated when a login failure occurs.

Now take a look at the filters provided by fail2ban (fail2ban/filter.d). One of them may already be configured to react to the kind of error messages you saw earlier if so then all you need to do is enable it in fail2ban/jail.conf.

If none of the pre supplied filters will do what you want it's generally fairly straightforward to construct your own.

You could also try mod_security. This is a open source rule based webapplication firewall. There are some free default rules provided by SpiderLabs - if they are not enough, you might be able to construct your own.