Is Configuration Management useable for a small number of servers?

Nope, it's not defeating the purpose. I, in fact, use Ansible to set up single servers for hobby/side-project use quite frequently. It allows me to keep a version-controlled, repeatable, self-documenting configuration for the server.

I have to disagree with the current consensus. In no way can it be reasonable to learn any CM system for a pool of 2 servers, unless you simply want an excuse to learn that CM system. The amount of time you spend learning, writing, testing, and applying the config, will definitely exceed the hand built time. If you knew one of those systems, it's a much more reasonable proposition.