How to remember/cache or specify private key passphrase for Ansible

The feature is called ssh-agent:

$ eval `ssh-agent`  # you might have agent already running so this might not be needed
$ ssh-add /etc/ansible/private_keys/confman.crt

now ansible should be able to find the key in agent and authenticate without asking for passphrase every time. From: Documentation: Your first commands