DUH! It was right in the ansible config file (/etc/ansible/ansible.cfg)

# if set, always use this private key file for authentication, same as
# if passing --private-key to ansible or ansible-playbook
private_key_file = /home/<username>/.ssh/id_rsa

Now everything seems to work

10.0.0.5 | success >> {
"changed": false,
"ping": "pong"
}

If you ever run into SSH problems with ansible again, add -vvv to the command line options and it will tell you what went wrong.