.bashrc or .bash_profile for ssh-agent?

It depends somewhat on how you personally use shells.

An interactive shell is anything that has a terminal connected to its input and output. Login shells are spawned by /bin/login. Login shells source your .bash_profile. Most terminal emulators such as xterm start an interactive shell that is not a login shell. Interactive, non-login shells source your .bashrc.

What I typically do is have my .bash_profile source my .bashrc (after checking if it has already been sourced), and then I put my ssh-agent setup in my .bashrc.

You should use .bash_profile because .bashrc is sourced for every interactive shell and you only need one agent per login session.