Stop ssh client from offering all the public keys it can find?

Solution 1:

This is expected behaviour according to the man page of ssh_config:

 IdentityFile
         Specifies a file from which the user's DSA, ECDSA or DSA authentica‐
         tion identity is read.  The default is ~/.ssh/identity for protocol
         version 1, and ~/.ssh/id_dsa, ~/.ssh/id_ecdsa and ~/.ssh/id_rsa for
         protocol version 2.  Additionally, any identities represented by the
         authentication agent will be used for authentication.  

         [...]

         It is possible to have multiple identity files specified in configu‐
         ration files; all these identities will be tried in sequence.  Mul‐
         tiple IdentityFile directives will add to the list of identities
         tried (this behaviour differs from that of other configuration
         directives).

Basically, specifying IdentityFiles just adds keys to a current list the SSH agent already presented to the client.

Try overriding this behaviour with this at the bottom of your .ssh/config file:

Host *
  IdentitiesOnly yes

You can also override this setting on the host level, e.g.:

Host foo
  User bar
  IdentityFile /path/to/key
  IdentitiesOnly yes

Solution 2:

Although others have hinted at this with configuration-based solutions, it's probably worth pointing out that you can easily do this one-time-only on the command line with:

ssh -o 'PubkeyAuthentication no' myhostname.mydomain