Can I make ssh-agent wait until I use ssh to prompt for a password?

I have ssh-agent set up to automatically start and prompt when I open a terminal window, as is explained here. It gets somewhat annoying though since many times when I open a shell I don't plan on using ssh for anything. Is there a way to delay this password prompt until I try to use ssh?

In other words, the first time I use SSH I want a password prompt, all subsequent times I don't want one, and I don't want one if I'm not about to use SSH.

You'd have to write (or find) an agent proxy that would detect absence of the key in the agent and prompt for the passphrase when the key is requested, then load the key into the real agent.