How do I keep my SSH private key passphrase away from OSX?

I have no inside information about the MacOS internals, but from my experience using this feature, the passphrase is accepted by the GUI only to add the corresponding key to an ssh-agent. On my system it did not store the passphrase in the keychain (which I also do not want), but I'm not sure that's default behavior. My terminal would report "saving to keychain failed", but I could then ssh using that key until I logged out / restarted. See the "AskPassGUI" and "KeychainIntegration" options in ssh_config (superseded in more recent MacOS versions, though). Because it did not save the passphrase in the keychain, I happily entered my passphrase only once a week (shutting down on the weekend and never logging out).