Monterey: SSH with hardware key only works once

I'm having an issue using my hardware security key with SSH on Monterey. The Apple-included OpenSSH build seems not to support hardware keys at all, as I get an error message saying Cannot load FIDO key <key> without provider.

To fix this, I have installed OpenSSH 8.8 from Homebrew. The first time I SSH with my key, it works as expected; however, all subsequent attempts generate the following error:

sign_and_send_pubkey: signing failed for ED25519-SK <key> from agent: agent refused operation
<REDACTED>: Permission denied (publickey).

In order to fix it, I need to killall ssh-agent every time. I never had this problem on Big Sur, and I'm not sure where to begin troubleshooting.

Looking a little deeper, whenever I run SSH, my Mac starts /usr/bin/ssh-agent and not my /opt/homebrew/bin/ssh-agent. Manually starting the Homebrew version of ssh-agent before running ssh does not fix this; the Mac still spawns the bundled ssh-agent so that both are running. It does this even when I explicitly run /opt/homebrew/bin/ssh.

Have you tried disabling the built-in agent? I found a blog post that explains this:

launchctl disable user/$UID/com.openssh.ssh-agent

I haven't tried it, but maybe it helps.