Can I set a limit on attempt number of ssh authentication passphrase?

I recently changed to ssh-key from password for security.

But ssh-key can also be compromised when the private key file is somehow stolen.

So I set a passphrase on the private key.

But it seems that I can try as many as I wish.

So is there any way I can set a limit on how many one can try passphrase?

No, you can't.

The passphrase is used to add a layer of encryption to the key. The process is completely offline, and the methods employed are well known, as the software is open source. It's basically a function taking the key and passphrase as input, and returning unencrypted key.

As it's completely offline there's no meaningful way to limit the number of attempts.