What is the cryptographic relationship between an ssh key and my Yubikey?
I am curious what is the cryptographic relationship between the generated ssh private (and public) key when I use my Yubikey to add an extra layer of protection.
Does ssh-keygen write anything into the Yubikey itself or just communicate with it? If it writes, what are the limitations?
There are two possible algorithms with secure key options: Ed25519-SK and ecdsa-sk. As far as I know, Ed25519-SK is suggested, whenever it is possible.
I am curious what is the cryptographic relationship between the generated ssh private (and public) key when I use my Yubikey to add an extra layer of protection.
The one universal thing is that the private key is never actually revealed to you. That "private key" file is unusable until its information is loaded back into the Yubikey, at which point the Yubikey will be able to create signatures upon request.
Exactly how that's implemented can vary, as not all U2F "security keys" are Yubikeys, and even Yubikeys are not required to do it the same way; at least 3 different mechanisms can be used to achieve this result.
Does ssh-keygen write anything into the Yubikey itself
When generating *-sk keys, the short answer is "technically it depends, but in practice no (unless you use -O resident, in which case yes)".
In theory, the U2F token could indeed store all those keypairs in memory, like it used to be done with smartcards (compare Yubikey's "PIV" mode), but in practice tokens achieve this without having any writable memory at all.
Even with the same U2F token, each website is meant to have a different key, to prevent user tracking. During U2F credential generation, the token returns a public key plus a "key handle" that would be stored by the website – later during login, that "key handle" is sent back from the website to the token, so that it would know which keypair to use for this specific site.
(And whenever you run ssh-keygen for an -sk key, the resulting id_*-sk private key file contains just the public key and the "key handle", the same information that a website would. The only difference is that you'll be sharing the same keypair across all SSH hosts, instead of automatically creating unique keypairs per server.)
The trick is that each Yubikey token has an AES key factory-written to it, and whenever it's asked to generate a new U2F credential, it'll encrypt the new private key with its internal AES key and return the whole private key as the "key handle" to the website. Then it just forgets the key completely and waits for the next request.
To the website, that "key handle" is opaque – it looks just like a very large key ID, and gets stored in a regular database table. But during login, as the website sends the "key handle" back to the token, the Yubikey can decrypt it and recover the same private key again – and when done, throw it away again.
I think this is generally called 'key wrapping', and it allows the same token to support a literally unlimited amount of keypairs. It is documented that Yubikeys use this method, but it's also how most other U2F/FIDO token work as well. The same trick is also used when generating keys on TPM2 chips, too.
(There's also another similar approach, which I think I've read about one manufacturer using on their U2F tokens: deterministic key generation, where the token has a burned-in seed for key generation, and uses something vaguely like Hash(seed || key_handle || AppID) in order to generate the same key for the same website from scratch every time.)
So in short, generating an 'ed25519-sk' key with a Yubikey will not write anything to its internal storage – all necessary information is stored encrypted in your private key file, and uploaded back to the Yubikey in RAM.
But one more thing to mention is the resident keys feature of FIDO2. While the original U2F was only meant as an additional factor, FIDO2 is supposed to fully replace the password entry – in fact, apparently the intent is to make FIDO2 tokens usable without any keyboard input at all.
To make this work, FIDO2 tokens do have persistent memory for a certain amount of keypairs (together with the corresponding usernames and appIDs), but it is not used by default – the key is only stored on the token if the application specifically requests a "resident" credential. For OpenSSH, ssh-keygen's -O resident option will cause the keypair to be stored on the token itself, with the -K option allowing it to be downloaded back into 'id_whatever' files.
Note that U2F/FIDO is completely independent the PIV and OpenPGP smartcard-emulation modes that can be found in various Yubikey models. The three modes use different tools and different protocols.
In the PIV mode, ECP256 keypairs can be generated on the PIV smartcard using ykman and accessed through ssh's PKCS11Provider, and they are written to the Yubikey's internal memory – no local files are kept, the PKCS#11 module will retrieve everything from the Yubikey as needed. The PIV card that Yubikeys emulate has a stricter structure than most smartcards; I think it can hold 3 keypairs. Of course, they can be erased and re-generated at any time.
Similarly in the OpenPGP mode, gpg can generate ECP256 keypairs on the OpenPGP smartcard and they can be used for SSH through gpg-agent – those, too, are stored internally to the Yubikey, although this time only partial information is stored. The entire private key is held in the card, but gpg keeps the PGP metadata and a "stub" reminding it that the corresponding private key is on the card. OpenPGP cards can hold 3 keypairs total, but only one of them can be used for SSH.