RDP with client authentication via a certificate or a key file

Forward the RDP and Wake-on-LAN ports to my machine, naturally.

This is not recommended, as the RDP service in Windows has historically had many security issues that were exploitable without authentication. Although requiring NLA (which is the default) improves the situation a bit, it's still considered an unnecessarily high risk.

So really, consider using a VPN tunnel just for that reason – or SSH if you want to carry it around to random PCs where VPN apps would be difficult to install. (Yet another alternative is Guacamole which is a browser-based gateway to VNC and RDP servers.)

confusing IPsec setup which also seems to only work on Windows Server, or smart cards

IPsec works just fine on client versions of Windows, although it is quite confusing to set up – Windows 10 has like three IPsec clients; the legacy XP one (obsolete IKEv1-only), the new one that's part of Windows Firewall (wf.msc); and the one that is configured as an "IKEv2 VPN" connection.

It's smart cards that are likely to require Windows Server.

or smart cards (what I'd like to use is specifically a USB flash drive).

"Smart Card Authentication" doesn't strictly require the certificate to be on a physical smartcard (which do come in the shape of self-contained USB tokens) – it only requires the certificate to be available through Windows CAPI, but it'll actually accept certificates whose private key was simply imported from a .pfx file into the "software-based" Windows certificate store.

But the biggest issue with this plan is that then a copy of your private key remains in whichever machine you install it on (as the RDP client doesn't support reading it directly from a file), so you'd be trusting any random computer with your password and your certificate keypair. Really even if you didn't have to install the .pfx file, you'd still be trusting any random computer with your certificate keypair on the USB stick.

That's actually the whole point of smart cards – the private key cannot be extracted from them, even by a malicious PC; the card performs signing operations internally and returns only that result. But you don't have to carry a whole card reader around – as mentioned, there are USB-stick-shaped smart cards, of which Yubikey is a popular one. (I am specifically referring to the "CCID/PIV" mode in Yubikey NEO and later models, emulating a PIV card which doesn't even need drivers like other smartcards do.)

Another issue is that from what I've found out, "Smart Card Authentication" in RDP relies on Active Directory, as it's implemented through Kerberos PKINIT and not through ordinary direct certificate authentication. (Not necessarily Windows Server – Samba can do AD – but it still needs AD.) It is also an alternative for the user's AD password, not a second factor.

The final problem is that some of the security issues that previously affected Windows RDP service were pre-authentication, meaning that they could have been exploited before verifying your password – and therefore also before verifying your "smart card" credentials. (Although they've been patched, it's definitely not a great track record compared to e.g. OpenSSH or WireGuard.)

(which would probably mean ditching RDP altogether and using VNC or a similar alternative)?

RDP still works over an SSH tunnel; just make sure to configure it with the "local" port other than 3389. (You do lose the UDP-based transport mode but it's optional and the difference in performance isn't even very noticeable.)