Preventing certificate export in Windows for non-admin user

I just want to block/password protect the export of the certificate (and the private key), or invalidate the cert if it's done

You can apply password only to key access operations. If user has access to the key, then user can export it.

The only thing you can do is to store keys in TPM. User can freely access the key, but won't be able to export it to a file.