Which certificate does kubelet use?

This is a kubeadm based 1.16 cluster. The way I understand mTLS work between an apiserver & kubelet is:

apiserver -> kubelet

--kubelet-client-certificate & --kubelet-client-key => The certs & key given here(apiserver) is for apiserver(client) to kubelet (server)
--client-ca => The CA given here(kubelet) is what verfies if request from apiserver is indeed the right one.

And then we have the other way:

kubelet -> apiserver

--tls-cert-file & --tls-key-file => Passed in kubelet conf. Generally present at /var/lib/kubelet/pki/kubelet.{crt,key}
--kubelet-certificate-authority => The above cert gets verified by this CA passed in apiserver

However, there is a kubelet.conf file present at /etc/kubernetes which again has client-certificate-data and client-key-data . When is this client-cert used?

Also, if we enable auto-rotation in kubelet, there is a client-cert under /var/lib/kubelet/pki/kubelet-client-current.pem which may or may not be referenced in kubelet.conf based on the kubeadm version used (because of the bug in kubeadm prior to 1.17). I'm confused on how & when are the kubelet client cert used. Any help on this?

The client-certificate-data and client-key-data are there due to a bug. This is well reflected by the official docs:

Warning: On nodes created with kubeadm init, prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of kubelet.conf. After kubeadm init finishes, you should update kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with:

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

Some useful sources with more info:

• PKI certificates and requirements

• Certificate Management with kubeadm

EDIT:

Based on the kubelet integration:

When you run kubeadm join, kubeadm uses the Bootstrap Token credential to perform a TLS bootstrap, which fetches the credential needed to download the kubelet-config-1.X ConfigMap and writes it to /var/lib/kubelet/config.yaml. The dynamic environment file is generated in exactly the same way as kubeadm init.

Next, kubeadm runs the following two commands to load the new configuration into the kubelet:

systemctl daemon-reload && systemctl restart kubelet

After the kubelet loads the new configuration, kubeadm writes the /etc/kubernetes/bootstrap-kubelet.conf KubeConfig file, which contains a CA certificate and Bootstrap Token. These are used by the kubelet to perform the TLS Bootstrap and obtain a unique credential, which is stored in /etc/kubernetes/kubelet.conf. When this file is written, the kubelet has finished performing the TLS Bootstrap.

This indicates that cert/credentials stored in kubelet.conf are used during kubelet startup process. Removing this file or changing its content will be verified during kubelet restart. This can be tested yourself as below:

• If you remove the file:

failed to run kubelet: unable to load bootstrap kubeconfig

• If you misconfigure the file or provide wrong certs:

config loaded from file: /etc/kubernetes/kubelet/conf
bootstrap.go:240 unable to read existing bootstrap client config: f.e. invalid configuration
or unable to load TLS certificates from existing bootstrap client config: data does not contain any valid RSA or ECDSA certificate

Also, as a side note:

I strongly recommend using a newer version of Kubernetes as yours is 5 major versions older than the current stable one: 1.21 / April 8, 2021. Keeping things up-to-date will save you a lot of trouble in the future.