kubectl logs error: You must be logged in to the server (the server has asked for the client to provide credentials

While looking around ... I saw this on my worker nodes
Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid

After further troubleshooting ... only 2 of my 3 masters were causing this error
kubectl logs error: You must be logged in to the server (the server has asked for the client to provide credentials

After checking a lot of resources ,, I really couldn't find what is causing the problem , so I decided to reboot each of the 2 failing masters one at a time and that did the trick. I guess some of the pods in kube-system required restarting.

Additionally , I restarted kubelet on all worker nodes , but not sure if this had an effect or not.

Note that in https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#manual-certificate-renewal there is no mention about rebooting the masters

A final note .. I am not sure why the cert renew was not as smooth .. Before running into the kubectl logs problem described in this post ... I ran into this error on my first master ("bootstrap-kubelet.conf does not exist" issue which would not allow kubelet to restart) so I had to follow fix in https://stackoverflow.com/questions/56320930/renew-kubernetes-pki-after-expired