No Pods reachable or schedulable on kubernetes cluster

Solution 1:

Problem resolved…

The cause of the problem was an update of the cluster to the kubernetes version 1.21 while my cluster was meeting the following conditions:

• private and public service endpoint enabled
• VRF disabled

Root cause:

In Kubernetes version 1.21, Konnectivity replaces OpenVPN as the network proxy that is used to secure the communication of the Kubernetes API server master to worker nodes in the cluster.
When using Konnectivity, a problem exists with masters to cluster nodes communication when all of the above mentioned conditions are met.

Solution steps:

• disabled the private service endpoint (the public one seems not to be a problem) by using the command
  ibmcloud ks cluster master private-service-endpoint disable --cluster <CLUSTER_NAME> (this command is provider specific, if you are experiencing the same problem with a different provider or on a local installation, find out how to disable that private service endpoint)
• refreshed the cluster master using ibmcloud ks cluster master refresh --cluster <CLUSTER_NAME> and finally
• reloaded all the worker nodes (in the web console, should be possible through a command as well)
• waited for about 30 minutes:
  • Dashboard available / reachable again
  • Pods accessible and schedulable again

General recommendation:

BEFORE you update any cluster to kubernetes 1.21, check if you have enabled the private service endpoint. If you have, either disable it or delay the update until you can, or enable VRF (virtual routing and forwarding), which I couldn't but was told it was likely to resolve the issue.