Kubernetes API-server wont' start after reboot
Every time I install Kubernetes 1.15.3 it works great, until I reboot the VM and then it no longer listens on port 6443. Swap is off. /etc/tab only contains the root volume on my Ubuntu 18.04 server VM. Note, using flannel
cat /proc/swaps
Filename Type Size Used Priority
kubectl only returns refused
kubectl get nodes
The connection to the server <myip>:6443 was refused - did you specify the right host or port?
sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
770/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
1694/sshd
tcp 0 0 127.0.0.1:36957 0.0.0.0:* LISTEN
1269/containerd
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN
2720/kubelet
tcp 0 0 127.0.0.1:33385 0.0.0.0:* LISTEN
2720/kubelet
tcp6 0 0 :::22 :::* LISTEN
1694/sshd
tcp6 0 0 :::10250 :::* LISTEN
2720/kubelet
udp 0 0 127.0.0.53:53 0.0.0.0:*
770/systemd-resolve
systemctl status kubelet
Oct 21 12:47:23 kubernetes1 kubelet[2720]: E1021 12:47:23.811807 2720
reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list
*v1beta1.RuntimeClass: Get https://<MYIP>:6443/apis/nodOct 21
12:47:23 kubernetes1 kubelet[2720]: E1021 12:47:23.824760 2720
kubelet.go:2248] node "kubernetes1" not found
ps -ef | grep kube
root 2720 1 0 12:32 ? 00:00:07 /usr/bin/kubelet --
bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --
kubeconfig=/etc/kubernetes/kubelet.conf --
config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-
plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-
conf=/run/systemd/resolve/resolv.conf
Using flannel
sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Any ideas? The "interweb" only tells me to check swap, but I've checked it literally 6 different ways and all indications is that it's off.
My the admin conf file copied to .kube shows the right server and right port. The API-server is just not up. Thoughts anyone?
journalctl -xeu kubelet
Note, kubernetes1 is the name of my master server.
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.521057 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.610016 2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://<myip>:6443/api/v1/podsOct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.621170 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.721293 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.810053 2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://<myip>:6443/api/v1/servicesOct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.823444 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.923615 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.010365 2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://<myip>:6443/api/v1/nodes?fieldOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.023809 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.123954 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.210384 2720 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://<myip>:6443/apis/nodOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.225717 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.325890 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.410429 2720 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https://<myip>:6443/apis/storagOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.425998 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.526126 2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.610341 2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://<myip>:6443/api/v1/podsOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.626218 2720 kubelet.go:2248] node "kubernetes1" not found
Solution 1:
After re-installing the OS and Kubernetes 1.15.5 the issue is with DNS. Docker images would not pull because the docker server could not resolve after a restart.
I previously tried to modify the resolv.conf directly as mentioned above, but Ubuntu wants to manage resolv.conf itself with netplan. Making manual changes and rebooting looks like it results in those changes being ignored.
If I modify the netplan yaml, and run netplan apply, it puts 127.0.0.53 in the resolv.conf and runs a local dns proxy, which then forwards properly to the bind9 dns server I have configured. I expect to see my dns servers in the resolv.conf and that's what caused the problem.
Do not modify the resolv.conf directly on Ubuntu. Use netplan.