Kubernetes API-server wont' start after reboot

Every time I install Kubernetes 1.15.3 it works great, until I reboot the VM and then it no longer listens on port 6443. Swap is off. /etc/tab only contains the root volume on my Ubuntu 18.04 server VM. Note, using flannel

cat /proc/swaps
Filename                                Type            Size    Used    Priority

kubectl only returns refused

kubectl get nodes
The connection to the server <myip>:6443 was refused - did you specify the right host or port?

sudo netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      
770/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      
1694/sshd
tcp        0      0 127.0.0.1:36957         0.0.0.0:*               LISTEN      
1269/containerd
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      
2720/kubelet
tcp        0      0 127.0.0.1:33385         0.0.0.0:*               LISTEN      
2720/kubelet
tcp6       0      0 :::22                   :::*                    LISTEN      
1694/sshd
tcp6       0      0 :::10250                :::*                    LISTEN      
2720/kubelet
udp        0      0 127.0.0.53:53           0.0.0.0:*                           
770/systemd-resolve

systemctl status kubelet

Oct 21 12:47:23 kubernetes1 kubelet[2720]: E1021 12:47:23.811807    2720 
reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list 
*v1beta1.RuntimeClass: Get https://<MYIP>:6443/apis/nodOct 21 
12:47:23 kubernetes1 kubelet[2720]: E1021 12:47:23.824760    2720 
kubelet.go:2248] node "kubernetes1" not found

ps -ef | grep kube

root      2720     1  0 12:32 ?        00:00:07 /usr/bin/kubelet -- 
bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf -- 
kubeconfig=/etc/kubernetes/kubelet.conf -- 
config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network- 
plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv- 
conf=/run/systemd/resolve/resolv.conf

Using flannel

sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 

Any ideas? The "interweb" only tells me to check swap, but I've checked it literally 6 different ways and all indications is that it's off.

My the admin conf file copied to .kube shows the right server and right port. The API-server is just not up. Thoughts anyone?

journalctl -xeu kubelet

Note, kubernetes1 is the name of my master server.

Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.521057    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.610016    2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://<myip>:6443/api/v1/podsOct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.621170    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.721293    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.810053    2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://<myip>:6443/api/v1/servicesOct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.823444    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.923615    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.010365    2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://<myip>:6443/api/v1/nodes?fieldOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.023809    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.123954    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.210384    2720 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://<myip>:6443/apis/nodOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.225717    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.325890    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.410429    2720 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https://<myip>:6443/apis/storagOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.425998    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.526126    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.610341    2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://<myip>:6443/api/v1/podsOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.626218    2720 kubelet.go:2248] node "kubernetes1" not found

Solution 1:

After re-installing the OS and Kubernetes 1.15.5 the issue is with DNS. Docker images would not pull because the docker server could not resolve after a restart.

I previously tried to modify the resolv.conf directly as mentioned above, but Ubuntu wants to manage resolv.conf itself with netplan. Making manual changes and rebooting looks like it results in those changes being ignored.

If I modify the netplan yaml, and run netplan apply, it puts 127.0.0.53 in the resolv.conf and runs a local dns proxy, which then forwards properly to the bind9 dns server I have configured. I expect to see my dns servers in the resolv.conf and that's what caused the problem.

Do not modify the resolv.conf directly on Ubuntu. Use netplan.