kube-apiserver exits while control plane joining the HA cluster
- Ubuntu 16.04.5
- All operation are done as root user
- software versions are as below:
ii kubeadm 1.20.4-00 amd64 Kubernetes Cluster Bootstrapping Tool
ii kubectl 1.20.4-00 amd64 Kubernetes Command Line Tool
ii kubelet 1.20.4-00 amd64 Kubernetes Node Agent
ii kubernetes-cni 0.8.7-00 amd64 Kubernetes CNI
ii containerd.io 1.2.6-3 amd64 An open and reliable container runtime
I'm Creating Highly Available clusters with kubeadm following the guide:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
Assumed that the control plane nodes are: A(devops1ar01n01 172.16.80.3) B(devops1ar01n02 172.16.80.4) C(devops1ar01n03 172.16.80.5).
I followed the link below to setup a load balancer using kube-vip. On both A and B I created file /etc/kube-vip/config.yaml and /etc/kubernetes/mainfest/kube-vip.yaml:
https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#kube-vip
The I ran command below to init the first control plane node A(kube-vip listens on port 16443):
kubeadm init --control-plane-endpoint kube-vip:16443 --upload-certs
The output are as follow:
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join kube-vip:16443 --token pa0bw2.gn6bqnyjlmh0o7xn \
--discovery-token-ca-cert-hash sha256:fd7bb5afe0307b8694c218f07c1f3adbf270254d1f37bcec75ed292b7223cc8b \
--control-plane --certificate-key 44995042d21c87ea5ed4f62443fe665cbfd7c71397485ca9f06d1483548c1883
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join kube-vip:16443 --token pa0bw2.gn6bqnyjlmh0o7xn \
--discovery-token-ca-cert-hash sha256:fd7bb5afe0307b8694c218f07c1f3adbf270254d1f37bcec75ed292b7223cc8b
Then I followed the output to run commands:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
Then I installed the CNI plugin weave on node A by running command below:
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=1.20.4"
Then check the pods:
root@devops1ar01n01:~# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-74ff55c5b-s2bh5 0/1 Running 0 36m
coredns-74ff55c5b-stm2l 1/1 Running 0 36m
etcd-devops1ar01n01 1/1 Running 0 36m
kube-apiserver-devops1ar01n01 1/1 Running 0 36m
kube-controller-manager-devops1ar01n01 1/1 Running 0 36m
kube-proxy-bnzpd 1/1 Running 0 36m
kube-scheduler-devops1ar01n01 1/1 Running 0 36m
kube-vip-devops1ar01n01 1/1 Running 0 36m
weave-net-8fmf9 2/2 Running 0 14s
Til then, everyting goes well, but something goes wrong while node B joining the cluster.(--v=8 to see the verbose output, --ignore-preflight-errors="DirAvailable--etc-kubernetes-manifests" due to the existing file /etc/kubernetes/mainfest/kube-vip), below command is run on node B.
kubeadm join kube-vip:16443 --token pa0bw2.gn6bqnyjlmh0o7xn \
--discovery-token-ca-cert-hash sha256:fd7bb5afe0307b8694c218f07c1f3adbf270254d1f37bcec75ed292b7223cc8b \
--control-plane \
--certificate-key 44995042d21c87ea5ed4f62443fe665cbfd7c71397485ca9f06d1483548c1883 \
--ignore-preflight-errors="DirAvailable--etc-kubernetes-manifests"
--v=8
The following message are shown then(172.16.80.4 is the ip of node B):
[kubelet-check] Initial timeout of 40s passed.
I0226 11:12:44.981744 11128 etcd.go:468] Failed to get etcd status for https://172.16.80.4:2379: failed to dial endpoint https://172.16.80.4:2379 with maintenance client: context deadline exceeded
I0226 11:12:52.890038 11128 etcd.go:468] Failed to get etcd status for https://172.16.80.4:2379: failed to dial endpoint https://172.16.80.4:2379 with maintenance client: context deadline exceeded
I0226 11:13:03.915500 11128 etcd.go:468] Failed to get etcd status for https://172.16.80.4:2379: failed to dial endpoint https://172.16.80.4:2379 with maintenance client: context deadline exceeded
I0226 11:13:19.337921 11128 etcd.go:468] Failed to get etcd status for https://172.16.80.4:2379: failed to dial endpoint https://172.16.80.4:2379 with maintenance client: context deadline exceeded
I found that there is no etcd container created on node B:
root@devops1ar01n02:~# docker ps | grep -v pause
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b0188090c251 ae5eb22e4a9d "kube-apiserver --ad…" 20 seconds ago Up 19 seconds k8s_kube-apiserver_kube-apiserver-devops1ar01n02_kube-system_50f7004f736896db78d143e1d44bfbb5_4
c8c93ad432e9 7f92d556d4ff "/usr/bin/launch.sh" 3 minutes ago Up 3 minutes k8s_weave-npc_weave-net-lthlv_kube-system_eac41670-a119-4085-99e7-7cf08185deb7_0
e9946edd52ba 5f8cb769bd73 "kube-scheduler --au…" 3 minutes ago Up 3 minutes k8s_kube-scheduler_kube-scheduler-devops1ar01n02_kube-system_90280dfce8bf44f46a3e41b6c4a9f551_0
4ffe61f78cf5 a00c858e350e "/kube-vip start -c …" 3 minutes ago Up 3 minutes k8s_kube-vip_kube-vip-devops1ar01n02_kube-system_dd4d116d758ec63efaf78fc4112d63e6_0
7019afbd1497 0a41a1414c53 "kube-controller-man…" 3 minutes ago Up 3 minutes k8s_kube-controller-manager_kube-controller-manager-devops1ar01n02_kube-system_9375c16649f1cd963bdbc6e4125314fc_0
32035400ad9d c29e6c583067 "/usr/local/bin/kube…" 3 minutes ago Up 3 minutes k8s_kube-proxy_kube-proxy-v5nwr_kube-system_d0c1ce98-b066-4349-89e0-6113b8fa1708_0
And when I went back to node A to check the pods, kubectl command hang and then timeout:
root@devops1ar01n01:~# kubectl get pod -n kube-system
Unable to connect to the server: net/http: TLS handshake timeout
And I checked the containers, found that the kube-apiserver is always restarting:
root@devops1ar01n01:~# docker ps -a | grep -v pause | grep kube-apiserver
d5f85d72d2dc ae5eb22e4a9d "kube-apiserver --ad…" 8 seconds ago Up 8 seconds k8s_kube-apiserver_kube-apiserver-devops1ar01n01_kube-system_860fed4d3a137b129887eb23f07be1b6_6
a3bd40ba5552 ae5eb22e4a9d "kube-apiserver --ad…" About a minute ago Exited (1) About a minute ago k8s_kube-apiserver_kube-apiserver-devops1ar01n01_kube-system_860fed4d3a137b129887eb23f07be1b6_5
I ran docker logs <container_id> on node A to get the exited kube-apiserver container's log, output is as below:
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
I0226 06:22:58.186450 1 server.go:632] external host was not specified, using 172.16.80.3
I0226 06:22:58.187781 1 server.go:182] Version: v1.20.4
I0226 06:22:59.289621 1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0226 06:22:59.294839 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0226 06:22:59.294939 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0226 06:22:59.299670 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0226 06:22:59.299772 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0226 06:22:59.318732 1 client.go:360] parsed scheme: "endpoint"
I0226 06:22:59.318985 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 <nil> 0 <nil>}]
I0226 06:23:00.290273 1 client.go:360] parsed scheme: "endpoint"
I0226 06:23:00.290377 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 <nil> 0 <nil>}]
Error: context deadline exceeded
And systemctl status kubelet outputs below:
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Fri 2021-02-26 10:31:19 CST; 3h 58min ago
Docs: https://kubernetes.io/docs/home/
Main PID: 19602 (kubelet)
Tasks: 17
Memory: 67.0M
CPU: 20min 47.367s
CGroup: /system.slice/kubelet.service
└─19602 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --p
Feb 26 14:29:34 devops1ar01n01 kubelet[19602]: Trace[336098419]: [10.001217223s] [10.001217223s] END
Feb 26 14:29:34 devops1ar01n01 kubelet[19602]: E0226 14:29:34.695789 19602 reflector.go:138] object-"kube-system"/"kube-proxy-token-x5lsv": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "
Feb 26 14:29:36 devops1ar01n01 kubelet[19602]: E0226 14:29:36.056753 19602 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://kube-vip:16443/apis/coordination.k8s.
Feb 26 14:29:40 devops1ar01n01 kubelet[19602]: E0226 14:29:40.068403 19602 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"
Feb 26 14:29:40 devops1ar01n01 kubelet[19602]: E0226 14:29:40.068717 19602 event.go:218] Unable to write event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k
Feb 26 14:29:40 devops1ar01n01 kubelet[19602]: E0226 14:29:40.069126 19602 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "devops1ar01n01": Get "https://kube-vi
Feb 26 14:29:44 devops1ar01n01 kubelet[19602]: I0226 14:29:44.012843 19602 scope.go:111] [topologymanager] RemoveContainer - Container ID: 3ab3a85e785ae39f705ca30aad59a52ec17d12e9f31cbf920695d7af9cf93
Feb 26 14:29:44 devops1ar01n01 kubelet[19602]: E0226 14:29:44.031056 19602 pod_workers.go:191] Error syncing pod 860fed4d3a137b129887eb23f07be1b6 ("kube-apiserver-devops1ar01n01_kube-system(860fed4d3a
Feb 26 14:29:50 devops1ar01n01 kubelet[19602]: E0226 14:29:50.070777 19602 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"
Feb 26 14:29:50 devops1ar01n01 kubelet[19602]: E0226 14:29:50.072142 19602 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "devops1ar01n01": Get "https://kube-vi
I've tried kubeadm reset both nodes and repeat the operations, but this situation will always show up. How shall I debug this furthur?
Solution 1:
The problem is finally solved by myself. This is probably caused by a wrong health check url in the load balancer.
I followd this guide https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#kube-vip to create the load balancer for my HA cluster and I picked the way of kube-vip, which seemed to be the most convenient one.
If you use kube-vip, he health check url might be /healthz as described in the haproxy config file of the second method(https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#keepalived-and-haproxy). However, I found that the health check url is /livez in /etc/kubernetes/mainfest/kube-apiserver.yaml
I guess that it should be the wrong health check url in kube-vip which caused the health check failure, and then lead to continous restarting of kube-apiserver.
To verify this, in order to edit the health check url in the load balancer, I chose the second way to create load balancer(https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#keepalived-and-haproxy) and changed /healthz to /livez in /etc/haproxy/haproxy.cfg
Then I followd the guide to run kube init and kube join, it can work well.