Don't log stack-traces from kubelet

I've got something wrong with my Kubernetes node, but it's hard to debug because I get pages and pages of stack trace like

Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000040f
Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: created by k8s.io/kubernetes/pkg/kubelet/config.newSourceApiserverFromLW
Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: goroutine 244 [select]:
Nov 13 10:55:51 corona kubelet[29656]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.contextForChannel.func
Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.contextForC
Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: goroutine 205 [sync.Cond.Wait]:
Nov 13 10:55:51 corona kubelet[29656]: runtime.goparkunlock(...)
Nov 13 10:55:51 corona kubelet[29656]:         /usr/local/go/src/runtime/proc.go:312
Nov 13 10:55:51 corona kubelet[29656]: sync.runtime_notifyListWait(0xc000b140c8, 0x1)
Nov 13 10:55:51 corona kubelet[29656]:         /usr/local/go/src/runtime/sema.go:513 +0xf8
Nov 13 10:55:51 corona kubelet[29656]: sync.(*Cond).Wait(0xc000b140b8)
Nov 13 10:55:51 corona kubelet[29656]:         /usr/local/go/src/sync/cond.go:56 +0x9d
Nov 13 10:55:51 corona kubelet[29656]: k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000b140
Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop(0
Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc
Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000def
Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000defe
Nov 13 10:55:51 corona kubelet[29656]:         /workspace/anago-v1.19.4-rc.0.51+5f1e5cafd33a88/src/k8s.io/kubernetes/_ou
Nov 13 10:55:51 corona kubelet[29656]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)

I tried adding -v=0 to the kubelet command in the systemd unit file, but it's still spewing. I looked at /var/lib/kubelet/config.yaml and it says logging: {}. I'm not sure if there's something I could put in there to make it quieter.

Is there any way to tell kubelet to skip the stack trace? The error messages before the trace are helpful but hard to find in the noise.

Solution 1:

First check it there aren’t any config files or environment variables that are used for running kubelet.

Best way to check:

1. check kubelet process (e.g. ps -ef | grep kubelet) to see all the args

2. if kubelet is using dynamic config check ConfigMap kubelet-config-1.18 in Kube-system to see if cgroupDriver is specified as you desire 3. if you run kubelet using systemd, then you could use the following method to see kubelet's logs:

# journalctl -u kubelet

See: system-component-logs.

See example how to narrow the scope of logs: managing-logs-kubernetes.

Error failed to run Kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd" is due to the Cgroup driver used in the Kubelet. Either the Kubelet and Docker should run using Systemd driver or with the Cgroupfs. Systemd is recommended.

Check on the worker nodes file /var/lib/kubelet/kubeadm-flags.env and in KUBELET_KUBEADM_ARGS if you have --cgroup-driver=cgroupfsflag. Changed it to systemd and kubelet will start working again.

Take a look: kubelete-cgroupfs-driver, kubelet-cgroupfs.

See also: kubelet-runtime-panic.