minikube/k8s/kubectl "failed to watch file [ ... ]: no space left on device"

Solution 1:

It sounds like the node has run out of inotify watches.

You can check or set the number of available watches using the sysctl fs.inotify.max_user_watches on each Kubernetes node.

You will probably find it has been set to some ridiculously low number; by default it is autotuned based on the amount of memory the system has.

[root@small ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

[root@large ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 1048576

Whatever the amount is currently, double it until the problem stops occurring.

(Use minikube ssh to access a Minikube node.)