Pods stuck in Terminating status

I tried to delete a ReplicationController with 12 pods and I could see that some of the pods are stuck in Terminating status.

My Kubernetes cluster consists of one control plane node and three worker nodes installed on Ubuntu virtual machines.

What could be the reason for this issue?

NAME        READY     STATUS        RESTARTS   AGE
pod-186o2   1/1       Terminating   0          2h
pod-4b6qc   1/1       Terminating   0          2h
pod-8xl86   1/1       Terminating   0          1h
pod-d6htc   1/1       Terminating   0          1h
pod-vlzov   1/1       Terminating   0          1h

You can use following command to delete the POD forcefully.

kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>

Force delete the pod:

kubectl delete pod --grace-period=0 --force --namespace <NAMESPACE> <PODNAME>

The --force flag is mandatory.


The original question is "What could be the reason for this issue?" and the answer is discussed at https://github.com/kubernetes/kubernetes/issues/51835 & https://github.com/kubernetes/kubernetes/issues/65569 & see https://www.bountysource.com/issues/33241128-unable-to-remove-a-stopped-container-device-or-resource-busy

Its caused by docker mount leaking into some other namespace.

You can logon to pod host to investigate.

minikube ssh
docker container ps | grep <id>
docker container stop <id>