Namespace "stuck" as Terminating, How I removed it

Solution 1:

Assuming you've already tried to force-delete resources like: Pods stuck at terminating status, and your at your wits' end trying to recover the namespace...

You can force-delete the namespace (perhaps leaving dangling resources):

(
NAMESPACE=your-rogue-namespace
kubectl proxy &
kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
)

• This is a refinement of the answer here, which is based on the comment here.

• I'm using the jq utility to programmatically delete elements in the finalizers section. You could do that manually instead.

• kubectl proxy creates the listener at 127.0.0.1:8001 by default. If you know the hostname/IP of your cluster master, you may be able to use that instead.

• The funny thing is that this approach seems to work even when using kubectl edit making the same change has no effect.

Solution 2:

This is caused by resources still existing in the namespace that the namespace controller is unable to remove.

This command (with kubectl 1.11+) will show you what resources remain in the namespace:

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

Once you find those and resolve and remove them, the namespace will be cleaned up

Solution 3:

As mentioned before in this thread there is another way to terminate a namespace using API not exposed by kubectl by using a modern version of kubectl where kubectl replace --raw is available (not sure from which version). This way you will not have to spawn a kubectl proxy process and avoid dependency with curl (that in some environment like busybox is not available). In the hope that this will help someone else I left this here:

kubectl get namespace "stucked-namespace" -o json \
  | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" \
  | kubectl replace --raw /api/v1/namespaces/stucked-namespace/finalize -f -

Solution 4:

Need to remove the finalizer for kubernetes.

Step 1:

kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json

• remove kubernetes from finalizers array which is under spec

Step 2:

kubectl replace --raw "/api/v1/namespaces/<YOUR_NAMESPACE>/finalize" -f ./<YOUR_NAMESPACE>.json

Step 3:

kubectl get namespace

You can see that the annoying namespace is gone.