Pods stuck in 'Pending', no events being logged

Solution 1:

Posting this answer to give more of a general idea for where to look for information why Pod is in Pending state as for now it's impossible to tell on this specific setup.

The ways to check why the Pod can be in Pending state:

• $ kubectl describe pod POD_NAME
• $ kubectl get events -A
• Inspecting the Cloud Logging (more on that below)

Assuming following situation where the Pod is in Pending state:

• $ kubectl get pods

NAME                           READY   STATUS    RESTARTS   AGE
nginx-four-99d88fccb-rwzmp     0/1     Pending   0          2s
nginx-one-8584c66446-h92rm     1/1     Running   0          5d22h
nginx-three-5bcb988986-tmshp   1/1     Running   0          5d22h
nginx-two-6c9545d7d4-2zlmh     1/1     Running   0          5d22h

To get more information about it's state you can run:

• $ kubectl describe pod POD_NAME

The Event part of above output:

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  26s (x2 over 114s)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.

As you can see there is an information on why the Pod is in Pending state (Insufficient CPU).

You can also run:

• $ kubectl get events

LAST SEEN   TYPE      REASON              OBJECT                            MESSAGE
20s         Warning   FailedScheduling    pod/nginx-four-99d88fccb-rwzmp    0/1 nodes are available: 1 Insufficient cpu.
14m         Normal    SuccessfulCreate    replicaset/nginx-four-99d88fccb   Created pod: nginx-four-99d88fccb-rwzmp
14m         Normal    ScalingReplicaSet   deployment/nginx-four             Scaled up replica set nginx-four-99d88fccb to 1

Disclaimer!

Kubernetes events are stored in the etcd for the 1 hour. If the message of Pod state was not repeating over time, it will be deleted after 1 hour. Additional reference on this particular topic:

• Github.com: Kubernetes: Issues: Events will disappear after one hour

Retrieving logs from Cloud Logging:

You can run below query to get the Pods that were in Pending state:

resource.type="k8s_cluster"
resource.labels.cluster_name="gke-serverfault"
protoPayload.response.status.phase="Pending"

This query will not show the reason (like Insufficient CPU) of why Pod is in Pending state. There is a feature request on Issuetracker.google.com for this reason. You can follow it to receive further updates:

• Issuetracker.google.com: Issue: GKE Cloud Logging Reason for Pod pending state

Additional resources:

• Cloud.google.com: Kubernetes Engine: Docs: Troubleshooting
• Cloud.google.com: Products: Operations
• Cloud.google.com: Stackdriver: Docs: Release notes