Why do pods on a node that was recreated after being preempted get stuck in ContainerCreating?

Solution 1:

The issue experienced may rest outside of GKE and with Kubernetes itself. In the Kubernetes github issue page, we see “https://github.com/kubernetes/kubernetes/issues/62666#issue-314798540” which pertains to Pod deletion but has a similar error your received :

 “ spec: Forbidden: pod updates may not change fields other than `spec.containers[].image`, `spec.initContainers[].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)" 

If your experience a similar issue again, I’ll recommend opening a Private Issue here in here, and provide full error you might be getting, GKE version, etc.

By the way, when a node gets preempted, the nodes gets deleted and recreated, so a reset could not have the same impact.