Resize instance types on Container Engine cluster

Some of our containers run better with memory above the instance type currently deployed in our Container Engine cluster. Is there a recommended practice to rebuild the container engine template for larger instances after the container engine cluster has been created?

For example, go from GCE instances n1-standard-2 to n1-highmem-8 for running containers with above 8GB RAM?

Container Engine doesn't currently have an API for doing this, but since it uses a Compute Engine instance group for the nodes in your cluster, you can actually update it without needing GKE's help. In the Developers Console, copy the instance template that looks like "gke--" and modify the machine type in it, then edit the similarly named instance group to use the new template. You can find these options under Compute > Compute Engine > Instance templates and Compute > Compute Engine > Instance groups, respectively.

A different approach would be:

(1) to create a new node-pool to the GKE cluster with vertically scaled machine types ...

$ gcloud container node-pools create pool-n1std2 --zone europe-west1-d --cluster prod-cluster-1 --machine-type  n1-standard-2  --image-type gci --disk-size=250 --num-nodes 3

(2) then, migrate the workloads off the old nodes ...

$ kubectl drain gke-prod-cluster-1-default-pool-f1eabad5-9ml5 --delete-local-data --force

(3) and finally, to delete the old node-pool

$ gcloud container node-pools delete default-pool --cluster=prod-cluster-1

Notes:

• Warning: Step 2 deletes node local volumes like emptyDir !!!
• Step 2 needs to be repeated for each node in the pool
• Instead of draining the nodes, one might configure a proper nodeSelector to schedule the pods onto the new pool. Label to be matched against would be cloud.google.com/gke-nodepool: pool-n1std2

• Create a new node pool with a custom machine type, disk-size, num-nodes and with any other required parameters.
• All steps are at Google Container Engine GKE node pools.