Run 'kubectl' commands from my localhost to GKE - but via tunnelling through a bastion host
Currently...
I have a GKE/kubernetes/k8s cluster in GCP. I have a bastion host (Compute Engine VM Instance) in GCP. I have allowlisted my bastion host's IP in the GKE cluster's Master authorized networks section. Hence, in order to run kubectl commands to my GKE, I first need to SSH into my bastion host by running the gcloud beta compute ssh command; then I run the gcloud container clusters get-credentials command to authenticate with GKE, then from there I can run kubectl commands like usual.
Later...
I want to be able to run kubectl commands to my GKE cluster directly from my local development CLI. In order to do that, I can add my local development machine IP as an allowlisted entry into my GKE's Master authorized networks, and that should be it. Then i can run the gcloud container clusters get-credentials first and then run kubectl commands like usual.
However...
I am looking for a way to avoid having to allowlist my local development machine IP. Every time i take my laptop somewhere new, i have to update the allowlist my new IP from there before i can run the gcloud container clusters get-credentials command before running kubectl commands.
I wonder...
Is there a way to assign a port number in the bastion-host that can be used to invoke kubectl commands to the remote GKE cluster securely? And then, i can just use the gcloud compute start-iap-tunnel command (which BTW takes care of all permission issues using Cloud IAM) from my local dev CLI to establish a ssh-tunnel to that specific port number in the bastion host. That way, for the GKE cluster, it is receiving kubectl commands from the bastion host (which is already allowlisted in its Master authorized networks). But behind the scene, i am authenticating with the bastion host from my local dev CLI (using my glcoud auth credentails) and invoking kubectl commands from there securely.
Is this possible? Any ideas from anyone?
Adding your IP to the Master authorized network would be easier. You can write a script which gets your laptop external IP and adds it to GKE's master authorized network list and use the same script to remove the IP once done. But since this doesn't seem to be your preference, let me give a long answer to how you can accomplish this with a jump host.
First, you will need a port redirector program on the bastion host. This will forward requests hitting bastion port to be redirected to the GKE master IP address. I am assuming here you are using a private cluster - both nodes and master api server are in private network.
On bastion host -
sudo apt-get update && sudo apt-get install redir -y
redir --laddr=0.0.0.0 --lport=8443 --caddr=172.16.0.32 --cport=443 -l debug
Above command will redirect requests on bastion port of 8443 to GKE master node (172.16.0.3) - feel free to change this based on your setup.
On your laptop -
gcloud compute ssh bastion --zone $ZONE --ssh-flag="-L 8443:localhost:8443"
You have now created an ssh tunnel from your laptop to bastion host. Calls to localhost on port 8443 will be redirected to the GKE api server.
Generate kube config -
gcloud container clusters get-credentials $CLUSTER_NAME [--zone $ZONE | --region $REGION]
Finally, update the server section in ~/.kube/config as follows -
server: https://127.0.0.1:8443
If you run the kubectl commands now, you will still encounter ssl certificate error -
$ kubectl version --short
Client Version: v1.15.11-dispatcher
Unable to connect to the server: x509: certificate is valid for ***, not 127.0.0.1
To verify the setup is working, you can skip tls verifiction with --insecure-skip-tls-verify. NOTE - skipping TLS verification is NOT recommended for security reasons.
$ kubectl version --short --insecure-skip-tls-verify
Client Version: v1.15.11-dispatcher
Server Version: v1.15.12-gke.2
One way you can address the certificate verification issue is, to create an alias IP address on your laptop which matches the GKE api server private address. In addition to this, update your kube config to match this.