Internet connectivity for GKE nodes
I created a GKE cluster with the follwing command:
gcloud container clusters create experiment --num-nodes=1 --network default --subnetwork default --enable-private-nodes --enable-private-endpoint --enable-ip-alias --master-ipv4-cidr 172.16.0.16/28 --no-enable-basic-auth --no-issue-client-certificate
I have no egress rules in my VPC Firewall I have a autocreated default route under VPC routes that applies to the GKE nodes and allows internet access.
On the GKE node I can:
$ docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
bb79b6b2107f: Pull complete
111447d5894d: Pull complete
a95689b8e6cb: Pull complete
1a0022e444c2: Pull complete
32b7488a3833: Pull complete
Digest: sha256:ed7f815851b5299f616220a63edac69a4cc200e7f536a56e421988da82e44ed8
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest
docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
6a5697faee43: Pull complete
ba13d3bc422b: Pull complete
a254829d9e55: Pull complete
Digest: sha256:fff16eea1a8ae92867721d90c59a75652ea66d29c05294e6e2f898704bdb8cf1
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest
But I can't:
$ wget https://www.amazon.com
--2020-10-31 19:22:44-- https://www.amazon.com/
Resolving www.amazon.com... 13.226.21.44
Connecting to www.amazon.com|13.226.21.44|:443...
But I can:
$ wget https://www.google.com
--2020-10-31 19:23:15-- https://www.google.com/
Resolving www.google.com... 172.217.212.147, 172.217.212.99, 172.217.212.106, ...
Connecting to www.google.com|172.217.212.147|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html.1'
index.html.1 [ <=> ] 12.48K --.-KB/s in 0s
2020-10-31 19:23:15 (72.1 MB/s) - 'index.html.1' saved [12782]
route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.128.0.1 0.0.0.0 UG 1024 0 0 eth0
10.108.2.0 0.0.0.0 255.255.255.0 U 0 0 0 cbr0
10.128.0.1 0.0.0.0 255.255.255.255 UH 1024 0 0 eth0
169.254.123.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
Whats happening with internet connectivity on GKE nodes. I can reach docker hub but not www.amazon.com ? Little confused here.
Solution 1:
I created a GKE cluster with the follwing command:
gcloud container clusters create experiment --num-nodes=1 --network default --subnetwork default --enable-private-nodes --enable-private-endpoint --enable-ip-alias --master-ipv4-cidr 172.16.0.16/28 --no-enable-basic-auth --no-issue-client-certificate
By this command you created a private (--enable-private-nodes) GKE cluster.
The official documentation states:
In a private cluster, nodes only have internal IP addresses, which means that nodes and Pods are isolated from the internet by default.
-- Cloud.google.com: Kubernetes Engine: How to: Private clusters
By default you won't have access to site likes amazon.com, microsoft.com etc.
Your node connected successfully to google.com because of the Private Google Access:
VM instances that only have internal IP addresses (no external IP addresses) can use Private Google Access. They can reach the external IP addresses of Google APIs and services. The source IP address of the packet can be the primary internal IP address of the network interface or an address in an alias IP range that is assigned to the interface. If you disable Private Google Access, the VM instances can no longer reach Google APIs and services; they can only send traffic within the VPC network.
-- Cloud.google.com: VPC: Private access options
You can try to create 2 VM's with only internal IP address in 2 separate networks where one of the networks has PGA enabled. You should be able to communicate with google.com from the VM that resides in PGA enabled network (or enable/disable the PGA on a network that has a VM).
As a side note:
You can configure Cloud NAT to allow your private
GKEnodes to have access to the Internet:
- Cloud.google.com: NAT: Docs: GKE Example
As for pulling images:
Pulling container images from an image registry
In a private cluster, the container runtime can pull container images from Container Registry; it cannot pull images from any other container image registry on the internet. This is because the nodes in a private cluster do not have external IP addresses, so by default they cannot communicate with services outside of the Google network.
The nodes in a private cluster can communicate with Google services, like Container Registry, if they are on a subnet that has Private Google Access enabled.
The following commands create a Deployment that pulls a sample image from a Google-owned Container Registry repository:
$ kubectl run hello-deployment --image gcr.io/google-samples/hello-app:2.0Note: While Container Registry's Docker Hub mirror is accessible from a private cluster, it should not be exclusively relied upon. The mirror is only a cache, so images are periodically removed, and a private cluster is not able to fall back to Docker Hub.
Your ubuntu and nginx images were downloaded from a mirror accessible from private GKE nodes. If you try to download an image that is not in a mirror you will get the following error:
$ kubectl describe pod
Pulling image "MYOWNPUBLICREPO/ubuntu-test:latest"
Warning Failed 13m (x184 over 58m) kubelet, GKE-CLUSTER-POOL Error: ImagePullBackOff
Warning Failed 8m6s (x14 over 58m) kubelet, GKE-CLUSTER-POOL Failed to pull image "MYOWNPUBLICREPO/ubuntu-test:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
It's recommended to use the Container Registry with private GKE clusters. You can read more about it by following official documentation:
- Cloud.google.com: Container Registry