How to call a service exposed by a Kubernetes cluster from another Kubernetes cluster in same project

I can think of a couple of ways to access services across multiple clusters connected to the same GCP private network:

1. Bastion route into k2 for all of k2's services:

   Find the SERVICE_CLUSTER_IP_RANGE for the k2 cluster. On GKE, it will be the servicesIpv4Cidr field in the output of cluster describe:

   $ gcloud beta container clusters describe k2
   ...
   servicesIpv4Cidr: 10.143.240.0/20
   ...
   

   Add an advanced routing rule to take traffic destined for that range and route it to a node in k2:

   $ gcloud compute routes create --destination-range 10.143.240.0/20 --next-hop-instance k2-node-0
   

   This will cause k2-node-0 to proxy requests from the private network for any of k2's services. This has the obvious downside of giving k2-node-0 extra work, but it is simple.

2. Install k2's kube-proxy on all nodes in k1.

   Take a look at the currently running kube-proxy on any node in k2:

   $ ps aux | grep kube-proxy
   ... /usr/local/bin/kube-proxy --master=https://k2-master-ip --kubeconfig=/var/lib/kube-proxy/kubeconfig --v=2
   

   Copy k2's kubeconfig file to each node in k1 (say /var/lib/kube-proxy/kubeconfig-v2) and start a second kube-proxy on each node:

   $ /usr/local/bin/kube-proxy --master=https://k2-master-ip --kubeconfig=/var/lib/kube-proxy/kubeconfig-k2 --healthz-port=10247
   

   Now, each node in k1 handles proxying to k2 locally. A little tougher to set up, but has better scaling properties.

As you can see, neither solution is all that elegant. Discussions are happening about how this type of setup should ideally work in Kubernetes. You can take a look at the Cluster Federation proposal doc (specifically the Cross Cluster Service Discovery section), and join the discussion by opening up issues/sending PRs.

GKE now supports Internal Load Balancers: https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing

It's primary use case is to have a load balancer that's not exposed to the public internet so a service running on GKE can be reached from other GCE VMs or other GKE clusters in the same network.