Why does kubectl port-forward require the destination service to be bound to localhost?

I have noticed that kubectl port-forward only works when the process being forwarded to is listening on 127.0.0.1 or localhost. If the process is bound to another address, like the pod IP, then port forwarding doesn't work.

For example, suppose I have a program that opens a server socket on the pod IP and port 8080, then this port forward does not work:

kubectl port-forward hasura-console-5d48cdb974-sv4l8 8080:8080

When I try to use it, its logs the following:

E0314 21:11:33.613318   93858 portforward.go:400] an error occurred forwarding 8080 -> 8080: error forwarding port 8080 to pod 9902636a8681f668702476c9e4148d6e4cd63ba20f4b3270fdc3c57d9ef9cb97, uid : exit status 1: 2021/03/15 01:11:34 socat[1486564] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused

When I restart the pod with the service listening on 127.0.0.1, the port forward works and the above error is not logged.

Do services running in pods need to listen on localhost?

I have noticed that kubectl port-forward only works when the process being forwarded to is listening on 127.0.0.1 or localhost. If the process is bound to another address, like the pod IP, then port forwarding doesn't work.

The unit that has an IP address in Kubernetes is a Pod. A Pod can contain one or more containers and they can expose Ports. You can connect to these ports using:

kubectl port-forward <pod-name> <your-local-port>:<container-port>

To see what port a container in a Pod (your pod name as example) is exposing you can run this command:

kubectl get pod hasura-console-5d48cdb974-sv4l8 --template='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'

Note that kubectl port-forward only works with TCP.

See also Use Port Forwarding to Access Applications in a Cluster

What address should the main process in a container listen to?

When creating processes to run within a container in Kubernetes, it must listen to 0.0.0.0. See e.g. this answer on Stackoverflow regarding what IP to listen to.

To clarify the original question if still needed:

Do services running in pods need to listen on localhost?

They do not need to listen on localhost, but using kubectl port-forward without specifying the option --address and accessing the service from where you are running this command will set 127.0.0.1 as the destination when you access 'your' localhost (the default behaviour as per the official documentation).

After running it, kubectl port-forward then uses the Kubernetes API to tunnel the TCP traffic from your local machine to your Pod and the ports provided in the command.

That is why it works when listening on localhost, but not on the Pod's IP, as you are not accessing it directly.

Bear in mind that Pod IPs are meant to change as they get recreated and won't usually be accessible directly due to being mostly for internal use within the Kubernetes cluster, so it makes sense for the service to listen to 0.0.0.0 if you want it to be reached from the outside when accessing the cluster's IP.

Accessing the Pod directly via its current ephemeral IP is usually reserved for debugging or testing purposes and not meant for production.