GCP Load Balancer - Instance Failing - Debug?

Over the weekend, the GCP LB flagged one of our instances as unheathly. Nothing had changed, no redeployments and the other instances are fine.

I'm using a TCP healthcheck, the following turns 0 as expected.

nc localhost 80 -z; echo $?

How do I debug why GCP considers a node unhealthy?

UPDATE:

I changed the healthcheck to SSL and then back to TCP80 and now Google considers the node healthy. This is a little concerning.

The most common cause of health check failure is to bind a service only to the instance's external IP address and not on the load balancer's address.

You need to make your server process listen to all the configured addresses on the host. That way the health check will be operational for the load balancer’s external address.

You will find extensive info on how to debug health-checks in load balancing on Google cloud Platform here.

In my case it was because of firewall rules. The instance is in a custom subnet in a shared VPC and needed manually adding the load balancer's and health checker's IP addresses as allowed (as described in the documentation).