Why can't I access Metadata Server of GCP Instance?

Solution 1:

Although I use my own VPC with custom firewall, I don't think this is the issue of firewall because according to Google Metadata Server traffic never leaves the Instance.

GCP metadata traffic never leaves the physical host running the instance. Such requests go out the guest's interfaces, but never get forwarded.

A host level firewall in the instance can prevent traffic from reaching the metadata server. As it drops packets in the instance's network stack before they leave the guest.

Normally the metadata service responds to http, DNS, and ICMP echo, and all of them are not working for you. Only one hop away so not like it can be routed incorrectly. Sure sounds like a firewall.

Solution 2:

This can be caused by disabling the service account for the instance. You need authorization to access metadata.

In the Google Cloud Console:

• Go to Compute Engine -> VM instances.
• Stop the instance.
• Click edit and scroll down to "Service account".
• Choose the correct service account, usually "Compute Engine default service account".
• Under Access scopes, choose your desired configuration.

I recommend either "Allow default access" or "Allow full access to all Cloud APIs". The real access control is determined by the roles assigned to the service account. The "Access scopes" selection can only limit these roles and never increase them.

Next, get rid of your custom entry for metadata in your hosts file.