DNS issues on Google compute engine

I'm experiencing some interesting network issues, which I can reproduce and work around, although not explain - therefore I ask you! :)

I have a setup with two servers, both located in Google's network:

• Server A (Google App Engine): a public-facing web server, serving the website
• Server B (Google Compute Engine): scraping server, taking HTML images of Server A

A few days ago I started experiencing many timeout errors on requests from Server B to Server A. In most cases it could not connect to Server A, even though Server A is fully available from any other IP address.

I ssh'd into Server B and have these findings:

• I can do a curl www.ServerA.dk approx. 3 times within a few seconds, and then the connection dies, and I have to wait approx. 20 seconds, before I can do more curls

• I can with no problems curl any other URL as many times as I want

• The ping time to Server A is ridiculously low (< 1 ms)

I can solve the problem by:

• Changing the DNS in /etc/resolv.conf to Googles Public DNS

Or:

• By calling Server A with the default Google URL (ServerA.appspot.com) instead of my Danish domain www.ServerA.dk

I have a feeling this is a DNS issue on Google's behalf or my Danish DNS provider. Do you have any clue?

Actual domains

• www.kobstaden.dk (Custom domain)
• kobstaden-hrd.appspot.com (Default Google domain)

Some (random) DNS info:

From my DNS provider i have setup:

• C-Name for www.kobstaden.dk points to ghs.google.com

Cheers!

Solution 1:

Here's a best-guess answer to this. Many large DNS providers will have their DNS servers multihomed - i.e. the same IP address will send you to the server that's closest to you (in terms of network topology, not geography).

If one of the servers responding to the multihomed IP is malconfigured, then any request from a client topologically close to it may get a wrong answer or no answer. So by changing DNS servers you may get one that's topologically closer to you, and by using a different domain name you will also get to a different server which may respond better.