Why does my EC2 nameserver (correctly) resolve names to private IPs and then switches to public IPs?

I'm running a nameserver for a few machines on EC2. This nameserver is not reachable via the public Internet. I want all my EC2 instances to use it to be able to find each other and to resolve all public domain names (e.g. google.com). Using CNAMEs I'm supposed to be able to create my own names but defer to Amazon's internal EC2 nameserver for (internal) IP addresses. It works for a while, but eventually some names incorrectly resolve to their public IP addresses.

Here's what is supposed to happen:

foo$ host bar
bar.tld is an alias for ec2-88-88-88-88.compute-1.amazonaws.com.
ec2-88-88-88-88.compute-1.amazonaws.com has address 192.168.50.17

Here's what actually happens after a while:

foo$ host bar
bar.tld is an alias for ec2-88-88-88-88.compute-1.amazonaws.com.
ec2-88-88-88-88.compute-1.amazonaws.com has address 88.88.88.88

Ack! If I restart BIND, it goes back to correctly resolving the name to the private IP for a while, then eventually serves the public IP. It never seems to revert to serving the private IP again unless I restart BIND.

By the way, this always seems to work:

foo$ host ec2-88-88-88-88.compute-1.amazonaws.com 172.16.0.23
ec2-23-21-222-199.compute-1.amazonaws.com has address 192.168.50.17

If it seems like I don't know what I'm doing because I made a simple mistake in one of these config files, that's because I don't know what I'm doing.

I set up my own top-level domain: ".tld" (obfuscated--hopefully the real characters chosen for the top-level domain don't matter).

I tried to do what Eric Hammond recommends. Here's my /etc/bind/named.conf.options:

options {
  directory "/var/cache/bind";
  forwarders { 172.16.0.23; };
  auth-nxdomain no;
  listen-on-v6 { any; };
  allow-query { 127.0.0.1; 192.168.50.0/24; };
};

172.16.0.23 is the ip for Amazon's DNS server, only reachable from inside EC2. It is the only one that knows the correct internal/private IPs for my machines. It is almost as if my server is failing to resolve foo.tld sometimes, then goes off to some other [public] nameserver and the lookup is then borked (thereafter resolves to the public IP). The trick that I don't seem to be accomplishing is that my nameserver must never go beyond 172.16.0.23 to resolve these x.compute-1.amazonaws.com names. Another possibility is that sometimes Amazon's nameserver screws up and gives me a public IP, but then why would my nameserver not eventually correct the error and resolve the private IP again sometimes?

Anyway, moving on... here's /etc/bind/named.conf.local:

zone "tld" {
  type master;
  file "/etc/bind/db.tld";
  notify no;
};

For completeness sake, here's /etc/bind/named.conf:

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";

Here's my obfuscated bind database for my zone, /etc/bind/db.tld:

$TTL    1h
@       IN      SOA     tld. ns.tld. (
          2012021425         ; Serial
                  1m         ; Refresh
                  2m         ; Retry
                  1w         ; Expire
                  1h )       ; Negative Cache TTL
;
@       IN      NS      ns.tld.
@       IN      A       192.168.50.5
ns              A       192.168.50.5

foo     CNAME   ec2-99-99-99-99.compute-1.amazonaws.com.
bar     CNAME   ec2-88-88-88-88.compute-1.amazonaws.com.

Whew. Ok, here are a couple more files that are probably necessary for the whole picture. Here's /etc/resolv.conf on the nameserver, ns:

domain tld
search tld
nameserver 127.0.0.1

Here's /etc/resolv.conf on foo.tld and bar.tld:

domain tld
search tld compute-1.internal
nameserver 192.168.50.5
nameserver 172.16.0.23

I don't know if that's correct. Maybe I should only have nameserver 192.168.50.5?

This setup uses BIND 9.7.3 on Ubuntu Server 11.10.

(FYI: cross-posted at http://alestic.com/2009/06/ec2-elastic-ip-internal)

options {
    forward only;
    ...

From BIND ARM97:

forward This option is only meaningful if the forwarders list is not empty. A value of first, the default, causes the server to query the forwarders first — and if that doesn’t answer the question, the server will then look for the answer itself. If only is specified, the server will only query the forwarders.

In your case, as you actually need amazon DNS servers for .amazonaws.com only, you may also do:

options {
    forwarders { };
...
}

zone "amazonaws.com" {
    type forward;
    forwarders { 172.16.0.23; };
    forward only;
};