DNS Server Behind NAT

I've got a Bind 9 DNS server sitting behind a NAT firewall, assume the Internet facing IP is 1.2.3.4

There are no restrictions on outgoing traffic, and port 53 (TCP/UDP) is forwarded from 1.2.3.4 to the internal DNS server (10.0.0.1). There are no IP Tables rules on either the VPS or the internal Bind 9 server.

From a remote Linux VPS located elsewhere on the internet, nslookup works fine

# nslookup foo.example.com 1.2.3.4
Server:    1.2.3.4
Address:   1.2.3.4#53

Name:      foo.example.com
Addresss:  9.9.9.9

However, when using the host command on the remote VPS, I receive the following output:

# host foo.example.com 1.2.3.4
;; reply from unexpected source: 1.2.3.4#13731, expected 1.2.3.4#53
;; reply from unexpected source: 1.2.3.4#13731, expected 1.2.3.4#53
;; connection timed out; no servers could be reached.

From the VPS, I can establish a connection (using telnet) to 1.2.3.4:53

From the internal DNS server (10.0.0.1), the host command appears to be fine:

# host foo.example.com 127.0.0.1
Using domain server:
Name: 127.0.0.1
Address: 127.0.0.1#53
Aliases:

foo.example.com has address 9.9.9.9

Any suggestions as to why the host command on my VPS is complaining about the reply coming back from another port, and what can I do to fix this?

Further info:

From a windows host external to the network

>nslookup foo.example.com 1.2.3.4
DNS request timeout
    timeout was 2 seconds
Server: UnKnown
Address: 1.2.3.4

DNS request timed out.
    timeout was 2 seconds
DNS request timed out.
    timeout was 2 seconds
DNS request timed out.
    timeout was 2 seconds
DNS request timed out.
    timeout was 2 seconds
*** Request to UnKnown timed-out

This is a default install of bind from Ubuntu 12.04 LTS, with around 11 zones configured.

$ named -v
BIND 9.8.1-P1

TCP Dump (filtered) from internal DNS server

20:36:29.175701 IP pc.external.com.57226 > dns.example.com.domain: 1+ PTR? 4.3.2.1.in-addr.arpa. (45)
20:36:29.175948 IP dns.example.com.domain > pc.external.com.57226: 1 Refused- 0/0/0 (45)
20:36:31.179786 IP pc.external.com.57227 > dns.example.com.domain: 2+[|domain]
20:36:31.179960 IP dns.example.com.domain > pc.external.com.57227: 2 Refused-[|domain]
20:36:33.180653 IP pc.external.com.57228 > dns.example.com.domain: 3+[|domain]
20:36:33.180906 IP dns.example.com.domain > pc.external.com.57228: 3 Refused-[|domain]
20:36:35.185182 IP pc.external.com.57229 > dns.example.com.domain: 4+ A? foo.example.com. (45)
20:36:35.185362 IP dns.example.com.domain > pc.external.com.57229: 4*- 1/1/1 (95)
20:36:37.182844 IP pc.external.com.57230 > dns.example.com.domain: 5+ AAAA? foo.example.com. (45)
20:36:37.182991 IP dns.example.com.domain > pc.external.com.57230: 5*- 0/1/0 (119)

TCP Dump from client during query

21:24:52.054374 IP pc.external.com.43845 > dns.example.com.53: 6142+ A? foo.example.com. (45)
21:24:52.104694 IP dns.example.com.29242 > pc.external.com.43845: UDP, length 95

Solution 1:

To sum up what has been previously written in comments with some further explanation:

It looks a bit like your NAT rules for UDP were broken. An indication is the error message reply from unexpected source: 1.2.3.4#13731, expected 1.2.3.4#53 and your trace taken from the client where the response looks like dns.example.com.29242 > pc.external.com.43845: UDP, length 95. The source port for the response packet should be 53, it is correct in your dump taken from the DNS server (where resolves to domain for display purposes).

While some (especially historic) resolvers may accept DNS responses from different ports / IPs, most would not - mainly due to security reasons to impede DNS spoofing and cache poisoning attacks.

At any rate, for connectionless UDP NAT traffic, your router should preserve state data from the previously received UDP DNS query packet and re-map the IP:port tuple for the response packet back to 1.2.3.4:53 - which it apparently does not. It may be a configuration error or a bug in the way the router is handling the UDP state table for port forwarding cases - so your best bet would be to open a case with the manufacturer's customer support (having upgraded the code to the latest/greatest beforehand - such an issue is likely to have been noticed by other users previously and thus likely to already be fixed).