dig & nslookup can't resolve local hosts unless I specify the (local) DNS server to use

I am setting up a small private network within my University network. I have a Centos 6 box (sun) sitting between my private network and the university WAN. eth0 on the sun is connected to the WAN and eth1 to my private network via a netgear smart switch. sun acts as a router for the private network, forwards traffic from eth1 to eth0 using NAT configured using iptables. The clients on the private network (of which there is just at the moment, mercury) are assigned an IP, gateway etc and hostname via dnsmasq running on the sun. dnsmasq is configured to send a specific IP and hostname to the MAC address(es) of the client(s).

I have hardcode the hostname/IPs in /etc/hosts on sun only:

# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.42.1    sun sun.beowulf.iecs
192.168.42.2    mercury mercury.beowulf.iecs

At the moment, the IP address of eth0 (WAN facing) is assigned via DHCP (sun eth0 gets assigned the same IP all the time regardless; this is how the University does things but I'm in the process of getting this changed to allow me to assign a static IP). My /etc/resolv.conf on sun is:

; generated by /sbin/dhclient-script
search biol.uregina.ca
nameserver 142.3.102.202
nameserver 142.3.100.15

which seems to be being overwritten when I reboot so I have the following /etc/dnsmasq-resolv.conf

search beowulf.iecs biol.uregina.ca
nameserver 127.0.0.1
nameserver 8.8.8.8

and have instructed dnsmasq to use it instead of /etc/resolv.conf

# Change this line if you want dns to get its upstream servers from
# somewhere other that /etc/resolv.conf
resolv-file=/etc/dnsmasq-resolv.conf

# By  default,  dnsmasq  will  send queries to any of the upstream
# servers it knows about and tries to favour servers to are  known
# to  be  up.  Uncommenting this forces dnsmasq to try each query
# with  each  server  strictly  in  the  order  they   appear   in
# /etc/resolv.conf
strict-order

Mostly this setup is working. The problem I have (and I'm not sure it is a major one, but...) is that nslookup and dig both fail to resolve names for sun and mercury unless I inform these commands which DNS server to query:

# nslookup sun.beowulf.iecs sun.beowulf.iecs
Server:     sun.beowulf.iecs
Address:    192.168.42.1#53

Name:   sun.beowulf.iecs
Address: 192.168.42.1

# nslookup sun sun.beowulf.iecs
Server:     sun.beowulf.iecs
Address:    192.168.42.1#53

Name:   sun
Address: 192.168.42.1

# nslookup sun
Server:     142.3.102.202
Address:    142.3.102.202#53

** server can't find sun: NXDOMAIN

# nslookup sun.beowulf.iecs
Server:     142.3.102.202
Address:    142.3.102.202#53

** server can't find sun.beowulf.iecs: NXDOMAIN

The same output is given for mercury instead of **sun*. Representative dig output is:

# dig @192.168.42.1 mercury.beowulf.iecs

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> @192.168.42.1 mercury.beowulf.iecs
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65090
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;mercury.beowulf.iecs.      IN  A

;; ANSWER SECTION:
mercury.beowulf.iecs.   0   IN  A   192.168.42.2

;; Query time: 0 msec
;; SERVER: 192.168.42.1#53(192.168.42.1)
;; WHEN: Wed Jun 25 12:05:31 2014
;; MSG SIZE  rcvd: 54

and not working when I don't specify the nameserver to use:

# dig mercury.beowulf.iecs

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> mercury.beowulf.iecs
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 29153
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;mercury.beowulf.iecs.      IN  A

;; AUTHORITY SECTION:
.           7988    IN  SOA a.root-servers.net. nstld.verisign-grs.com. 2014062500 1800 900 604800 86400

;; Query time: 0 msec
;; SERVER: 142.3.102.202#53(142.3.102.202)
;; WHEN: Wed Jun 25 12:05:37 2014
;; MSG SIZE  rcvd: 113

In the last example above (dig) the SERVER: 142.3.102.202#53(142.3.102.202) is the first DNS server in /etc/resolv.conf so by default sun seems to be using this nameserver instead of dnsmasq that I would have expected it to use because of the settings in /etc/dnsmasq.conf.

So, is this a problem? I can ping sun and mercury from sun, and connect to mercury from sun using SSH.

If this is a problem (or even if it isn't), is there a way to configure sun and dnsmasq running on it to resolve the private host names automatically?

Solution 1:

dig (domain information groper) and nslookup (query Internet name servers interactively) are tools that query name servers. Unless a specific name server is specified as a commandline argument they will query the name server(s) found in /etc/resolv.conf. They simply don't look at alternative sources of host information such as the /etc/hosts file or other sources specified in /etc/nsswitch.conf.

If you want to force all dns queries through dnsmasq on your sun host, the /etc/resolv.conf there should point to dnsmasq, i.e. it should look like:

#/etc/resolv.conf on sun
nameserver 127.0.0.1

To prevent that file from getting overwritten when you restart the network interface eth0 edit /etc/sysconfig/network-scripts/ifcfg-eth0 and add the option PEERDNS=no

Second in /etc/dnsmasq-resolv.conf you're trying to configure dnsmasq to use itself as the upstream name server nameserver 127.0.0.1... That file should look like:

#/etc/dnsmasq-resolv.conf
search beowulf.iecs biol.uregina.ca
nameserver 8.8.8.8

if you want to use Google's name server. It might be a good idea to use the University name servers 142.3.102.202 & 142.3.100.15 there instead as it is not uncommon to have certain resources only visible from the campus network.

If your mercury host is configured by DHCP it should get it's configuration from dnsmasq and the /etc/resolv.conf there points to the dnsmasq name server on 192.168.42.1

Solution 2:

The 142.3.102.202 and 142.3.100.15 do not know anything about your local private network so they correctly return NXDOMAIN. These are also the default name servers for the system so when you don't specify a name server they will be used.

You will need to overwrite the contents of /etc/resolv.conf and configure the nameserver directives to point to your local dnsmasq name server which in turn should be configured to forward queries it cannot answer upstream.