How is the FQDN determined?

I'm using Ubuntu, and managing my hosts using Puppet. Until a machine or two ago, my hosts resolved to hostname.backend.example.com. The last two machines resolve to hostname.staging.internal. I'm a bit perplexed as to how those names were arrived at.

# /etc/resolv.conf
nameserver 173.203.4.8
nameserver 173.203.4.9

domain backend.example.com

And

# /etc/hosts
10.182.230.38  web01.staging.internal web01
127.0.0.1     localhost localhost.localdomain

/bin/hostname replies like this:

# hostname -a
web01

# hostname -A
108-166-97-91.static.cloud-ips.com web01.staging.internal 

# hostname -I
108.166.97.91 10.182.230.38

There exists a DNS record that maps 108.166.97.91 to web01.backend.example.com. Why, where and how did hostname pick up web01.staging.internal rather than the public DNS record?

The hostname command returns results from DNS and /etc/hosts.

hostname is equivilant to uname -n and is the actual "hostname" or "nodename" of the box.
All the other hostname arguments use this nodename to look up info.

So before going any further, I should explain the /etc/hosts file format.
The first field is fairly obvious, its the IP address all the hostnames on the line should resolve to. The second field is the primary hostname for that IP. The remaining fields are aliases.

So if you run hostname -f it will first try to resolve the IP for your nodename. Depending on how you have the hosts: entry configured in /etc/nsswitch.conf this method will vary.

• If you have it configured to use dns, it will use the search domains configured in /etc/resolv.conf until it gets an IP back from DNS.
• If you have it configured to use files it will look in /etc/hosts to find a line where either the primary hostname or the alias name is your current nodename (uname -n), and then return the primary hostname in that line.

Once it has the IP it will then try a reverse lookup on that IP. Again it will use DNS for this and your hosts file based on your nsswitch.conf. In the case of using your hosts file, it will return the primary entry (which is the first field after the IP in the file).

hostname -a will only work with the hosts file since doing a reverse lookup in DNS only gives you 1 result. With the hosts file it return the alises in the matching line (which is everything after the first entry, the primary hostname).

Examples
If your nodename is 'foobar', and you have an entry in /etc/hosts such as this:

127.0.0.1 foobar.example.com foobar localhost.localdomain localhost

Then you will get the following command results:

# hostname
foobar
# uname -n
foobar

# hostname -f
foobar.example.com

# hostname -a
foobar localhost.localdomain localhost