Why would 127.0.0.1 in resolv.conf cause problems in DNS resolution?

[update and update II: see my awkward and proper solutions at the end of the question, based on one of the answers]

I have an extremely slow DNS lookup when my resolv.conf file includes the 127.0.0.1 IP address, actually, many names do not get resolved at all as it (most certainly) times out.

I found one question with what looks like a valid answer here:

Extremely slow DNS lookup

except that I can see the the dnsmasq tool is running for the IP range 192.168.122.2 to 192.168.122.254. This looks like the IPs used by VirtualBox and qemu, so I would imagine that if I turn off dnsmasq, accessing the Internet from a virtual system will fail!

Would there be another reason for the slowness and/or timeout? (note that the problem is more prominent with specific systems such as the user pictures on all stackoverflow websites, including askubuntu, when the problem occurs.)

At this point I remove the IP from the resolv.conf file and that bypasses the issue while I'm browsing, but I don't think that's the best solution (and of course that IP is reinstalled there on each reboot!) I'd like a more permanent solution to this problem that still allows me to run the virtual systems as expected.

P.S. I do not run the Network Manager.

Contents of /etc/network/interfaces

Note that I had the problem before adding the 2nd IP on eth1 (i.e. eth1:0).

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth1
iface eth1 inet static
    address 162.226.130.121
    netmask 255.255.255.248
    network 162.226.130.120
    broadcast 162.226.130.127
    gateway 162.226.130.126

auto eth1:0
iface eth1:0 inet static
    name Local network
    address 192.168.1.1
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.254

# bridge for virtual box
auto br0
iface br0 inet static
    address 192.168.2.1
    netmask 255.255.255.0
    network 192.168.2.0
    broadcast 192.168.2.255
    bridge_ports    eth3
    bridge_stp      off
    bridge_maxwait  0
    bridge_fd       0

Contents of the /etc/resolv.conf (which is still a soft link as expected) after a boot:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
search m2osw.com
nameserver 192.168.122.1
nameserver 206.13.31.12
nameserver 206.13.28.12

Update:

I do not write an answer myself because I very much used another answer here to resolve my issue, although it is really not what I wanted to do, it is the easiest solution at this point. The bug referenced by the resolvconf author below, found here:

https://bugs.launchpad.net/ubuntu/+source/bind9/+bug/933723

clearly states that if you want to use bind9, you WILL get the namespace 127.0.0.1 in your resolv.conf file. No choice. (RESOLVCONF=no does not seem to do anything, although I may have a surprise after a reboot which I will do very soon!)

As a side note: If I point the /etc/resolvconf/resolv.conf.d/tail softlink to /dev/null (as mentioned below) then I get a resolv.conf that looks like this:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1

In other words, I still get the 127.0.0.1 which is the culprit at this point. But also I completely lose the other two nameservers, even though they are clearly specified in my eth1 interface. This being said, looking in the /run/resolvconf/interface/ directory, I see an eth1.net and lo.named files. The lo.named is copied to the resolv.conf, but not the eth1.net. I have no clue how those files get created and then copied to resolv.conf but that is the dynamism used by resolvconf...

Anyway, there are two solutions mentioned in the bug:

(1) Delete the /etc/resolv.conf soft link and replace it with a plain file with exactly what you want. This may work for you, but I thought that keeping the default dynamic file would be best.

(2) Change your bind9 settings so BIND can respond to those local DNS requests. The interesting thing with that is that names will now be cached on your computer. So it is not all bad. This being said, I was not too interested in opening my nameserver to all domain names... But this works and I do not have to destroy the dynamic resolv.conf.

Just in case, here is an example setup for the bind to make that work:

options {
    [...]
    forwarders {
          // Google DNSes
          8.8.8.8;
          8.8.4.4;
    };
    [...]
};

Update II:

Okay! The reboot got rid of the lo.named file in the /run/resolvconf/interface directory. That must be because bind knows to create it if RESOLVCONF=yes, but it does not delete it if RESOLVCONF=no. However, a reboot takes care of that because the /run directory is a RAM disk.

Without that file in the way, the /etc/resolv.conf is setup with exactly what I'd expect which is the list of nameservers as defined in my eth1 definition found in /etc/network/interface as shown earlier.

So... it's a big mess because there are many factors and in some cases a reboot is required!

Solution 1:

1: Add dns-nameservers and dns-search options to /etc/network/interfaces.

auto eth1
iface eth1 inet static
    address 162.226.130.121
    netmask 255.255.255.0
    gateway 162.226.130.126
    dns-nameservers 8.8.8.8 162.226.130.126
    dns-search m2osw.com

2: Remove all dns- options from files in /etc/resolvconf/resolv.conf.d/. That resolv.conf includes nameserver options after nameserver 127.0.0.1 indicates that such cruft is present. If /etc/resolvconf/resolv.conf.d/tail is a symbolic link, make it a symbolic link to /dev/null.

3: Downup eth1.

sudo ifdown eth1
sudo ifup eth1

4: Look in /etc/resolv.conf. Is nameserver 127.0.0.1 still there and are replies to DNS queries still delayed? If so then figure out where the nameserver 127.0.0.1 line is coming from. Something is registering the listen address 127.0.0.1 without starting a local nameserver at 127.0.0.1. (i) One possibility is the bind9 package. If you aren't running a local BIND nameserver then purge the bind9 package (sudo apt-get purge bind9). If you are running a BIND nameserver that doesn't provide general Internet name service then edit /etc/default/bind9 and set RESOLVCONF=no, then restart the nameserver. See: https://bugs.launchpad.net/ubuntu/+source/bind9/+bug/933723 (ii) Another possibility is that you have remnants of dnsmasq or similar package on the system. Purge that package. Also purge network-manager since you aren't using it.

5: Reboot and see if things have improved, then report back here.

Solution 2:

Using a dns server with a loopback address (e.g. 127.0.0.1) causes problems:

All other dns server with a lower priority are ignored by resolvconf.

The priority of dns servers is defined by the network interfaces with which the dns-server is defined.

See /etc/resolvconf/interface-order and man 5 interface-order.

Luckily there is an environment variable to change this behaviour:

TRUNCATE_NAMESERVER_LIST_AFTER_LOOPBACK_ADDRESS

See man 8 resolvconf

If put

TRUNCATE_NAMESERVER_LIST_AFTER_LOOPBACK_ADDRESS=no

in /etc/default/resolvconf and restart the resolvconf service all other dns server will show up in /etc/resolv.conf.