16.10 fail to resolve DNS
After upgrading my 16.04-installation to 16.10, I have trouble with DNS.
First I got problems a couple of times when connected to WiFi, while it worked on ethernet. Now it seems to work on WiFi also. Not sure why, and if it is in any way related to the problem I face now:
When connecting to a VPN host with Cisco Anyconnect VPN, it adds a line in '/etc/resolv.conf'. I understand that Ubuntu is now using systemd-resolve, and the man page says that there are three different modes for handling /etc/resolv.conf. My /etc/resolv.conf is not a symlink, and does not list 127.0.0.53 as a DNS server, so as far as I understand systemd-resolved should "read it for DNS configuration data". However, it does not seem to care about it.
dig
The strange thing (for me) is that dig host.customer.tld, returns a nice answer with an ANSWER SECTION showing the ip of the requested host, and it refers to the dns server added to /etc/resolv.conf by vpn client as the SERVER. When vpn connection is disabled I get no answer. I.e. dig reads /etc/resolv.conf.
ping
The browser, on the other side, does not get to /etc/resolv.conf, and is not able to resolve the host name. Neither is ping/curl, by the way.
nmcli
I found a related post, and tried running
nmcli device show <interfacename> | grep IP4.DNS
but it lists no dns for the cscotun0 device. (It does not in 16.04 neither, though.) Also, nmcli lists my dhcp server (my router) as IP4.DNS host for my eth/wlan connections. Using dig @192.168.0.1 xxx for any public domain works fine.
configuration
There are some other DNS servers listed in my /run/systemd/resolve/resolv.conf:
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
# Too many DNS servers configured, the following entries may be ignored.
nameserver 2001:4860:4860::8844
These are not served by my DHCP server. the file /etc/systemd/resolved.conf contains only commented lines, except the section header:
[Resolve]
#DNS=
#FallbackDNS=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844
The man page for resolved.conf says that
DNS= A space-separated list of IPv4 and IPv6 addresses to use as system DNS servers. ... For compatibility reasons, if this setting is not specified, the DNS servers listed in /etc/resolv.conf are used instead, if that file exists and any servers are configured in it. This setting defaults to the empty list.
FallbackDNS= A space-separated list of IPv4 and IPv6 addresses to use as the fallback DNS servers. Any per-link DNS servers obtained from systemd-networkd.service(8) take precedence over this setting, as do any servers set via DNS= above or /etc/resolv.conf. This setting is hence only used if no other DNS server information is known. If this option is not given, a compiled-in list of DNS servers is used instead.
Seems like the fallback ends up in /run/systemd/resolve/resolv.conf in my case.
EDIT: I was not certain what was the problem, and to be honest I still don't know exactly how this works, but at least it turned out that the solution in my case was to disable the systemd-resolved service. I thought that service was required, that it was the component that provided DNS service to all the local applications, but apparently there are something else in there doing that job.
Solution 1:
DNS behavior during OpenVPN connection improved immediately when I followed a suggestion on ubuntuforums:
- Open
/etc/NetworkManager/NetworkManager.confin an editor with root rights. - Delete (or comment out with a hash
#) the line that readsdns=dnsmasq - Restart NetworkManager via
sudo service NetworkManager restart
Solution 2:
I experienced similar problems, for example with adding an extra USB wifi dongle. First I disabled dnsmasq in networkmanager as described above and I stopped dnsmasq (service dnsmasq stop)
I noticed that when resolving broke during my VPN connecting, the routing table looks slightly different (output of route command). The name of the Gateway is DD-WRT in the case it does not work and simply 'gateway' when it does work. The output of this did not change:
nmcli device show wlp1s0 | grep IP4.DNS
It kept showing my router IP. A workaround to get it to work for a while is to restart systemd-resolvd:
sudo service systemd-resolved restart
Since dnsmasq is out of the equation, it is either systemd-resolvd that is the cause of the issue, or anything changing the routing table.
So this is the only difference I see:
ubuntu@ubuntu-Lenovo-Yoga-2-11:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 601 0 0
which works. And this when it does NOT work:
ubuntu@ubuntu-Lenovo-Yoga-2-11:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default DD-WRT 0.0.0.0 UG 601 0 0 wlp1s0
And the same name difference on the VPN line :
vpn-dns.name gateway 255.255.255.255 UGH 0 0 0 wlp1s0
Who knows what may influence the routing table? It would be great if we can identify this so a bug report can be filed. I am getting seriously sick and tired of pursuing all these bugs, but I would like to get them fixed so future users and us will be happy :).
[update] It seems stopping systemd-resolved may fix this and not negatively impact other stuff. You can try that and let it know if it does break stuff. I saw when running systemd-resolvd in debug when it broke:
Removing scope on link wlp1s0, protocol llmnr, family AF_INET
Removing scope on link wlp1s0, protocol llmnr, family AF_INET6
Removing scope on link *, protocol dns, family *
To disable:
sudo systemctl disable systemd-resolved.service
I updated the Ubuntu report with suggestions. [/update] Add: Note: the bug report : https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624317 has a patch for 17.04 for some issues. Please check the bug report and if possible test the patch. Thank you!
[update]
Please check the above mentioned bug report, the issue seems to be resolved for 17.10 and with a simple command DNS leakage can be disabled too.
[/update]