Ubuntu 18.04 systemd-resolved error NXDOMAIN
Solution 1:
The shortest and simplest answer is this:
These 'errors' in your syslog are generated as a result of the workaround identified in DVE-2018-0001, indicating that the system is attempting to mitigate further cases of DVE-2018-0001 violations by retrying the same request without EDNS extensions. This does not indicate an error you yourself can fix, nor does it mean there's anything to fix in this case.
It simply indicates that the systemd-resolved service is attempting to retry the same query to try and prevent further issues of DVE-2018-0001 which can be caused by captive portals not supporting EDNS0 extensions.
There is no further action needed by you in this case, this is more 'informational' information provided to the syslog by systemd-resolved.
However... if you want to know a little more about what exactly is going on here, then read on further, I provide a summary of this and some example data to show you exactly what's going on behind the scenes of these messages.
The Details (if you care to read them):
The error you are seeing is not actually an error message you have to act upon, and the statement of "Server returned error" is misleading. Rather, this is a notice that we're following a workaround defined in DNS Violation DVE-2018-0001.
This is an example of the log itself from my computer here:
Feb 5 13:29:17 overlord systemd-resolved[857]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Per DVE-2018-0001, it was discovered that, in Ubuntu 17.10 and later:
When trying to connect and authorise with the captive portal at securelogin.arubanetworks.com, at some point it fails to resolve correctly and thus users are failing to pass captive portal authorisation and access internet.
Further, this goes on to explain the problem:
Originally this was reports on Ubuntu bug tracker at https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1727237
This might be a bug in Ubuntu/systemd-resolved and/or securelogin.arubanetworks.com DNS spoofing/captivity and/or both.
From packet capture it appears that DNS query with EDNS0 DO (DNSSEC OK) bit set to zero, is responded to with NXDOMAIN.
Requests without EDNS0, are responded to with the correct portal IP address.
What this means is that there are certain responses going to the captive portals here that do not properly support EDNS0 queries; and that this needs to be adjusted by manufacturers.
The current workaround implemented in systemd-resolved within Ubuntu is what is generating the informational messages you see. This is in conjunction with the proposed workaround in the DVE (bold text emphasis and plain-text formatting for command names and codes/responses are mine):
In
systemd-resolved, when performing lookup of domain names with 'secure' in them, and receivingNXDOMAINrepsonse, retry again without EDNS0.
This therefore is what your messages are stating, but with a more broad coverage. Namely, when a DNS request is filed and you get an NXDOMAIN response from the DNS lookup, we want to make sure that we aren't hitting this EDNS0 problem; therefore, systemd-resolved is attempting the DNS lookup again but without the EDNS0 extensions - we can see this in my Bind9 nameserver which handles all requests on my machine as well with an example (though this is a prime example of a non-domain being queried, and in turn a prime example of how this mitigation can show itself when the 'mitigation' is not actually going to solve the headache):
05-Feb-2019 13:29:17.976 queries: info: client @0x7f6cd400aee0 127.0.0.1#41213 (favicon.png): query: favicon.png IN A +E(0) (127.0.2.1)
05-Feb-2019 13:29:17.976 queries: info: client @0x7f6cd400aee0 127.0.0.1#41213 (favicon.png): query: favicon.png IN A + (127.0.2.1)
As you can see, instead of +E(0) the request is retried without that extension; this is the 'reduced feature level UDP' request behavior.
Solution 2:
Although this is not an actual error, if you want these messages to stop, you can change the target the /etc/resolv.conf symlink is pointing to. By default it points to /run/systemd/resolve/stub-resolv.conf. By changing it to point to /run/systemd/resolve/resolv.conf the problem is gone.
The command to do this, is sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
Solution 3:
As was pointed out, the NXDOMAIN error is not your problem, but it can cause a default Ubuntu 18.04 (18.10?) installation to fail on certain DNS queries like pop3.comcast.net (see launchpad bugs systemd-resolved has issues when the answer is over 512 bytes with EDNS disabled and systemd-resolved can't resolve Comcast mail server addresses A simple fix is to install the libnss-resolve package, which alters the /etc/nsswitch.conf file to better handle the situation.
Solution 4:
You can remove a lot of that noise by adding one trailing dot in this config file:
diff -u -r1.1 /usr/lib/NetworkManager/conf.d/20-connectivity-ubuntu.conf
--- /usr/lib/NetworkManager/conf.d/20-connectivity-ubuntu.conf
+++ /usr/lib/NetworkManager/conf.d/20-connectivity-ubuntu.conf
@@ -1,2 +1,2 @@
[connectivity]
-uri=http://connectivity-check.ubuntu.com/
+uri=http://connectivity-check.ubuntu.com./
A trailing dot is for DNS the equivalent of a leading '/' for files: it makes any relative DNS name absolute. In other words it stops the system searching for a connectivity-check.ubuntu.com.yourlocal_domain. that is unlikely to exist (unless your ISP is trying to hijack this particular name).
https://askubuntu.com./
You can find other NXDOMAIN errors by keeping an eye on these two commands simultaneously:
journalctl --boot -u systemd-resolved -f
tcpdump -i any 'port domain'
See the before/after difference it makes:
nmcli c down "Wired connection 1"; nmcli c up "Wired connection 1"
# wait, observe
Edit file as above, then:
systemctl reload NetworkManager
nmcli c down "Wired connection 1"; nmcli c up "Wired connection 1"
# wait, observe
(There are other names causing NXDOMAIN, connectivity-check.ubuntu.com is just one of them)