systemd-resolved not starting on boot

When I boot my Ubuntu server, DNS resolving does not work. You can most easily see this by looking at resolv.conf:

~ # cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

I then enable the resolved service:

~ # systemctl enable systemd-resolved.service

And things work as expected:

~ # cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53

Config

• Ubuntu 19.04
• 5.0.0-23-generic

Related packages installed:

~ # cat /etc/resodpkg -l |grep systemd
ii  libnss-resolve:amd64                  240-6ubuntu5.2                        amd64        nss module to resolve names via systemd-resolved
ii  libnss-systemd:amd64                  240-6ubuntu5.2                        amd64        nss module providing dynamic user and group name resolution
ii  libpam-systemd:amd64                  240-6ubuntu5.2                        amd64        system and service manager - PAM module
ii  libsystemd0:amd64                     240-6ubuntu5.2                        amd64        systemd utility library
ii  networkd-dispatcher                   2.0-2                                 all          Dispatcher service for systemd-networkd connection status changes
ii  python3-systemd                       234-2build2                           amd64        Python 3 bindings for systemd
ii  systemd                               240-6ubuntu5.2                        amd64        system and service manager
ii  systemd-sysv                          240-6ubuntu5.2                        amd64        system and service manager - SysV links
~ # dpkg -l |grepdpkg -l |grep resolv
ii  libc-ares2:amd64                      1.14.0-1                              amd64        asynchronous name resolver
ii  libgeoip1:amd64                       1.6.12-1                              amd64        non-DNS IP-to-country resolver library
ii  libnss-resolve:amd64                  240-6ubuntu5.2                        amd64        nss module to resolve names via systemd-resolved
ii  resolvconf                            1.79ubuntu13                          all          name server information handler

Name resolution worked on boot prior to updating to 19.04.

No apparent errors in syslog. Related excerpt:

Aug  3 11:41:16 green systemd[1]: Starting Network Name Resolution...
Aug  3 11:41:16 green systemd-resolved[807]: Positive Trust Anchors:
Aug  3 11:41:16 green systemd-resolved[807]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Aug  3 11:41:16 green systemd-resolved[807]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Aug  3 11:41:16 green systemd-resolved[807]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arp
Aug  3 11:41:16 green systemd[1]: Started Network Name Resolution.

Status before reboot (with nameserver listed in /etc/resolv.conf):

~ # systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2019-08-03 11:39:32 EDT; 2 days ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 807 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 7.9M
   CGroup: /system.slice/systemd-resolved.service
           └─807 /lib/systemd/systemd-resolved

Aug 06 10:00:38 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:04:06 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:04:06 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:04:06 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:04:06 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:15:53 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:17:05 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:17:05 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:17:05 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:17:05 green systemd-resolved[807]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature

Status before reboot (with nothing listed in /etc/resolv.conf):

~ # systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-08-06 10:23:53 EDT; 1min 9s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 798 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 7.8M
   CGroup: /system.slice/systemd-resolved.service
           └─798 /lib/systemd/systemd-resolved

Aug 06 10:23:51 green systemd[1]: Starting Network Name Resolution...
Aug 06 10:23:52 green systemd-resolved[798]: Positive Trust Anchors:
Aug 06 10:23:52 green systemd-resolved[798]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Aug 06 10:23:52 green systemd-resolved[798]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Aug 06 10:23:52 green systemd-resolved[798]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.
Aug 06 10:23:52 green systemd-resolved[798]: Using system hostname 'green'.
Aug 06 10:23:53 green systemd[1]: Started Network Name Resolution.

~ # systemd-resolve --status
Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 8.8.8.8
         DNS Servers: 8.8.8.8
                      8.8.4.4
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (eno2)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (eno1)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

After systemctl enable systemd-resolved.service(with nameserver listed in /etc/resolv.conf):

~ # systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-08-06 10:23:53 EDT; 3min 47s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 798 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 7.9M
   CGroup: /system.slice/systemd-resolved.service
           └─798 /lib/systemd/systemd-resolved

Aug 06 10:23:52 green systemd-resolved[798]: Positive Trust Anchors:
Aug 06 10:23:52 green systemd-resolved[798]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Aug 06 10:23:52 green systemd-resolved[798]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Aug 06 10:23:52 green systemd-resolved[798]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.
Aug 06 10:23:52 green systemd-resolved[798]: Using system hostname 'green'.
Aug 06 10:23:53 green systemd[1]: Started Network Name Resolution.
Aug 06 10:27:32 green systemd-resolved[798]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:27:35 green systemd-resolved[798]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:27:35 green systemd-resolved[798]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature
Aug 06 10:27:35 green systemd-resolved[798]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature

systemd-resolve --status shows the same as before.

Solution 1:

You installed something like resolvconf which cuts sytemd-resolved off entirely by commenting out the 53 listening address in stub-resolv.conf which the /etc/resolv.conf should link to.For a fresh Ubuntu 18.04 install, the only additional name resolution package I needed was libnss-resolve. I never use resolvconf, so don't know why you need it.