docker.io DNS doesn't work, it's trying to use 8.8.8.8

When the Ubuntu Docker package updated to using systemd, it dropped support for the /etc/default/docker config file, so the initial solution suggested by rocketman10404 will no longer work (disabling dnsmasq would still work, but it has the downside of preventing Ubuntu from automatically updating the DNS server).

Fixing in the new daemon.json config file

Find your network's DNS server:

$ nmcli dev show | grep 'IP4.DNS'
IP4.DNS[1]:                             10.0.0.2

Open or create, if it doesn't exist, /etc/docker/daemon.json and add the DNS servers that you want to use:

# /etc/docker/daemon.json
{
    "dns": ["10.0.0.2", "8.8.8.8"]
}

Restart the docker daemon:

$ sudo service docker restart

I wrote an in-depth blog post and also filed a bug about this issue if you'd like more details.

(Originally I solved it by opening up /lib/systemd/system/docker.service and add DNS settings to the ExecStart line, but that's bad - we shouldn't edit systemd files directly.)


I don't use docker myself, so I normally wouldn't butt-in here on a docker question, but I just happened to be reading about it and stumbled on some docker documentation that appears to address this exact problem. To sum-up...

The documentation suggests a few workarounds. The first is to specify the DNS server to be used by the docker daemon for the containers by adding the following line to /etc/default/docker:

docker_OPTS="--dns 8.8.8.8"

where the DNS provided could be a local DNS server, such as 192.168.1.1 (gateway). Then, restart with

sudo restart docker

An alternative solution involves disabling dnsmasq in NetworkManager by commenting out the config in /etc/NetworkManager/NetworkManager.conf like so:

#dns=dnsmasq

then, restart both

sudo restart network-manager
sudo restart docker

I ran into this in my situation which is specifically

  • Running Docker containers on my local development machine
  • Which is connected to a VPN
  • Some of our container build scripts do things like run npm install from a custom repository on the VPN, inside the container.
    • This works from a CI pipeline but not from our developer machines, because npm can't do a successful DNS lookup
    • We also had problems with containers that need to do lookups to call external REST APIs

Ubuntu by default uses dnsmasq started up by NetworkManager to cache DNS requests, and configures /etc/resolv.conf to point to this instance on 127.0.1.1

  • The VPN client we're using is not NetworkManager compatible and forces it's own /etc/resolv.conf which overwrites the NetworkManager config
  • This configures the DNS servers for the VPN
  • Docker shadows your /etc/resolv.conf to the container by default
    • Usually on Ubuntu, it passes the Google DNS servers to the container (because it knows about the dnsmasq situation.
    • But it's happy to pass the VPN DNS server config to the container
    • There's no route from the container on the docker0 network bridge, to the DNS servers over the VPN tap0 adapter.
  • Ergo, all DNS lookups in the container fail because it can't reach the only DNS servers it's supplied with
  • Additionally, some networks block requests to the Google DNS servers because they want to be able to snoop all your DNS lookups

The solution :

It would seem to be more elegant to use NetworkManager and it's captive dnsmasq instance in the way it was designed.

  1. Tell Docker to use your dnsmasq instance for DNS

    • Add or edit file /etc/docker/daemon.json to tell docker to use the docker0 bridge adapter for DNS

      {
        "dns": ["172.17.0.1"]
      }
      
  2. Configure the NM dnsmasq instance to listen to the Docker bridges as well, because by default it only listens to 127.0.1.1 - create file /etc/NetworkManager/dnsmasq.d/docker-bridge.conf

    # Default Docker bridge
    interface=docker0
    # Other Docker bridges
    interface=br-*
    
  3. I don't like the rude behaviour of that VPN client and I'd rather only use the DNS at the VPN end for VPN lookups (if you have a polite VPN client that uses NetworkManager configured correctly, you won't have to do this)

    • Turn off DNS feature in VPN client (it stops overwriting resolv.conf on connect and now all DNS goes through dnsmasq again)
    • Add a config file to tell dnsmasq to direct DNS requests for your domain appropriately - add file `/etc/NetworkManager/dnsmasq.d/vpn-dns.conf

      server=/myprivatedomain.net/10.0.0.1  
      # or whatever your private DNS server is
      
    • Optionally, add a search domain for your domain so you can use short names

      • I just added our local domain to the search list in my default network connection
  4. Restart NetworkManager and Docker

    sudo service network-manager restart
    sudo service docker restart
    

At this point your Docker containers should be able to do nslookup without issues when you're on VPN, for domains both inside and outside your VPN.