Bare metal kubernetes DNS not forwarding

I have a bare metal kubernetes cluster set up for my own education purposes. It's three VMs - one master, two workers, running Ubuntu 20.04. I'm having problems with pods forwarding DNS requests.

CoreDNS is running

kubectl get po --all-namespaces |grep dns
kube-system      coredns-74ff55c5b-cdvcv                   1/1     Running   11         4d

It has an expected cluster IP

kubectl get svc kube-dns -n kube-system
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   4d

However it cannot resolve names, such as this attempt on master node:

nslookup github.com 10.96.0.10
Server:     10.96.0.10
Address:    10.96.0.10#53

** server can't find github.com: SERVFAIL

If I edit ConfigMap.yaml

forward . /etc/resolv.conf {
    max_concurrent 1000
}

to forward . 8.8.8.8 { then it will work.

nslookup github.com 10.96.0.10
Server:     10.96.0.10
Address:    10.96.0.10#53

Non-authoritative answer:
Name:   github.com
Address: 140.82.112.3

However this won't resolve private LAN DNS names, obviously. I need CoreDNS to forward to the DNS of the machine it runs on.

I am running k8s 1.20.3, which as I understand no longer has a problem with an infinite loop due to Ubuntu's stub resolv.conf. Besides I've tried brute force deleting the symlink /etc/resolv.conf and recreating it to point to /run/systemd/resolve/resolv.conf on each VM, reboot each, and nothing improved.

I discovered that a pod cannot ping the host machine's DNS (192.168.149.2), so it seems like that's the issue, but I don't know what to do about it.

What else should I be looking at?

Solution 1:

Resolved per Matt's clues.

Pay close attention to initial cluster setup. Instructions online might lead you to overlook an overlapped IP range. Kubernetes private IPs must be separate from private IPs external to the cluster.

Use a tool like kubectl run curl-busybox --image=radial/busyboxplus:curl -i --tty --rm to curl, nslookup and ip a which gives some visibility to the pod's view of the network.