Why my local DNS is not used?

Solution 1:

By default, NetorkManager is running and it will write /etc/resolv.conf with the settings received via DHCP.

As you run a local DNS server, you don't want that, and you basically want to always use 127.0.0.1 as nameserver. You need to tell NetworkManager that you want that:

add /etc/NetworkManager/conf.d/90-dns-none.conf with

[main]
dns=none

And issue systemctl reload NetworkManager. See also man NetworkManager.conf.

Afterwards edit /etc/resolv.conf to your liking.

Btw, you could have also marked the file as readonly with chattr -i /etc/resolv.conf followed by systemctl reload NetworkManager. Similar result.

Or, you could replace /etc/resolv.conf with a symlink to the actual file with nameserver 127.0.0.1, followed by systemctl reload NetworkManager. That also tells NetworkManager to stay away.

Solution 2:

apparently (on GCP machines) command sudo systemctl restart NetworkManager.service overrides /etc/resolv.conf every time and my nameserver is lost.. not sure why but its a different question.