DNS over TLS with systemd-resolved

I was trying to enable DNS over TLS via systemd-resolved. I changed /etc/systemd/resolved.conf as follows:

[Resolve]
DNS=1.1.1.1
#FallbackDNS=
Domains=~.
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
DNSOverTLS=opportunistic
#Cache=yes
#DNSStubListener=yes

While monitoring the network (with tcpdump) to see if the resulting behavior was the intended one, it seems that a TLS session is established with the target server; but, then the server closes the connection. I get the same results with 1.1.1.1, 8.8.8.8 and others.

Any ideas on how to fix this?

P.S.: systemd-resolved ends up doing parallel resolution with traditional DNS (despite the setting of "Domains" above). But my main question for this post is what may be going wrong with the TLS one.

Recently I've implemented DNS over TLS for home network (using AsusWRT-Merlin powered router). While exploring approaches to implement DoT for Linux workstations and servers (in the cloud - I mean someone else' computers lol) outside of home network, I found systemd-resolved as recommended by the DNS Privacy Project.

While @opinion-no9 provided a Ubuntu 18.04 specific solution (limited by the systemd version shipped with the LTS), I'd like to share a more generic and close-to-upstream one:

A bit of background - timeline for DNS over TLS support in systemd

• 236 DNSSEC support for RFC 8080 (ed25519 keys and signatures)
• 239 systemd-resolved now supports opportunistic DNS-over-TLS, Off by default
• 243 systemd-resolved gained support for a new strict DNS-over-TLS mode

Oh NO! Ubuntu 18.04 LTS ships systemd 237...

UPDATE - 23 April 2020 Ubuntu 20.04 LTS which ships systemd 245 (without systemd-homed ;-).

Use DNS Over TLS (DoT) on generic Linux systems

So for generic Linux distro with relative close-to-upstream kernel, systemd, glibc, toolchain, GNU utils, etc

1. edit /etc/systemd/resolved.conf

[Resolve]
DNS=1.1.1.1 1.0.0.1 8.8.8.8
#FallbackDNS=1.1.1.1 9.9.9.10 8.8.8.8 2606:4700:4700::1111 2620:fe::10 2001:4860:4860::8888
#Domains=
#LLMNR=yes
#MulticastDNS=yes
#DNSSEC=allow-downgrade
#DNSOverTLS=opportunistic
DNSSEC=yes
DNSOverTLS=yes
#Cache=yes
#DNSStubListener=yes
#ReadEtcHosts=yes

Ubuntu 20.04 specific resolved.conf

[Resolve]
DNS=1.1.1.1 1.0.0.1 8.8.8.8
FallbackDNS=1.1.1.1 8.8.8.10 8.8.8.8
#Domains=
#LLMNR=no
#MulticastDNS=no
DNSSEC=yes
DNSOverTLS=yes
#Cache=yes
#DNSStubListener=yes
#ReadEtcHosts=yes

2. Assume systemd-resolved is enabled, restart the service

systemctl restart systemd-resolved.service

3. Use local stub resolver

systemd-resolved provides a local DNS stub listener on IP address 127.0.0.53 on the local loopback interface, so to use the DNS over TLS capable stub resolver, we'll need to somehow manage /etc/resolv.conf and make sure 127.0.0.53 is used as nameserver.

NOTE: systemd maintains /run/systemd/resolve/stub-resolv.conf for compatibility with traditional Linux programs. We can simply symlink to this file ;-)

ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

NOTE: For Arch Linux, I have to replace openresolvconf with systemd-resolvconf.

Done.

Validation

Generate DNS query, then check TCP connection to specified upstream DNS server via port 853, in the following example we explicitly used Cloudflare's 1.1.1.1.

root@netbook:/etc# uname -a
Linux netbook 5.4.5-arch1-1 #1 SMP PREEMPT Wed, 18 Dec 2019 19:48:51 +0000 x86_64 GNU/Linux

root@netbook:~#  kdig -d github.com
;; DEBUG: Querying for owner(github.com.), class(1), type(1), server(127.0.0.53), port(53), protocol(UDP)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 55366
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; github.com.                  IN      A

;; ANSWER SECTION:
github.com.             58      IN      A       13.236.229.21

;; Received 44 B
;; Time 2019-12-21 22:55:13 AEDT
;; From 127.0.0.53@53(UDP) in 58.0 ms

root@netbook:~# ss -tuna | grep :853
tcp   ESTAB  0       0        192.168.1.150:50504         1.1.1.1:853
tcp   ESTAB  0       0        192.168.1.150:50506         1.1.1.1:853

Or, if you want to be simple and rough, use tcpdump ;-)

tcpdump -tttt -nn -XX -vv -i <interface> dst 1.1.1.1 and port 853

Conclusion

Last but not least: This is a generic guide to enable DNS over TLS on a Linux host, it is not specific to distro or Desktop Environment (as we haven't even touched NetworkManager or alternatives ;-). Adjustments may be needed for different distros and different DEs/WMs.

Reference

• Personal DoT Implementation Notes
• DNS Privacy Clients
• Arch Wiki - systemd-resolved
• systemd-resolved.service

DNS-over-TLS in Ubuntu 18.04 .... 20.04

Based on my experience, it will not work the way mentioned above on Ubuntu 18.04+ (i.e. U19, U20, U21).

Since Ubuntu 18+ uses Netplan in parallel with NetworkManager things have changed drastically:
No more manual config in the traditional ways ;-( Documentation is .... sparse.

snap may bring further changes. It establishes additional resolved.conf files. However, the following actually worked (/w good performance).

For Info about Netplan look here.

What DNS-over-TLS was working successful (works4me):

1. In /etc/systemd/resolved.conf ONLY change the DNSOverTLS setting to

   DNSOverTLS=opportunistic

   There is NO other option (see explanation here: DNS over TLS

2. Use a capable DNS Server. I prefer Digitalcourage servers

   46.182.19.48 resp. 2a02:2970:1002::18

   or https://dnsforge.de/

   Why? Privacy!

3. This should be done in Network-Manager via GUI!

   Enter the DNS server address in the GUI field for your connection under IPv4 Settings / DNS Servers and v6 respectively.

   The entries will NOT show up in /etc/resolv.conf !! Which is correct. Instead, you will see nameserver 127.0.0.53

   This is new Ubuntu .... no longer suitable for hobby-admins.

4. Not recommended but possible alternative

   Setting appropriate DNS Servers can be done directly in /etc/resolv.conf in the usual format, remove the 127.0.0.53 or something else.

   Problem: Gets overwritten by Network-Manager in Ubuntu!

   Remedy: As true root (!) chattr the file /etc/resolv.conf

   chattr +i /etc/resolv.conf

   This is brute force and may disable automatic DNS caching via resolved.

Credit to the Arch documentation

However, works fine ;-) but needs manual maintenance as true root!

Tip:

You are well advised to make resolv.conf a link. This is required by resolved to work properly. As sudo-root move away the old file then

sudo ln -s /run/resolvconf/resolv.conf /etc/resolv.conf

I do not like this way but is inherently working proper.

.

Then reboot. Or restart network.

.

How to check

1. Some basic checks:

Check the DNS actually used by systemd-resolved:

resolvectl status

Check if DNS resolves with resolvectl:

resolvectl query archlinux.org 

(Try some examples)

Check what DNS is actually used, check for leaks in VPN:

https://dnsleaktest.com/

. 2. Start Wireshark and filter for "port 53" and make web traffic.

This should not show connects on port 53 any longer. Then filter for port 853. Here should a lot go on.

Important: If all traffic uses port 853 and no traffic uses 53 you did it successfully!

Wireshark examples here.

Remark: I tried stubby. stubby does not integrate well in Ubuntu but you can get it working even with NetworkManager. There is one manual to do it successfully: How to use DNS-over-TLS on Ubuntu Linux Problem: performance was a bit of a pain. Something is weird and I did not find out the cause.

Enabling DNSSEC=yes in /etc/systemd/resolved.conf should be possible now.

Important:

This solution improves privacy a lot.

BUT is NOT sufficient if your personal integrity depends on data privacy and security!! See caveats in resolvd description. It is NOT sufficient to have opportunistic mode. Then better watch out for Tails Linux. A sad greeting to all political prisoners around the world.

updated version of this is available -- in german only -- via TOR url: http://mutant21.onastroidst6krpn.onion/schweinekraftland/DNS_sicherer_machen/sicherer%20surfen%20und%20mailen%20mit%20gesichertem%20DNS.htm

Just to be sure:

• 1.1.1.1 uses port 853 for DoT (DnsOverTLS), see page at https://1.1.1.1/
• systemd-resolved uses the port automatically
• remember to open the port on your firewall (imPORTant)

I still use shell commands with "ifconfig" or "ip" to up/down interfaces and set routes, "netstat" and "ss" for monitoring and so on, no systemd-networkd, network-manager or netplan. You just have to disable systemd-* packages or remove packages, to run your system the way you want to as there are no impediments for that.

BUT if you feel inclined to try new horizons and see the world from a different perspective, just in case you are configuring a desktop machine (not a workstation, nor a server) and you have a really well configured firewall all done by hand using shell scripting capabilities to configure it (cos you can) AND you also know your tcp flags well, when and HOW to allow PSH and URG, and you ALSO know sysctl options by heart and are not afraid to use it against the enemy, only then, you can use firefox implementation of a DoH client within the browser, and you can find it in configuration > General > Network Configuration, and use it in conjunction with nextdns, and you can also disable UDP and systemd-resolved entirelly and have just tcp working on your desktop machine ... Oh boy, pure bliss... or almost.