Bind DNS Recursion Slow

We found the issue. It was a NIC hardware offloading issue.

Running tcpdump -vvv -s 0 -l -n port 53 found a handful of [bad udp cksum 6279!] errors for each DNS query.

A little browse on Google pointed me in the right direction. As it turns out, due to our CentOS system running as VM on XenServer (similar issues reported with VMWare etc) NIC hardware offloading is enabled by default.

Running ethtool -k eth0 | grep on showed the following

x-checksumming: on
tx-checksum-ipv4: on
scatter-gather: on
tx-scatter-gather: on
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
tx-gso-robust: on [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]

Running ethtool -K eth0 tx off rx off disabled TCP TX offloading. I restarted the networking service for good measure

service network restart

and tested BIND. We are now getting very speedy response times from BIND

dig centos.org

; <<>> DiG 9.10.2-P4-RedHat-9.10.2-P4.el6 <<>> centos.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61933
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;centos.org.INA

;; ANSWER SECTION:
centos.org.60INA85.12.30.227

;; Query time: 268 msec
;; SERVER: 192.168.10.25#53(192.168.10.25)
;; WHEN: Thu Sep 17 08:25:39 AEST 2015
;; MSG SIZE  rcvd: 55

I had this same problem with very slow recursive queries on a physical CentOS 7 BIND server and found this answer (TX Offloading) and many IPv6-oriented fixes around various threads, none of which worked for me.

It turns out the location of the server in question had an older Cisco ASA firewall which was limiting the size of UDP response packets to 512 bytes; it seems these days UDP responses for DNS queries are often much larger, up to around 2000 bytes. There's a page about it here:

Why DNS through UDP has a 512 bytes limit?

I configured the ASA to allow larger UDP response packets (there's a specific fixup command for this) which resolved the issue:

https://supportforums.cisco.com/t5/getting-started-with-lans/dns-dropped-because-packets-to-big-for-configured-512/td-p/861718