DNS Server Responses and Timeouts

We're experiencing a frustrating problem on our LAN. Periodically, DNS queries to our ISP nameservers timeout forcing a 5 second delay. Even if I bypass /etc/resolv.conf by using a direct dig to one of our DNS servers, I still encounter the problem. Here's an example:

mv-m-dmouratis:~ dmourati$ time dig www.google.com @209.81.9.1 

; <<>> DiG 9.8.3-P1 <<>> www.google.com @209.81.9.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14473
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION:
;www.google.com.            IN  A

;; ANSWER SECTION:
www.google.com.     174 IN  A   74.125.239.148
www.google.com.     174 IN  A   74.125.239.147
www.google.com.     174 IN  A   74.125.239.146
www.google.com.     174 IN  A   74.125.239.144
www.google.com.     174 IN  A   74.125.239.145

;; AUTHORITY SECTION:
google.com.     34512   IN  NS  ns2.google.com.
google.com.     34512   IN  NS  ns1.google.com.
google.com.     34512   IN  NS  ns3.google.com.
google.com.     34512   IN  NS  ns4.google.com.

;; ADDITIONAL SECTION:
ns2.google.com.     212097  IN  A   216.239.34.10
ns3.google.com.     207312  IN  A   216.239.36.10
ns4.google.com.     212097  IN  A   216.239.38.10
ns1.google.com.     212096  IN  A   216.239.32.10

;; Query time: 8 msec
;; SERVER: 209.81.9.1#53(209.81.9.1)
;; WHEN: Fri Jul 26 14:44:25 2013
;; MSG SIZE  rcvd: 248


real    0m5.015s
user    0m0.004s
sys 0m0.002s

Other times, the queries respond instantly, as in under 20 ms or so. I've done a packet trace and discovered something interesting. The DNS server is responding but the client ignores the initial response, then sends a second identical query which is immediately responded to.

See packet trace. Note the identical source ports to the queries (62076).

Question: what is causing the first DNS query to fail?

UPDATE

Resources:

Packet trace:

http://www.cloudshark.org/captures/8b1c32d9d015

Dtruss (strace for mac):

https://gist.github.com/dmourati/6115180

Mountain Lion firewall is randomly delaying DNS requests from apple.stackexchange.com:

https://apple.stackexchange.com/questions/80678/mountain-lion-firewall-is-randomly-delaying-dns-requests

UPDATE 2

System Software Overview:

  System Version:   OS X 10.8.4 (12E55)
  Kernel Version:   Darwin 12.4.0
  Boot Volume:  Macintosh HD
  Boot Mode:    Normal
  Computer Name:    mv-m-dmouratis
  User Name:    Demetri Mouratis (dmourati)
  Secure Virtual Memory:    Enabled
  Time since boot:  43 minutes

Hardware Overview:

  Model Name:   MacBook Pro
  Model Identifier: MacBookPro10,1
  Processor Name:   Intel Core i7
  Processor Speed:  2.7 GHz
  Number of Processors: 1
  Total Number of Cores:    4
  L2 Cache (per Core):  256 KB
  L3 Cache: 6 MB
  Memory:   16 GB

Firewall Settings:

  Mode: Limit incoming connections to specific services and applications
  Services:
  Apple Remote Desktop: Allow all connections
  Screen Sharing:   Allow all connections
  Applications:
  com.apple.java.VisualVM.launcher: Block all connections
  com.getdropbox.dropbox:   Allow all connections
  com.jetbrains.intellij.ce:    Allow all connections
  com.skype.skype:  Allow all connections
  com.yourcompany.Bitcoin-Qt:   Allow all connections
  org.m0k.transmission: Allow all connections
  org.python.python:    Allow all connections
  Firewall Logging: Yes
  Stealth Mode: No

This appears to be a bug in Lion's firewall. Is it enabled on your system?

In this MacRumors thread (DNS problems after updating to Mountain Lion (10.8)), a possible workaround is discussed:

Try reducing MTU size.

System Preferences > Network > WiFi > Advanced > Hardware > Manually > MTU: Custom > 1300

Worked for me.

Could you check whether reducing the MTU size mitigates your problem?