DNS Resolution Fails for ping and curl, but not dig

I'm running DNSMasq as a local DNS server, so I can resolve *.local.pcfdev.io (as discussed here Using PCF Dev Offline with Mac OS X). Everything worked when I first set things up.

A couple of days later, after a few restarts of my MacBook, whilst offline I can no longer resolve things like api.local.pcfdev.io using curl or ping. However, dig does the right thing.

$ dig api.local.pcfdev.io

; <<>> DiG 9.8.3-P1 <<>> api.local.pcfdev.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46877
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;api.local.pcfdev.io.       IN      A

;; ANSWER SECTION:
api.local.pcfdev.io.    0       IN      A       192.168.11.11

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Sep  6 10:17:44 2016
;; MSG SIZE  rcvd: 53

$ curl api.local.pcfdev.io
curl: (6) Could not resolve host: api.local.pcfdev.io

I've tried adding -AlwaysAppendSearchDomains as an argument to /usr/sbin/mDNSResponder in /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist and restarted the mDNSResponder with launchctl, but to no avail.

UPDATE 1

There is definitely something listening on the right local IP:

$ nslookup api.local.pcfdev.io
Server:     127.0.0.1
Address:        127.0.0.1#53

Name:   api.local.pcfdev.io
Address: 192.168.11.11

$ ping api.local.pcfdev.io
ping: cannot resolve api.local.pcfdev.io: Unknown host

$ telnet 192.168.11.11 80
Trying 192.168.11.11...
Connected to 192.168.11.11.
Escape character is '^]'.

HTTP/1.1 400 Bad Request

Connection closed by foreign host.

UPDATE 2

After trying the suggestion below of removing all DNS servers from Network Preferences except 127.0.0.1, I can't resolve anything. I managed to get some debug logging out of mDNSResponder:

mDNSResponder[91]:  74: DNSServiceCreateConnection START PID[32612](ping)
mDNSResponder[91]:  74: Error socket 75 created 00000000 00000001
mDNSResponder[91]:  74: DNSServiceQueryRecord(15000, 0, api.local.pcfdev.io., Addr) START PID[32612]()
mDNSResponder[91]:  74: Error socket 75 closed  00000000 00000001 (0)
mDNSResponder[91]:  74: DNSServiceQueryRecord(api.local.pcfdev.io., Addr) ADD    0 api.local.pcfdev.io. Addr
mDNSResponder[91]:  74: Cancel 00000000 00000001
mDNSResponder[91]:  74: DNSServiceQueryRecord(api.local.pcfdev.io., Addr) STOP PID[32612]()
mDNSResponder[91]:  74: DNSServiceCreateConnection STOP PID[32612](ping)

I did also observe that as explained in the proposed answer, nslookup and dig don't cause anything to be logged by mDNSResponder, but other tools (ping, curl) do.

So it seems like for whatever reason either dnsmasq isn't working (I can establish a TCP connection to 127.0.0.1:53) or mDNSResponder isn't using it.

UPDATE 3

etc/resolve.conf ceases to exist when my wifi adapter is active, but I'm not connected to a network. Could this be why CLI tools don't use the local dnsmasq server?

Had this same issue. I think the local DNS cache had bad data from my previous testing. It was quickly fixed by:

sudo killall -HUP mDNSResponder

dig on the one hand and curl/ping on the other hand are retrieving data from different hosts:

dig queries a DNS server – in your case your localhost (127.0.0.1) – for a database entry: the IP address related to the FQDN api.local.pcfdev.io. The host itself doesn't have to run or even exist at all.

curl/ping try to resolve an IP-address with mDNSResponder or by other means and finally operate on/interact with the remote host. If the host 192.168.11.11 doesn't run or doesn't exist at all, both will fail.

Now, either the DNS entry is wrong (api.local.pcfdev.io has another IP than 192.168.11.11) or the DNS entry is correct but the host 192.168.11.11 is not running.

Adding -AlwaysAppendSearchDomains as an argument to /usr/sbin/mDNSResponder in /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist is not recommended. Instead you should add it to /Library/Preferences/com.apple.mDNSResponder.plist (source: man mDNSResponder):

To cause mDNSResponder to run with these optional arguments when it launches on OS X 10.11 (El Capitan) and later, set the AlwaysAppendSearchDomains or NoMulticastAdvertisements boolean keys to true in /Library/Preferences/com.apple.mDNSResponder.plist and reboot.

In your case it's not necessary at all to set this key, because it's not the cause of your problem.

After digging into VirtualBox, PCF Dev (failing repeatedly with some "wrong credentials" trying to log into the VM) and dnsmasq I recommend to devolve DNS queries to dnsmasq only:

• In System Preferences > Network > Interface > DNS server remove all DNS servers except 127.0.0.1 and apply the changes. You may also configure a second Location with a 127.0.0.1 only setup and keep your current DNS server in the other configuration.

• add a file /usr/local/etc/resolv.dnsmasq.conf with the content

  #use your preferred DNS servers here. In the example I use some Google name servers
  nameserver 8.8.8.8
  nameserver 8.8.4.4
  

• add resolv-file=/usr/local/etc/resolv.dnsmasq.conf at line ~46 of /usr/local/etc/dnsmasq.conf
• add or move address=/.local.pcfdev.io/192.168.11.11 at/to line ~80 of /usr/local/etc/dnsmasq.conf

• restart dnsmasq with:

  sudo launchctl stop homebrew.mxcl.dnsmasq
  sudo launchctl start homebrew.mxcl.dnsmasq