How to find out how DNS is resolved on client computer?

Your Windows machine must have some sort of mDNS resolver (Bonjour), let it be from iTunes, Skype, or other software that installs a mDNS resolver. ping will resolve correctly if a resolver is installed because LLMNR is now used in Windows Vista and above. LLMNR utilizes the mDNS resolver to return the record.

Synology uses a daemon called avahi, this is a Multicast DNS/mDNS/DNS-SD server. From the factory, Synology comes out of the box with it configured as "RackStation" or hostname from what I have experienced.

If you have a Mac, dns-sd -B will show you multicasted services. You will see _http._tcp.local. service listed for RackStation.

dns-sd -G v4 RackStation.local should give you the proper IP address for your RackStation

dns-sd -L RackStation _http._tcp. should return all the information of the service, this includes port numbers, serial, model, etc

This also applies to DiskStation as well. I did my testing on a DiskStation. RackStation should be the same.

(This answer is probably wrong. @xeon seems to be on the right track and I've learned something here.)

The reason a DNS lookup fails is because ping isn't using DNS to resolve that name.

nslookup is a command for DNS lookups. ping on the other hand will also attempt to use NetBIOS/WINS and the hosts file for name resolution. The most likely scenario is that this name was obtained via the former.

You can use nbtstat -c to list the NetBIOS names and IP addresses that have recently been seen, or nbtstat -a <device> if you want to actively search for a name. Keep in mind that -a will take longer as it has to attempt this once per interface. (you can't tell it to skip broadcast/only use WINS)

The -r option will let you compare what was resolved via broadcast with what was resolved via WINS, but that's getting a little more advanced.