Use hostnames to connect to machines on LAN

It is very unusual for one of these routers to act as a local DNS server, but that is a really nice feature. Short of setting up a new DNS server (I checked your manual and you indeed so not have this feature for your new router), the simplest thing to do really is to give them static IP addresses, and add entries to the HOSTS file for each.

It is fast, easy and effective.

The other way, and I am not a Mac expert, would be to ask a new question about how to enable NetBIOS name service (It has to do with enabling Samba).

There are basically two name based access methods used in many home environments.

1. A router as your WRT320N or TG585v7 uses a dedicated domain such as lan to provide by-name access to every computer that picked up a DHCP lease at the router. The router usually also propagates lan as a search domain to the clients (will show up as domain lan or search lan in the /etc/resolv.conf file of your Mac client for example), so that you don't even need to use the .lan prefix when using other computers host names.

2. Multicast DNS is a server-less method, highly used by Apple but also implemented in Linux (avahi) and Windows. It allows you to access other hosts by the domain .local. Instead of asking a configured name server, when trying to resolve a .local hostname your computer simply broadcasts the request in the local network and some computer might reply with "hey, thats my name, here, thats my IP address"

The latter often works out of the box (if you have multiple Mac's on the network you can definitely use that method to address them, thats also the way they "discover" each other so you can use file sharing etc. through Finder). In your case, mDNS is probably what makes it possible for you to resolve your Macs name from the Windows machine. Your windows however does not seem to reply to mDNS requests (probably firewalled? Can' really help here, don't use Windows anywhere)

If you want the first (more controlled method) you need to check your routers firmware on how to enable that feature. If your routers firmware doesn't support that feature you might want to try using OpenWRT instead of the stock firmware.