How to Windows clients use the correct IP to connect to the remote systems with multiple IPs

Solution 1:

how do clients find the correct IP from DNS

They assume that all IP addresses returned by the query are correct.

Yes, if the first address doesn't respond in time, the client will try the second one, then third, etc. (in undefined order – DNS records are unordered), but this can get noticeably slow, and the only time it should happen is during some kind of network/configuration issue, not during normal operation.

If some addresses are deliberately not reachable, they shouldn't be in DNS. The server's main DNS entry should represent the main entry point – it could still return multiple addresses, of course (such as IPv4 + IPv6 for a start), but all of them should behave exactly the same. Things like "management" addresses should have a different DNS entry (if they go in DNS at all).

When you're relying on automatic DNS updates through Active Directory, you can disable registration for unwanted network adapters (Ethernet ports). If it's a client or a member server (not a DC), disable "Register this connection's addresses in DNS" in the adapter's Properties dialog. (If the server is an AD DC, there are different steps.)