If I join esxi to an Active Directory domain, how does it choose which DC to authenticate to?

ESXi (just like any other system) will always allow local authentication (i.e. the local root user and any local user account you created) when other authentication methods are unavailable; if you have local credentials, you'll always be able to login to an ESXi server, even if vCenter, AD, or whatever else is not available.

Documentation:

http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.vsphere.security.doc/GUID-D7AEC653-EBC8-4573-B990-D8E58742F8ED.html

My experience with ESXi AD integration (actually Likewise) is it can be flaky. It's probably fine for small, simple topologies, but it can fall down with more complex, distributed topologies. In every case for me, a vanilla computer can join or authenticate with AD just fine using the same connection or network segment when ESXi is exhibiting issues.

Your best bet is to enable the logging for the Likewise components, otherwise you aren't going anywhere when there is an issue. And you can't do this through the UI, get the CLI.

Enabling logging for Likewise agents on ESXi/ESX (1026554)
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026554

Regarding the "how" is it should be doing exactly what a Windows client does, and follow the DC Locator process. I would suspect it is not, or is deviating in some way.

Domain Controller Location Process
http://technet.microsoft.com/en-us/library/cc978011.aspx

Notes Regarding ESX(i) AD Integration:

What i've discovered (on ESXi 5.0) is when joining the ESXi host to the domain (GUI) the process via Likewise agent (on host) enumerates the trusted domains and domain controllers at the time of the join and populates a file in /etc/likewise/krb5-affinity.conf with each child/domain and associated DC.

The process seems to only enumerate the domain at that single point in time. Examining the file showed me that the listed DC's were never automatically updated because there were many old DC IPs that were decommissioned or replaced and still in that list.