VMware vSphere 5.5 any authentication attempts fail

Solution 1:

What happened

Not sure what had happened, but maybe the whole description helps someone.

The internal LDAP directory of vSphere running typically at port 11711 has ended up with corrupt entries. Resulting in strange observation - one could authenticate towards e.g. webclient, but couldn't do more there as it flooded with messages like "Client is not authenticated to VMware Inventory Service - https://myserverFQDN:10443"

Solution

Using an LDAP client (e.g. LDAPAdmin) log on to the server using the following connection criteria:

host:          localhost
port:          11711
version:       3
Base:          dc=vsphere,dc=local
Username:      cn=Administrator,cn=Users,dc=vsphere,dc=local
Password:      the one for [email protected]

Navigate to cn=ServicePrincipals and check their vmwSTSSubjectDN to make sure they are unique. Not sure what are the proper ones. In some other installation I found:

InventoryService                1.2.840.113549.1.9.1=#1612737570706f727440766d776172652e636f6d,CN=VMware default certificate,OU=InventoryService_2013.07.31_145941,O=VMware\, Inc.
vCenterServer                   1.2.840.113549.1.9.1=#1612737570706f727440766d776172652e636f6d,CN=VMware default certificate,OU=vCenterServer_2013.07.31_150056,O=VMware\, Inc.

but the last parts (OU=, O=) may be different for some reason

Hints for similar problems

What I found useful was decompilation of the class com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider found in one of the JARs in the vSphere installation. It turns out it contains some hard-coded LDAP queries which might be useful in some other troubleshooting of VMWare's internal LDAP server. (I better not paste it here for licensing reasons). And for decompiling 3rd party Java classes I learned to use http://www.javadecompilers.com. The CFR one works really well.