vSphere 5.1 U1 Client - "The command has timed out as the remote server is taking too long to respond"

The Problem: What we are trying to do

When trying to log into vCenter using the VMware vSphere Client we are greeted with the following error when using both Windows Sessions credentials or manually supplied credentials (DOMAIN\Username):

The vSphere Client could not connect to "vCenter" The server "vCenter" took to long to respond. (The command has timed out as the remote server is taking too long to respond.)

The viclient-#-0000.log has the following:

[viclient:Critical:M:12] 2014-03-04 15:49:59.008  Connection State[vCenter]: Disconnected
[viclient:SoapMsg :M:12] 2014-03-04 15:49:59.009  Attempting graceful shutdown of service ...
[viclient:SoapMsg :M:12] 2014-03-04 15:49:59.010  Pending Invocation Count: 0
[viclient:SoapMsg :M:12] 2014-03-04 15:49:59.011  Graceful shutdown of service: Success
[        :Error   :M:12] 2014-03-04 15:49:59.018  Error occured during login
VirtualInfrastructure.Exceptions.LoginError: The server 'vCenter' took too long to respond. (The command has timed out as the remote server is taking too long to respond.)
   at VirtualInfrastructure.LoginMain.Process(BackgroundWorker worker, DoWorkEventArgs e)
   at VirtualInfrastructure.LoginWorkerImpl.Worker_DoWork(Object sender, DoWorkEventArgs e)

Looking at the vSphere SSO Logs did not reveal any recent activity with the exception of the ssoAdminServer.log which by my reading indicates that lookup of identity sources was successful:

name = kcelliott,
domain = ad.state.gov
inherited from com.vmware.vim.binding.sso.PrincipalId@2fa38f3c
[2014-03-04 16:58:36,301 INFO  opID=10C1719C-00000005-54 pool-13-thread-10  com.vmware.vim.sso.admin.vlsi.PrincipalDiscoveryServiceImpl] Vmodl method 'PrincipalDiscoveryService.findPersonUser' invoked by [ User {Name: vCenterServer_2013.12.19_140038, Domain: System-Domain} with role RegularUser] [caller:/10.5.216.251] Find person user {Name: kcelliott, Domain: ad.state.gov}
[2014-03-04 16:58:36,310 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DefaultCommandExecutor] Command com.rsa.admin.SearchIdentitySourcesCommand was executed successfully
[2014-03-04 16:58:36,318 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DefaultCommandExecutor] Command {'com.rsa.admin.LookupIdentitySourceCommand', 'com.rsa.admin.LookupIdentitySourceCommand', 'com.rsa.admin.LookupIdentitySourceCommand', 'com.rsa.admin.LookupIdentitySourceCommand', 'com.rsa.admin.LookupIdentitySourceCommand'} was executed successfully
[2014-03-04 16:58:36,318 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DomainManagementImpl] Got external domain: ad1.state.us
[2014-03-04 16:58:36,318 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DomainManagementImpl] Got external domain: ad2.local
[2014-03-04 16:58:36,318 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DomainManagementImpl] Got external domain: ad3.local
[2014-03-04 16:58:36,318 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DomainManagementImpl] Got external domain: ad.state.gov
[2014-03-04 16:58:36,318 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DomainManagementImpl] Got external domain: ad4.alaska.local
[2014-03-04 16:58:36,318 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.impl.KeepAlive] Pinging domain ad5.local
[2014-03-04 16:58:36,322 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DefaultCommandExecutor] Command com.rsa.admin.SearchIdentitySourcesCommand was executed successfully
[2014-03-04 17:00:06,215 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DefaultCommandExecutor] Command com.rsa.admin.SearchPrincipalsCommand was executed successfully
[2014-03-04 17:00:06,215 WARN  opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DefaultCommandExecutor] Command 'com.rsa.admin.SearchPrincipalsCommand' executed for 89892 millis
[2014-03-04 17:00:06,215 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.impl.KeepAlive] Ping result: null
[2014-03-04 17:00:06,215 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.impl.KeepAlive] Pinging domain ad5.local
[2014-03-04 17:00:06,221 DEBUG opID= DomainKeepAliveThread  com.vmware.vim.sso.admin.server.ims.impl.DefaultCommandExecutor] Command com.rsa.admin.SearchIdentitySourcesCommand was executed successfully

The Attempted Solutions: How we have tried to fix it

This seems consistent with the information in VMware KB 2038918 and VMware KB 2037408. I tried following the resolution path in VMware KB 2038918 by connecting to vSphere Web Client using the SSO Administrator account (admin@system-domain) and adjusting the Base DN for groups to be narrower instead of the base of the domain in case we running into timeout issues when doing group enumeration. This did not resolve the issue, however I was able to successfully Test Connection. The web client seems to just crawl though, for example it took over three minutes to open the 'Edit identity source' dialog window.

VMware KB 2037408 does not seem to apply in our case as authentication fails whether or not we use Windows session credentials or if I manually supply my Active Directory credentials.

I have restarted both the VMware vCenter service and failing resolution of the issue, the entire vCenter server. This did not resolve the issue.

The Environment: Our Stuff, Hallowed Be Its Breakage

Authentication fails for both vSphere Client and vSphere Web Client from my workstation and locally from the vCenter server. Authentication fails for multiple users. I verified that all of the users attempting to authenticate are members of the vCenter Administrators group (via membership in the Local Administrators on the Windows server vCenter is installed on).

I can successfully ping and connect the LDAPS port of the domain controllers used as identify sources.

The host server does not have any undue resource consumption.

We have made no changes to our vSphere install but we do not manage or have any visibility into our directory services (although I cannot imagine what changed there that would break vCenter SSO).

We are using vSphere 5.1.0 Build 1063329. I am using Firefox 27 with Adobe Flash 12.0.0.70 with the vSphere Web Client. The host operating system for vCenter is Windows Server 2008 R2 SP1 and MS SQL 2012 SP1.

Solution 1:

It turns out that when we installed vCenter's SSO it automatically detected every Active Directory domain that it could. Many of our Departments run and manage their own Active Directory domains instead of using our central Enterprise Active Directory domain that we are using. This meant that we had half a dozen sizable Active Directory domains in the SSO's Identity Sources (Administration > Sign-On and Discovery > Configuration).

Removing the unnecessary Identity Sources resolved the issue.