gMSA account authentication failure during password rotation

When our gMSA accounts are automatically rotated, we see login failures for around 1-10 minutes. This is particularly apparent for gMSA client accounts that connect to MS SQL server, but I think it happens for other gMSA accounts as well. MS SQL server is not running as a gMSA account, but our application uses gMSA to make a client connection to SQL. By default ManagedPasswordIntervalInDays is every 30 days, so we see this every month at the same time.

When I check the domain controller logs, I don't see any login failures for the gMSA user, but the SQL server logs the following error

SSPI handshake failed with error code 0x8009030c, state 14 while establishing a connection with integrated security; the connection has been closed. Reason: AcceptSecurityContext failed. The operating system error code indicates the cause of failure. The logon attempt failed [CLIENT: x.x.x.x]

From what I have found, this error usually indicates the wrong username/password combination.

This occurs on multiple clients, and each eventually starts connecting again after anywhere from 1-10 minutes. The clients don't all start connecting at the same time, but it seems to be randomly within that time window.

Initially I thought it might be related to AD replication of the changed password, so we modified the default inter-site replication interval to USE_NOTIFY to replicate immediately. If replication were the issue, I would expect to see login failures on DC's and I'm not seeing logon failures on DC's. I had also thought that maybe the SQL server is caching the authentication token, but if that were the case, I would expect to see all clients resolve at the same time (ie when the SQL server refreshed) Being that the clients each start working again at a different time, it doesn't appear to be on the SQL server side, but more likely something on the client side. Maybe caching the gMSA password or maybe something related to timeout and retry back offs.

Solution 1:

I found that this was due to the way the Windows service was configured. The Windows Service was configured as a standard service using a regular user account which happened to be gMSA account rather than Windows Service using a managed account.

This can be verified with:

>sc.exe qmanagedaccount ServiceName

[SC] QueryServiceConfig2 SUCCESS

ACCOUNT MANAGED : FALSE

This can be changed by running

sc.exe managedaccount ServiceName TRUE

After changing the Windows Service account type to be managed, initial testing shows that logins are now successful during the gMSA password rotation.