Identifying cause of too many CLOSE_WAIT in IIS

Solution 1:

OK I will post my findings here, just in case anyone needs it.

Around 10 hours before this issue started to happen, I had ran windows update and KB5005698 was installed. This update was installed on the 2 servers that support the android app.

Weirdly enough, the issue started at the same time on both servers, that's why I initially suspected it was an attack.

When the server wasn't on high load anymore, the issue stopped and I decided to migrate the web api from .net 5 to .net 6, I installed the server bundle and deployed it.

As the issue stopped before migrating .net version, nothing had changed so I just left it there.

Around 4 hours ago, I started getting alarms again, but this time it was because the web api was returning excessive http 500, but the number of connections were normal. So I decided to revert the app to the .net 5 version.

As soon as I did that, the number of connections started to increase and reached 5k more in just a minute and the timeouts were running free! I kept running iisreset and the same pattern was happening again.

So I swapped it again to .net 6 and no more connections increase but http 500s after a while.

Turns out the http 500 was an easy code fix so I fixed it and deployed again, targeting .net 6.

So no more high connections and everything seems to be working smoothly.

So I came to the conclusion that the issue is with KB5005698 and .net 5.

Deploying the same app targeting .net 6 fixed the problem.

After thousands of bad reviews and loss of revenue, it's all back again...

Lesson learned... I will never update the server again if I don't need to.

Hope it helps someone.