Windows Server - "Not enough storage available error"
We have two IIS 7 servers that run in web garden. There's a copy of every site on each server. Files are synchronized using DFS (only source files). Images & big files are located on different server using share.
Atleast few time a week, we get error "Not enough storage available" when reading any file or processing a server code. IIS Server is unable to map path to a local disk (not just the one that synchronizes with DFS), there also seems to be a problem loading group policy permissions (we get various errors) and we can't open any share on any server on lan. There are also I/O errors when reading files from other custom applications. Disk speed and latency during this fall-out are both ok. If we try to copy a file or test a drive speed, usually works. Sometimes, we can't even run "my computer" or any other software (for example performance monitor). It can only start if we "run as administrator". There's more than enough storage available on every disk.
Most of the sites stop working (but not all). After server restart, everything works ok for a few days and then the problem reoccurs.
Servers are virtualized using VMWARE, there are 50 IIS pools (some are mem limited) and around 200 running sites on each server. Both servers have 16GB of RAM (only about 80% used). CPU is constantly between 30 - 60%.
We tried changing the IRPStackSize registry property to a higher number, but there were no changes. Problems still exists. We tried to disable antivirus (NOD) – no success.
Any hint? Maybe we are close to IIS limits (to many sites/pools)? Maybe 32bit OS and 16GB of RAM? Maybe there is some 'secret' registry settings... We don't think that everyday restart is long-term solution.
The Not enough storage available error has nothing to do with disk storage. It is about internal memory.
This error is quite easy to reproduce. Simply select a bunch of large files (images or DLL's or whatever), right-click and use the open-with option to open them all with Notepad. This should eat up allot of internal memory. When the swap-file nears it's maximum capacity, random services will start logging Not enough storage available to the eventlog. Often they will stay in a faulted state until they get restarted.
I have come across this issue with my own services, and had to add some extra error handling to force the service process to exit when such an error occurred on a thread. That way the service would restart and recover in an automated manner.
IIS usually handles this situation quite well (compared to an average windows service).
In order to find the process that causes this (usually a memory leak) you should use the performance monitor mmc snap-in, and record the memory usage of running processes (be selective when deciding what to record, since these logfiles can become quite large). Another option would be to use xperf, which is actually more suited for debugging low-level drivers and so, but can be helpful in this scenario as well.
If it turns out to be the w3wc service that's eating up all memory, you could use a tool such as Microsoft's all-new Application Insights (or any other IIS monitoring tool) to figure out which application pool, web-application or website is responsible. Then you can restrict memory usage for that particular pool, or schedule it to recycle gracefully.
IIS can usually cycle a pool in such a way that no one will notice (depending on how the app handles sessions, if it is custom built you might lose sessions when all static variables are flushed). Often a workaround like this is good enough, and expensive fixing of the bug can be avoided (disclaimer: that's not "my style" of solving problems, but most stakeholders love it that way).
32bit Operating system with 16Gb ram will effectively leave 12Gb's unused. The upgrade to 64bit OS should defenitly be considered.
Use of the PAE switch (Physical Adress Extension) could offer a temporary fix.
For info with regards to max memory which a OS can use along with the PAE switch info look at:
http://msdn.microsoft.com/en-us/library/windows/hardware/gg487503.aspx
Regards,