How to explain memory usage on Windows Server outside the list of running processes

Solution 1:

JP, I believe your comment that this is an automated build / unit test server that is running less than stable code says it all. My guess is that this server needs to be rebooted frequently because of the unstable code running on it probably has memory leaks that are giving you the symptoms you are seeing. Memory leaks are not going to get listed on any of the process trees of running processes as they are memory that is grabbed by processes that no longer exist, and are still allocated, even though the process associated with them are long gone.

Get a frequent maintenance window to allow you to automagically reboot this machine, perhaps nightly, a couple times a week, or weekly depending upon the speed at which this behavior starts to exhibit performance issues.

We have Oracle Application Servers servers on Windoze a while back that needed to be rebooted twice daily. We kicked out the users at lunch time, and in the middle of the night every day. That was a few years back, with supposedly stable productions code from Oracle.

Solution 2:

Sorry to just paste some links - but we've been using some techniques to find memory leaks, using logparser and powershell, so I'll share.

http://blogs.msdn.com/debuggingtoolbox/archive/2008/08/22/powershell-script-chart-and-statistics-from-top-20-objects-leaking.aspx

(The Never Doubt Thy Debugger site is excellent): http://blogs.msdn.com/carloc/archive/2007/09/14/something-you-need-to-know-before-start-debugging.aspx

I'll also provide a link to Mark Russinovich's excellent insight to Memory Usage on Windows:

http://blogs.technet.com/markrussinovich/search.aspx?q=memory&p=1

Short of opening actual Microsoft cases, the Performance Team site has a lot of good info, too: http://blogs.technet.com/askperf/Default.aspx?p=2