What's using all my memory?

SOLUTION: There's a lot of stuff to read in this thread, so i'll just summarize here to help.

The problem is the VMWare ESX balloon driver which "inflates" the amount of memory "in use" that the OS sees in order to force the OS to dump caches and reduce working set size. This creates a large amount of "in use" physical ram that you can't account for by looking at the processes. You can look at the VMWare console performance graph for the VM's and see the balloon driver in use.

The solution to the performance was to raise the amount of reserved memory for the server, which limits the amount of memory the balloon driver can steal from you.

I have a 32-bit Windows 2003 Terminal Server that is running a local copy of SQL Server (full version, not express) and hosts a Visual Studio 2008 development environment for several developers. The machines ia Virtual Machine running on vSphere 4, with 4GB of memory allocated to it.

When I have 2 or 3 users logged in, it's using more than 4GB of committed memory, but I can't seem to find where most of this memory is being used. When I add up all the memory in Task Manager for all users, it only accounts for about 2GB at most. What's using the rest?

I've tried cycling the SQL server, that only takes the memory that I see allocated to SQL servr out. The same is true of visual studio, if I shut it down, it only takes the amount I see allocated in Task Manager out of the committed memory pool. I have also set the memory options for SQL Server to only use 512MB max.

I don't believe this is a system or file cache issue, since Windows doesn't include that memory in the committed statistics AFAIK.

It's driving me nuts. There should be more than enough memory for all these things, based on the amounts they seem to be using. How do I locate where this extra memory is being used?

Update

Regarding which memory statistics i'm looking at, none of them add up.

For example, right now with 1 user logged on and only running Visual Studio and the WebDev server, and SQL Server and IIS in the background.. I add up Private bytes and I get about 1049740K of memory, but task manager says commit charge is 1146M, leaving about 150Mb unaccounted for.

When 3 users are logged in, it's only about 2GB of private bytes (a lot of memory is shared because the users are just running Visual Studio).

VM Size doesn't add up either, as that is typically not that much different than private bytes. Although, oddly enough, the number is very different in Process Explorer for virtual memory than it is in Task Manager. PE shows virtual memory of SQL Server at 1,791,444K for SQL Server and 1,251,432K for Visual Studio, yet commit charge is only 1146M total for the machine.

There is no column for WS Size in Task Manager, but in PE it lists WS as both Private and Sharable, and those numbers are significantly less than the Commit charge, even added up.

UPDATE:

I know this question has a lot of detail, so maybe the question is not clear.

How do I find which processes are using all the committed memory in my system? The memory given by TaskManager and Process Explorer do not add up properly, and only show about 2GB of committed memory even though 4GB is committed.

Solution 1:

sql server and vmware tools fight for memory space in a vm. This is probably compounded by running terminal services and vs.

If you haven't already, go into the vm's settings and create a full 4Gb memory reservation under the resources tab. This'll prevent the baloon driver attempting to reclaim memory within the vm.

You may also want to configure the options within sql server to ensure it only allocates part of the system's ram. Maybe 2gb max.