Understanding virtual memory usage > swap + physical on Linux

I have a process that is reporting in 'top' that it has 6GB of resident memory and 70GB of virtual memory allocated. The strange thing is that this particular server only has 8GB physical and 35GB of swap space available.

From the 'top' manual:

   o: VIRT  --  Virtual Image (kb)
      The total amount of virtual memory used by the  task.   It  includes
      all  code,  data  and  shared  libraries  plus  pages that have been
      swapped out. (Note: you can define the STATSIZE=1 environment  vari-
      able  and  the VIRT will be calculated from the /proc/#/state VmSize
      field.)

      VIRT = SWAP + RES.

Given this explanation, I would expect the virutal memory allocation for a process to be limited to my swap + physical memory available.

According to 'pmap', the code, shared library, and shared memory sections of this process are all minimal - no more than 300M or so.

Obviously, the machine and the process are still functioning correctly (albeit slowly), so what am I missing here?

It may be demand zero memory which isn't in physical ram, or in the pagefile.

Some resources you may want to look at:

• http://www.gitam.edu/eresource/comp/gvr(os)/13.6.htm
• http://linux.derkeiler.com/Newsgroups/comp.os.linux.development.system/2006-11/msg00176.html

Does your application create a lot of empty memory pages? If so, your application might benefit greatly from:

• http://code.google.com/p/compcache/ (EXPERIMENTAL)

It allows you to compress and decompress in real-time memory pages. In turn, you are able to keep everything in RAM rather than swap to disk (very slow).

Here's a discussion of virt vs. resident memory:

https://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used

The discussion refers to Java processes, but is applicable to anything running under Linux. The main point with regard to virt is that the total includes a whole bunch of stuff that may never be used. Virt is something to look at for 32-bit OSes (since processes will hit limits on addressable space), but is largely not useful otherwise. As noted, the thing to pay attention to is the resident memory, which will be limited to available physical RAM and your swap.

This is likely because the process' address space is the size as you stated, but it is not really allocated by the OS.

From: http://lwn.net/Articles/428100/

In the process of trying to reach that goal of "low enough overhead and no significant latency," the Go developers have made some simplifying assumptions, one of which is that the memory being managed for a running application comes from a single, virtually-contiguous address range. Such assumptions can run into the same problem your editor hit with vi - other code can allocate pieces in the middle of the range - so the Go developers adopted the same solution: they simply allocate all the memory they think they might need (they figured, reasonably, that 16GB should suffice on a 64-bit system) at startup time.

So that's the unelegant way memory management is done sometimes - having a continous address space simplifies releasing unused mem.