What used the Linux memory? Low cache, low buffer, not a VM

Memory on Linux can be a strange beast to diagnose and understand.

Under normal operation most, if not all, your memory will be allocated to one task or another. Some will be allocated to the currently running foreground processes. Some will be storing data cached from disk. Some will be holding data associated with processes that aren't actively executing at that one specific moment in time.

A process in Linux has its own virtual address space (VIRT in the output of top). This contains all the data associated with the process and can be considered how "big" the process is. However it is rare for all that memory to be actively part of the "real" memory map (RES in the output of top). The RES, or resident memory, is the data which is directly accessible in RAM at the point in time. Then there is also shared memory (SHR) on top of that. That can be shared between multiple instances of the same process. So the memory in use by a process is at any one point in time RES plus SHR, but if there is more than one instance of the process using the shared memory the usage is RES plus RES plus RES ... plus SHR.

So why the difference between RES and VIRT? Surely if a process has a block of allocated memory it's allocated memory, isn't it? No. Memory is allocated in pages, and pages can be either Active or Inactive. Active ones are what are in RES. Inactive are "the rest". They can be pushed to one side as they aren't being accessed at the moment. That means they can be swapped out to disk if memory gets tight. But they don't just go straight to disk. First they sit in a cache. You don't want to be swapping all the time, so there's a buffer between the application and the swap space. Those buffers are constantly changing as the swapper selects a different process to execute and different pages become active and inactive. And all that's happening way to fast for a mere human to keep up with.

And the on top of all that there is the disk buffers. Not only does the inactive memory go to a cache, but when that cache gets swapped to disk it first goes to a disk buffer to be queued up for writing. So that's a second layer of cache in the mix. And those disk buffers are also used by other parts of the system for general IO buffering. So they are constantly changing too.

So what you are seeing in things like top and free etc are either instantaneous snapshots of the current state of the machine, or aggregated statistics over a period of time. By the time you have read the data it's out of date.

Any one process can access large amounts of the memory, but it's seldom sensible to do so. It can't be accessing all the memory at once anyway, so memory that it's not currently looking at gets moved to cache unless it's specifically flagged as being "locked in core".

So the amount of memory "used" by an application and the amount of memory it "has" are two completely different things. Much of an applications data space is actually in the cache, not in the "core" memory, but since the cache is in RAM most of the time it's instantly available and just needs "activating" to become "core" memory. That is unless it's been swapped out to disk, when it then needs unswapping (which might be fast if it's in the buffer).

Due to the high speed nature of the beast and the fact that the figures are always changing, the numbers may even change part way through calculating what they are, so it's never possible to exactly say "this is how much memory is in use" from a user's perspective. The meminfo is a snapshot in time provided by the kernel, but since it's the kernel that's executing then it's not necessarily showing the real state of any one processes memory usage, as no process is actively executing at that time - it's between processes.

Like I said, it's all very confusing.

But at the end of the day it really doesn't matter. What matters is not how much memory you have "free", but how much swap space you have used, and how often swap space is being accessed. It's swapping that slows a system down, not lack of memory (though lack of memory causes excess swapping). If you have lots of used memory, but you're not using any (or very little) swap space, then things are normal. Free memory in general isn't desirable, and is often purely transitional anyway, in that it was in use for one purpose, but hasn't yet been allocated for another - for instance it was cache memory, and it's been swapped to disk, but it hasn't yet been used for anything else, or it was disk buffers, the buffers have been flushed to disk, but no application has requested it for cache yet.