Why does systemctl show high memory usage when top and htop show much less?

Solution 1:

The tl;dr story is that systemctl more thoroughly accounts for the memory a process is using, and reports on the total memory usage of an entire service, while top gives a more lenient accounting of the memory used by a single process. For a service like apache2 which runs several processes, systemctl can be expected to show much higher memory usage than top.

Memory management in *nix is extremely complex because RAM is the limiting resource for most operations, so a lot of work has gone into making the most use of it. As such, there are different categories of memory or memory usage. I was going to list them all, but it gets kind of insane pretty quickly.

I can list some of the major categories, though. The number one distinction is RAM versus Disk. "Memory" just means a place to store data. In a lightly loaded system, all data a program needs, including the program code itself, is in RAM. As more programs want access to more data, "memory" expands to include data stored on disk. Moving "memory" between RAM and Disk is highly optimized (and complex), but since the kernel can do it, it does it in a lot of ways.

So, here is a simplified, incomplete list of categories of memory usage

• What we all thing of as memory usage: data stored in RAM that exists nowhere else.
• The majority of memory usage: data stored in RAM for use by the CPU but which actually just a copy of something stored on Disk, for which the Disk is the authority of what the data should be. For example, the operating system code.
• Data that a program stored in RAM but was not using, so the kernel wrote it to disk temporarily to free up RAM for someone who needed it right now. This is called "Swap" because memory is swapped between RAM and Disk and back.

On top of those categories, we also have:

• Shared memory, which is data multiple processes use at the same time. If you run 2 copies of the same program, the program code does not need to be duplicated, it can be stored in RAM once and the 2 processes can share it.
• Reserved memory. The program can ask the kernel for some memory to use to store data, but until it actually starts using it, it is merely reserved and does not take up any space at all (outside of the kernel memory management data structures).
• Virtual memory. This is really just the allocation of memory addresses to the program, making a convenient abstraction for a lot of things. It is often huge (my laptop is currently using over 5,000 GiB of Virtual Memory) and its size/usage is of little consequence.

So when you ask "how much memory is this program using?" there is no one answer. There really is not any good answer to such a vague question. (See this article for 6 different definitions of the question along with some discussion about why they are hard to answer.) So different programs give you different answers, often with labels attached.

Programs like top try to answer the question with something like "how much RAM would be freed up if I quit this program". The Linux top man page provides a lot of detail about how it categorized memory and what it displays.

On the other hand systemctl tries to account for all the memory a service is using, both in RAM and Swap and to some extent shared and file backed, in order to be able to automate limits on resource usage and go as far as killing a program if it uses too much memory. It uses cgroups to provide much more extensive accounting for the memory a process uses, and that is described in great detail in section 2.2 "Accounting" of this article.

So even if the service is only running one process, because systemctl includes in the total memory usage of a process any shared pages it loaded first (before any other process), any Swap it is using, any memory cache (such as file system or network buffers) it is using, and probably some other stuff, it can report a higher memory usage than top. When a service is running more than once process, then obviously you would expect memory usage to go up even further.