Linux top command. Memory usage
I am testing my web server with Jmeter. I launch 40 users test, then dump top command. What i see, is 40 (+1 host) apache processes. Each process uses appr. 7mb of RES memory. But 7*40 is 280 mb of memory. But top shows that there are 508mb total and 345mb free. So only 163mb used... Why i have such strange stuff?
top - 04:49:24 up 1 day, 10:49, 1 user, load average: 0.28, 0.18, 0.16
Tasks: 107 total, 2 running, 105 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.4%us, 0.4%sy, 0.0%ni, 97.6%id, 0.5%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 508132k total, 162428k used, 345704k free, 28340k buffers
Swap: 916476k total, 21800k used, 894676k free, 63480k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9816 www-data 20 0 69232 7260 3232 S 1.9 1.4 0:00.69 apache2
9890 www-data 20 0 69232 7260 3232 S 1.9 1.4 0:00.06 apache2
9900 www-data 20 0 69232 7260 3232 S 1.9 1.4 0:00.04 apache2
9906 www-data 20 0 69232 7256 3232 S 1.9 1.4 0:00.04 apache2
9908 www-data 20 0 69232 7256 3232 S 1.9 1.4 0:00.06 apache2
1 root 20 0 2836 760 460 S 0.0 0.1 0:01.50 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.99 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
6 root 20 0 0 0 0 S 0.0 0.0 0:04.20 events/0
7 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuset
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm
12 root 20 0 0 0 0 S 0.0 0.0 0:00.45 sync_supers
13 root 20 0 0 0 0 S 0.0 0.0 0:00.62 bdi-default
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0
15 root 20 0 0 0 0 S 0.0 0.0 0:05.89 kblockd/0
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpid
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_hotplug
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata_aux
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata_sff/0
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmmcd
25 root 20 0 0 0 0 S 0.0 0.0 0:00.08 khungtaskd
26 root 20 0 0 0 0 S 0.0 0.0 0:08.30 kswapd0
27 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/0
29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ecryptfs-kthrea
30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/0
35 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pciehpd
37 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
38 root 20 0 0 0 0 S 0.0 0.0 0:00.02 scsi_eh_1
41 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kstriped
42 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpathd/0
43 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kmpath_handlerd
44 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksnapd
45 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kondemand/0
46 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kconservative/0
239 root 20 0 0 0 0 S 0.0 0.0 0:03.95 mpt_poll_0
240 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mpt/0
241 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_2
258 root 20 0 0 0 0 S 0.0 0.0 0:05.60 jbd2/sda1-8
259 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit
304 root 20 0 2392 68 68 S 0.0 0.0 0:00.04 upstart-udev-br
306 root 16 -4 2440 72 68 S 0.0 0.0 0:00.06 udevd
414 root 18 -2 2328 64 60 S 0.0 0.0 0:00.00 udevd
415 root 18 -2 2328 64 60 S 0.0 0.0 0:00.00 udevd
518 root 20 0 0 0 0 S 0.0 0.0 0:02.87 vmmemctl
526 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
556 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kgameportd
618 syslog 20 0 33976 440 440 S 0.0 0.1 0:00.83 rsyslogd
689 root 20 0 1856 84 80 S 0.0 0.0 0:00.00 getty
693 root 20 0 1856 84 80 S 0.0 0.0 0:00.00 getty
697 root 20 0 1856 84 80 S 0.0 0.0 0:00.00 getty
698 root 20 0 1856 84 80 S 0.0 0.0 0:00.00 getty
701 root 20 0 1856 84 80 S 0.0 0.0 0:00.00 getty
703 memcache 20 0 54192 128 128 S 0.0 0.0 0:07.77 memcached
705 root 20 0 2456 268 204 S 0.0 0.1 0:00.42 cron
706 daemon 20 0 2316 0 0 S 0.0 0.0 0:00.00 atd
722 root 20 0 5640 360 256 S 0.0 0.1 0:00.53 sshd
753 mysql 20 0 153m 456 32 S 0.0 0.1 0:20.57 mysqld
9280 root 20 0 2780 1408 1064 S 0.0 0.3 0:00.05 login
9292 zim32 20 0 8828 6068 1536 S 0.0 1.2 0:00.26 bash
9324 root 20 0 7268 2968 2180 S 0.0 0.6 0:03.50 mc
9326 root 20 0 6252 3544 1588 S 0.0 0.7 0:00.21 bash
9735 root 20 0 0 0 0 S 0.0 0.0 0:00.00 flush-8:0
9808 root 20 0 68892 8624 4828 S 0.0 1.7 0:00.15 apache2
9814 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:01.03 apache2
9827 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.97 apache2
9842 www-data 20 0 69232 7264 3236 S 0.0 1.4 0:00.40 apache2
9844 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.40 apache2
9870 www-data 20 0 69232 7264 3236 S 0.0 1.4 0:00.22 apache2
9872 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.23 apache2
9877 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.18 apache2
9878 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.18 apache2
9888 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.07 apache2
9889 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.06 apache2
9891 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.08 apache2
9892 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.05 apache2
9893 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.06 apache2
9894 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.08 apache2
9895 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.06 apache2
9896 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.04 apache2
9897 www-data 20 0 69232 7248 3228 S 0.0 1.4 0:00.06 apache2
9898 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.03 apache2
9899 www-data 20 0 69232 7260 3236 S 0.0 1.4 0:00.06 apache2
9901 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.04 apache2
9902 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.04 apache2
9903 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.03 apache2
9904 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.08 apache2
9905 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.05 apache2
9907 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.05 apache2
9909 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.08 apache2
9911 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.04 apache2
9912 www-data 20 0 69232 7248 3228 S 0.0 1.4 0:00.04 apache2
9913 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.06 apache2
9914 www-data 20 0 69232 7260 3232 S 0.0 1.4 0:00.04 apache2
9915 www-data 20 0 69232 7260 3232 R 0.0 1.4 0:00.04 apache2
9916 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.04 apache2
9917 www-data 20 0 69232 7256 3232 S 0.0 1.4 0:00.06 apache2
9918 www-data 20 0 69232 7248 3228 S 0.0 1.4 0:00.02 apache2
9919 root 20 0 2632 1068 816 R 0.0 0.2 0:00.02 top
Solution 1:
It's a case of top reporting memory usage per process as if it where the only process running. In reality all the apache processes have shared memory (common linked libraries), and this shared memory is being reported for each process so the total usage across all apache processes appears to be more than it really is.
There are numerous posts that discuss how to underdstand and interpret the output of top and ps.
process memory usage on Linux is a complex matter; you can't just run ps and know what is going on. This is especially true when you deal with programs that create a lot of identical children processes, like Apache. ps might report that each Apache process uses 10 megabytes of memory, when the reality might be that the marginal cost of each Apache process is 1 megabyte of memory
Solution 2:
You forgot to calculate the SHR, or Shared portion.
Try (7*40) - (3*40) = 160, which quite closely matches your used memory.
Shared memory is the memory which is shared among the processes; the Apache libraries and stuff like that.