Where did my RAM go?
Note: before you jump in too fast, yes I read linuxatemyram.com!
I have a server with 64GB RAM.
free -m says that my RAM is full, and it's not because of disk caching:
total used free shared buffers cached
Mem: 64458 64117 340 201 67 331
-/+ buffers/cache: 63719 739
Swap: 1532 383 1149
However, top ordered by memory usage does not add up to 64GB:
KiB Mem: 66005116 total, 65652464 used, 352652 free, 67512 buffers
KiB Swap: 1569780 total, 392656 used, 1177124 free. 337464 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6258 mysql 20 0 38.665g 0.033t 4924 S 1.3 54.3 482:26.21 mysqld
2293 root 20 0 165896 102116 101964 S 0.0 0.2 0:43.53 systemd-journal
4909 root 20 0 377548 57840 57548 S 0.0 0.1 0:18.47 rsyslogd
26639 www 20 0 650076 53348 32968 S 0.0 0.1 11:32.27 php-fpm
26640 www 20 0 648344 51912 32984 S 0.0 0.1 11:37.43 php-fpm
26642 www 20 0 648600 51472 32580 S 0.0 0.1 11:37.16 php-fpm
26669 www 20 0 648148 50696 31988 S 0.0 0.1 11:35.24 php-fpm
26643 www 20 0 648452 50616 31628 S 0.0 0.1 11:36.19 php-fpm
26641 www 20 0 648620 50496 31340 S 0.0 0.1 11:36.51 php-fpm
28121 www 20 0 648620 48820 29660 S 0.0 0.1 11:35.75 php-fpm
27231 www 20 0 647508 48804 30760 S 0.0 0.1 11:35.61 php-fpm
28029 www 20 0 648044 48752 30172 S 0.0 0.1 11:37.20 php-fpm
28117 www 20 0 647868 48700 30296 S 0.0 0.1 11:36.45 php-fpm
28122 www 20 0 648340 48568 29676 S 0.0 0.1 11:35.73 php-fpm
8569 www 20 0 649028 40268 20704 S 0.0 0.1 11:31.50 php-fpm
10126 www 20 0 648432 39420 20700 S 0.0 0.1 9:58.52 php-fpm
22386 www 20 0 647996 39400 20868 S 0.0 0.1 11:25.00 php-fpm
9643 www 20 0 647976 39220 20704 S 0.0 0.1 11:29.23 php-fpm
23077 www 20 0 647852 39084 20692 S 0.0 0.1 11:11.80 php-fpm
10139 www 20 0 647580 38808 20692 S 0.0 0.1 9:59.94 php-fpm
6326 www 20 0 647368 38396 20696 S 0.7 0.1 8:32.34 php-fpm
4727 www 20 0 646128 37304 20692 S 0.0 0.1 8:30.20 php-fpm
5459 www 20 0 645988 37156 20688 S 0.0 0.1 7:15.13 php-fpm
2173 www 20 0 645240 36408 20684 S 0.0 0.1 4:39.13 php-fpm
20752 www 20 0 644536 35428 20680 S 0.0 0.1 4:29.78 php-fpm
5396 www 20 0 644468 35324 20692 S 0.0 0.1 4:14.65 php-fpm
17558 www 20 0 642668 33816 20740 S 0.0 0.1 1:28.34 php-fpm
28133 www 20 0 642780 33636 20704 S 0.0 0.1 0:49.88 php-fpm
10925 www 20 0 479584 29264 11212 S 3.0 0.0 0:00.09 php
26632 root 20 0 552136 26072 19468 S 0.0 0.0 0:37.74 php-fpm
4946 named 20 0 697996 18748 2104 S 0.0 0.0 3:46.96 named
15609 apache 20 0 2137056 8120 1592 S 0.0 0.0 0:56.18 httpd
8584 root 20 0 133432 4864 3700 S 0.0 0.0 0:00.08 sshd
MySQL uses 54.3% alone, this is perfectly normal as it has an innodb_buffer_pool_size of 32G. The other processes' memory usage add up to roughly 2.8%, that's a total of 57.1%.
Where are the 32% remaining?
Edit: contents of /proc/meminfo:
MemTotal: 66005116 kB
MemFree: 353272 kB
Buffers: 66328 kB
Cached: 736620 kB
SwapCached: 11348 kB
Active: 34396680 kB
Inactive: 2651132 kB
Active(anon): 34223240 kB
Inactive(anon): 2228020 kB
Active(file): 173440 kB
Inactive(file): 423112 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 1569780 kB
SwapFree: 1177448 kB
Dirty: 328 kB
Writeback: 0 kB
AnonPages: 36234364 kB
Mapped: 125208 kB
Shmem: 206396 kB
Slab: 28058904 kB
SReclaimable: 28010224 kB
SUnreclaim: 48680 kB
KernelStack: 2760 kB
PageTables: 94780 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 34572336 kB
Committed_AS: 38572348 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 382304 kB
VmallocChunk: 34359353572 kB
HardwareCorrupted: 0 kB
DirectMap4k: 9000 kB
DirectMap2M: 2054144 kB
DirectMap1G: 67108864 kB
Output of slabtop:
Active / Total Objects (% used) : 147380425 / 147413026 (100.0%)
Active / Total Slabs (% used) : 7005839 / 7005839 (100.0%)
Active / Total Caches (% used) : 71 / 144 (49.3%)
Active / Total Size (% used) : 27615020.12K / 27627490.91K (100.0%)
Minimum / Average / Maximum Object : 0.01K / 0.19K / 16.12K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
146851887 146851887 12% 0.19K 6992947 21 27971788K dentry
124936 124936 100% 0.07K 2231 56 8924K Acpi-ParseExt
105144 105144 100% 0.10K 2696 39 10784K buffer_head
49920 49172 98% 0.06K 780 64 3120K kmalloc-64
29916 29916 100% 0.11K 831 36 3324K sysfs_dir_cache
29856 29661 99% 0.12K 933 32 3732K kmalloc-128
21450 21128 98% 0.18K 975 22 3900K vm_area_struct
19328 19328 100% 0.03K 151 128 604K kmalloc-32
18258 13383 73% 0.93K 537 34 17184K ext4_inode_cache
17952 11651 64% 0.04K 176 102 704K ext4_extent_status
16828 6513 38% 0.55K 601 28 9616K radix_tree_node
14400 13996 97% 0.06K 225 64 900K anon_vma
11645 7903 67% 0.05K 137 85 548K shared_policy_node
10710 7006 65% 0.19K 510 21 2040K kmalloc-192
10608 10608 100% 0.04K 104 102 416K Acpi-Namespace
9728 9728 100% 0.01K 19 512 76K kmalloc-8
...
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
146851887 146851887 12% 0.19K 6992947 21 27971788K dentry
You say it's not because of disk caching, but clearly it is.
My bet is that you have code that makes lots of fetches for files that do not exist and you get a ton of negative caching. Linux will remove these entries if it's under memory pressure, so it may not be anything to worry about. For example, like in this NSS issue.
It might be some internal kernel structures and filesystem/directory related stuff, still. This is also perfectly normal even though confusing; try to see what is the output from slabtopand cat /proc/meminfo.