10 GB of RAM unaccounted for by top

Solution 1:

Your smem -tw output shows that your kernel is consuming over 9 GB of dynamic memory:

danslimmon@bad-server:~$ smem -tw
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory      12857576    2887440    9970136
userspace memory           17661400    1272468   16388932
free memory                  351592     351592          0
----------------------------------------------------------
                           30870568    4511500   26359068

So, some kernel modules are consuming much memory. Prime candidate are closed source blobs, as NVIDIA kernel driver.

Can you post the output of lsmod and cat /proc/slabinfo ?

Solution 2:

You're apparently aware of the shared memory issue when interpreting RSS figures, so I won't expand on that.

Linux systems vary widely (e.g. think of embedded systems), but I think the list of processes you are getting from top looks incomplete, with few system processes. You'll notice you have a init process with PID=1, and nothing else up to PID=869. I think you'll find that many of those PIDs are associated with live kernel processes. It's possible that something in there is using a lot of RAM. @shodanshok's answer certainly points that way. You'd see those process IDs in /proc if you have a procfs mounted.

I'm wondering if you have an in-memory file system with substantial content (e.g. /tmp). What does df -h look like?

If you have a procfs mounted on /proc, you'll see a complete list of process IDs in there. Also /proc/$PID/maps lists regions of physical memory that a process maps into its memory space, though interpreting it is far from trivial.