Used memory on Solaris 10
ZFS is likely using most of your memory as ARC cache. Should you want to know how your RAM is used, run this command as root:
# echo ::memstat | mdb -k
On Solaris 10 10/09 and newer, this displays something like this:
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 60569 236 16%
ZFS File Data 53270 208 14%
Anon 41305 161 11%
Exec and libs 5891 23 2%
Page cache 1190 4 0%
Free (cachelist) 7006 27 2%
Free (freelist) 212607 830 56%
Total 381838 1491
As you see, there is a line stating how much of the RAM is used to cache ZFS file data. Unfortunately, you are running an older Solaris 10 release so memstat doesn't show this ZFS statistic separately. It is included with the Kernel used memory which is confusing. A kernel shouldn't use 13 GB of RAM under normal circumstances.
Anyway, there is still a way to display the full ARC size on your server.
Just run this command:
# kstat zfs::arcstats:size
module: zfs instance: 0
name: arcstats class: misc
size 273469024
It shows that on my machine, 273 MB of RAM are currently used to handle the ZFS ARC cache. memstat shows that from these 273 MB, 208 MB are used as file cache. Up to these 208 MB of RAM could be released automatically on demand should applications need it.
Now lets look at processes memory usage. If you use the -Z option with prstat, it shows a summary per zone under the per process statistics. Here the global (and only) zone is using 185 MB of RAM. This should (roughly) match the sum of all processes rss column.
# prstat -Z
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
741 noaccess 129M 113M sleep 59 0 0:00:35 1,4% java/18
973 root 5148K 832K run 29 0 0:00:00 0,4% script/1
972 root 5072K 900K sleep 59 0 0:00:00 0,2% script/1
998 root 7148K 2812K cpu0 49 0 0:00:00 0,1% prstat/1
974 root 3456K 968K sleep 49 0 0:00:00 0,1% ksh/1
5 root 0K 0K sleep 99 -20 0:00:01 0,1% zpool-rpool/37
241 root 5400K 1608K sleep 59 0 0:00:00 0,0% VBoxService/5
77 root 7620K 2356K sleep 59 0 0:00:00 0,0% devfsadm/7
969 root 3372K 936K sleep 59 0 0:00:00 0,0% script/1
126 root 9664K 2844K sleep 59 0 0:00:00 0,0% nscd/31
480 root 9420K 2036K sleep 59 0 0:00:00 0,0% sendmail/1
11 root 9164K 7860K sleep 59 0 0:00:29 0,0% svc.configd/17
1 root 2504K 1432K sleep 59 0 0:00:00 0,0% init/1
413 root 15M 9644K sleep 59 0 0:00:00 0,0% fmd/19
377 root 6536K 2848K sleep 59 0 0:00:02 0,0% inetd/4
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
0 48 177M 185M 12% 0:01:24 2,5% global
These 185 MB correspond to sum of two lines in memstat output: "Anon" which is RAM used by applications to store data and "Exec and libs" which is the applications and their libraries code.
The memory is filled with unmapped pages of data read from disk. It's kept in memory because those files may be read again and keeping the data in memory saves a disk read. Free memory is forever wasted, so the computer tries to keep as little of it as possible.
For example, say you run a program. The program terminates. The program is still in memory, but those pages of memory are not used by any process since the program isn't running. If the system isn't under memory pressure, the pages are kept in memory. If the program runs again, this will save the effort of making it free just to have to allocate more memory for the program and then read it in again. And if the pages are needed for something else, it's still a win for the system because it's easier to move a page of memory directly from use to another than make it free only to make it used again.
Memory is not a saveable resource. If you leave 1GB free for an hour, anything you could have done with that data is forever lost.