Linux Kernel (/proc/meminfo) Not Reporting All Memory

I have two machines that are identical in every way (hardware, distribution, application workload, /proc/sys/vm settings, etc), except for the the kernel version. One is 2.6.33 and the other is 3.1.0. The 2.6.33 machine performs significantly better for the given workload (both are I/O bound mostly on reads). I've noticed the Cached/Active/Active(file) is quite a bit smaller on the 3.1.0 machine (over 20GB) and doesn't seem to be accounted for in any other metric being reported. This is also confirmed by the fact that there are a lot more reads happening on the 3.1.0 machine (due to less memory available for page cache most likely). I've looked at every tunable, /proc/buddyinfo for fragmentation, /proc/slabinfo for slab use (uses a a few GB more, but not ~20GB worth), and nothing looks off. Any ideas would be much appreciated.

This is from the machine running 2.6.33 kernel and things look normal.

> cat /proc/meminfo
MemTotal:       74372248 kB
MemFree:          200492 kB
Buffers:            2976 kB
Cached:         65324256 kB
SwapCached:            0 kB
Active:         32949324 kB
Inactive:       32689844 kB
Active(anon):     287904 kB
Inactive(anon):    27272 kB
Active(file):   32661420 kB
Inactive(file): 32662572 kB
Unevictable:       19832 kB
Mlocked:           19832 kB
SwapTotal:       8393952 kB
SwapFree:        8393952 kB
Dirty:              8324 kB
Writeback:             0 kB
AnonPages:        332036 kB
Mapped:            12576 kB
Shmem:               304 kB
Slab:            8217640 kB
SReclaimable:    7859644 kB
SUnreclaim:       357996 kB
KernelStack:        4592 kB
PageTables:        10652 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    45580076 kB
Committed_AS:     934328 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      314652 kB
VmallocChunk:   34359294955 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6384 kB
DirectMap2M:     2080768 kB
DirectMap1G:    73400320 kB

This is from the machine running 3.1.0 kernel. Note the Cached/Active sizes are over 20G smaller and don't seem to be made up in any other metric.

> cat /proc/meminfo
MemTotal:       74370628 kB
MemFree:          415680 kB
Buffers:          384916 kB
Cached:         42088392 kB
SwapCached:            0 kB
Active:          5636160 kB
Inactive:       37170092 kB
Active(anon):     300656 kB
Inactive(anon):    36620 kB
Active(file):    5335504 kB
Inactive(file): 37133472 kB
Unevictable:       19880 kB
Mlocked:            7616 kB
SwapTotal:       8393956 kB
SwapFree:        8393956 kB
Dirty:              6524 kB
Writeback:             0 kB
AnonPages:        354084 kB
Mapped:            14588 kB
Shmem:               472 kB
Slab:           11419580 kB
SReclaimable:    9835632 kB
SUnreclaim:      1583948 kB
KernelStack:        2944 kB
PageTables:        12084 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    45579268 kB
Committed_AS:    1006028 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      290448 kB
VmallocChunk:   34321698548 kB
HardwareCorrupted:     0 kB
AnonHugePages:    135168 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      100592 kB
DirectMap2M:     6180864 kB
DirectMap1G:    69206016 kB

Solution 1:

It turns out the memory was being consumed by XFS metadata buffers. They moved from the kernel page cache to XFS specific buffers in the 2.6.39 kernel. The patch that changed the behavior can be found here:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0e6e847ffe37436e331c132639f9f872febce82e

The performance difference is due to a different balance between file data and metadata caching as a result of the XFS change.