How to diagnose ballooning OS X kernel_task and wired memory usage?

I have a very strange issue, which I'm having a hard time diagnosing as to the root cause.

I have a Mac Pro (2008, 8-core 2.8 GHz, 8800GT) with 14 GB of RAM (recently upgraded because of this issue!).

When I boot my system and log in, vm_stat / top / Activity Monitor will show that kernel_task has about 150 MB allocated, and the machine has about 800 MB of Wired memory being allocated.

Even initially, 800 MB seems an awful lot of wired memory to be allocated with no applications running - but, it gets worse. (NB: Wired is locked, unswappable memory)

After a very short time, sometimes triggered by something as simple as launching a terminal, kernel_task will balloon to 8-900 MB of Real Mem (RSIZE), and Wired Memory will accelerate to 1.6 GB (implying that all the extra memory requests are for wired RAM in the kernel).

If I quit everything (I.E: no running applications, bar an activity monitor or terminal to view top), there is no appreciable reduction in either kernel_task RSIZE, or Wired Memory usage. Going the opposite way, and loading the system with tasks also shows that wired memory does not get reduced - and that importantly it is not reduced in preference to heavy swapping.

If I log out and log back in again, it reduces a bit (450 MB kernel_task, 1.28 GB Wired), but not back to the start.

I'm not running any wacky kexts - and futhermore, kextstat shows no huge memory allocations there; the largest being com.apple.nvidia.nv50hal at about 4 MB of Memory.

The machine feels overall more sluggish when this has happened - unsurprisingly because such a huge amount of RAM has been marked as non-pageable.

So I have a few questions:

1) Is there a good way to diagnose what has allocated all of this wired memory? It's often over 2 times the kernel_task size, running no applications. The real memory total doesn't seem to add up - it seems that there is a bunch of RAM that isn't being accounted for anywhere.

2) What is happening to cause the kernel to suddenly require 6 times as much memory?

Solution 1:

To investigate why the kernel uses more memory than usual, you can use different tools.

1. Run Activity Monitor to check which processes using the most of memory, so it's either a kernel_task, not any other task using more memory than usual (then consider killing it).
2. Run in Terminal vm_stat 1 to see real-time memory statistics and if your memory is really increasing every second.
3. Run fs_usage (as root) tool to monitor system calls and page faults in real-time.

4. To check the sum of dirty/anonymous allocations of multiple processes run in Terminal:

   sudo footprint -all -categories -swapped -collapseSharing
   

   It would gathers memory information such as how much it was swapped (per user or kernel memory).

5. Further more, if you think it's kernel which is using most memory, try zprint tool:

   sudo zprint -t -s | head -n20
   

   It will show the information about kernel zones

If you'd like to force disk cache to be purged (to free up some memory), you may try:

sync && sudo purge

See also: How to investigate high kernel task memory usage? at AD SE

Solution 2:

Kernel extensions are just one of many, many, many code fragments that can be executed by the operating system without your knowledge. I have a small Python based utility called Consultant’s Canary that will help you find quite a few of them:

• Archived Link
• Documentation Link 1
• Documentation/Download link 2

If that doesn't turn up any potential culprits, then I'd say boot from a clean installation and see if you can reproduce the problem there.