How to tell which process is hogging my CPU when they don't add up to 100%?

Ubuntu's System Monitor applet shows 100% CPU usage continuously. If I click it, the resources tab shows it at 100% continuously, too. If I go to processes, though, to find out which process is the culprit, there is nothing above 10%. If I run top there is nothing above 10%. The individual processes do not add up to 100%. I try killing lots of processes, but the overall usage continues to be 100%. How can I find out what's hogging the CPU?

This is an unusual situation on a computer I use daily, which is never anywhere near 100% CPU unless I'm doing something that requires it (like loading 32 Firefox tabs), after which it goes back to a normal idle level. It's not a new install or anything. There is no reason the processor should be maxed out. I'm not sure when it started or if I changed something that caused it to happen.

Normally I would use top or System Monitor and find the process that had gone out of control, but I can't find anything with those tools this time. It persists after reboots and everything.

And the processor is obviously hot, so it's not an erroneous reading.

Update: I tried killing every process, one at a time, until the problem went away, and killing vino-server finally fixed it, even though that process never went above 5%. I had enabled Remote Desktop a few days ago (and have obviously now disabled it).

But the question remains: How did a single process manage to use 100% CPU while top only showed that process at 5%? How do I identify culprits like this in the future?

Looks like I'm not the only one who's had this problem:

Still a problem in both jaunty & karmic. Interestingly, both System Monitor & htop do not show the sum of individual processes being anywhere near 100% cpu.

When you say "nothing above 10%" do you mean "no one thing above 10%"? It only takes 11 things at ~9% to hit 100% in total.

Also, some of the measured used CPU time could be I/O wait time - time in which a process could be doing something but isn't because it is waiting for an I/O device to respond (i.e. it has made a disk request and the disk has not yet found the right bit of data and returned it). In top, this can be seen on the CPU line(s) as the %wa measure. The I/O wait state is a little odd - the CPU isn't actually busy doing anything (so if another process wanted to do some computation it would be able to) but it does count towards the system load figures as system load represenets more than just the CPU activiy.

It would be helpful to add the output of top -n 1 to your question.

Also, if your kernel is recent enough to have the IO accounting feature (if you are using a recent release of Ubuntu it should be) and it is turned on, you can see what processes are actively performing I/O operations by running iotop -n 1 -ob (iotop might not be installed by default, if you don't have it try install with apt-get install iotop or by finding it in the GUI package manager.

I had a similar problem. The processor was chugging along on full, and none of the processes in System Monitor showed anything. I found one answer by using the "top" command in a terminal window. It showed that apt was hogging my cpu for no good reason. So why didn't System Manager show this? Turns out there's an option under View for All Processes; mine was checked as My Processes, so I never saw the culprit. Hope this helps.