100% CPU load on Ubuntu 10.04.3 LTS 64bit

I have trying to fix this issue for two days with no success. The server is a MySQL database server.

Hardware: DELL Poweredge 1950, 2x Intel Xeon Quad Core E5345 @ 2.33GHz, 16 Gb mem, 2x 146Gb SAS (software RAID1)

Software: Ubuntu 10.04.3 LTS, MySQL 5.1.41

Issue: while MySQL is not used and runs with no database, everything seems alright. As soon as I install a database, it has the reason to bring all 8 cores in 100% with low memory consumption. So, you can imagine the load average goes high (I saw 212 load average for the first time). The server doesn't become unresponsive, but you can see it's slow while browsing the project installed.

Additional info:

• The database used is not more than 24MB and it was moved from a server with less resources and a lot more larger databases. So it's not the database/project.
• my.cnf is not a reason also, as I used both default one and the one I use on the same distribution on another server.What is interesting is that mysql doesn't close any process and runs to the limit of the max_connections.
• Logs are quiet. Nothing there.
• I switched to this Ubuntu version after I suspected some problems in the newly Ubuntu 11.10 server. This one worked alright for an hour after I made a kernel upgrade to 3.0.1 (it was using the memory also)

I tested disk speed and seems alright.

Some more output on the running server:

dstat -cndymlp -N total -D total 3:

htop command:

Did anyone meet the same problem? Any fix you can think of?

The interesting part (in addition to MySQL being so busy...) is that majority of the CPU time seems to go to system, not user time.

Can you trigger this same kind of behaviour with some other program than MySQL? Try, for example, stress and make it consume the CPU with stress -c 128. For me that spawned 128 threads consuming 93% user, 7% system CPU (according to top). This in Fedora 15, anyway. If even stress makes your server consume mostly system time, something is very wrong.

Sometimes unexpected things like outdated server firmware can cause all kind of funky side-effects. Have you upgraded your server firmware? Also newish kernel features, like dynamic ticks can make things laggy under certain conditions. You may try disabling those by passing nohz=off as boot parameter to GRUB.