24 Core Server Performance

Solution 1:

cat /proc/cpuinfo

are all CPUs actually recognized? I don't know about CentOS, but, some kernels turn on SMP but limit the number of cores to 8 as a default.

Does top show processes on each cpu? shift-O, j

Is your application not multithreaded?

-- It does recognize all 24 cores, so, that theory is effectively negated. The only other simple thing I can see is that your cpu powersave governor MIGHT be running the cpus at 800mhz rather than the 2.6 ghz.

What does your application do? typical webapp? or is there a part that is not able to be multithreaded? Is mysql configured to use more cores (thread_concurrency)? When you watch the machine running, what is slower? network connection time? CPU throughput? disk access? sysctl.conf settings migrated over from the other box? any tweaks/modifications to mysql/apache/php configs that didn't get moved over?

can you paste: cat /proc/mtrr

Solution 2:

Cache memory issues. You need to be careful about how those cores are socketed and how the tasks spawn across them. I had a server go from fast 6-core to slow 8-core in a 12-core (2-socket) system. So, just having lots of cores and RAM does not mean that you should just blindly make use of them all.

I would suggest using your 24-core system to run a virtualisation host like KVM. Then, it may be easier to partition your workload across a number of smaller VMs. You can pin specific cores to specific VMs. So, if your 24-core system is a 4-socketed 6-core or a 6-socketed 4-core, you can have fine-grained control over the performance. There will be a difference in performance between the two.

Cheers.

Solution 3:

Put a newer MySQL on it - 5.1 at least, or better, Percona's XtraDB. Make sure you're using InnoDB. MySQL 5.0 scales negatively (i.e. performance decreases) with more than 8 cores in my experiences.

Solution 4:

Depending on the workload and kernel version, increasing the number of CPU may very well decrease performance (semaphore implementation of the Big Kernel Lock with heavy POSIX file locking usage comes to mind, this is something that has been observed in RHEL/CentOS 5).

Have you checked whether you get better performance by reducing the number of CPUs? (maxcpus=)

Otherwise, you should profile your application/system to track down the bottleneck. If this is CPU-bound, OProfile should help. SystemTap also comes to mind.