Is the system load tied directly to CPU usage?

It's a dual-core machine and the load is around 24 but CPU usage is bouncing around from 60-80% and never reaching 100% so how can there be that many processes waiting? I/O is pretty much nill since everything is being read from memory databases and the network is only using around 7mbit from a gigabit connection. Would adding additional cores reduce the load or is the CPU not the issue since it's not even topping out?

Solution 1:

No.

Try running nothing but Seti @ Home (or whatever its current incarnation is), and you'll see your server load pegged at 1.0, although your CPU would be pegged at 100% usage.

From the UPTIME(1) man page:

System load averages is the average number of processes that are either in a runnable or uninterruptable state. A process in a runnable state is either using the CPU or waiting to use the CPU. A process in uninterruptable state is waiting for some I/O access, eg waiting for disk. The averages are taken over the three time intervals. Load averages are not normalized for the number of CPUs in a system, so a load average of 1 means a single CPU system is loaded all the time while on a 4 CPU system it means it was idle 75% of the time.

For example, we have a server that does nothing but run Spamassassin with two Xeon CPUs (for a total of 4 virtual CPUs) It sometimes reaches 100% CPU utilization, but more often it uses a lot of disk IO by reading and writing to the Bayes database. As such, the system load is often very high at around 20-40 because there are many processes that are waiting a little while for the disk.

Solution 2:

Load average is the number of processes in runnable state: press "i" in top to limit it to these. This includes processes which are (R) running and (D) waiting for disk. Is it really not doing any IO? Has it not gone into swap?