How to find the cause for growing server load

I'm having load problems with my server and even though I'm a somewhat experienced Linux admin I'm out of ideas now.

The problem is a slowly but steadily increasing load on the server without any apparent cause.

The Server is a AMD Athlon(tm) 64 X2 Dual Core Processor 6000+ with 6GB RAM. It is running Debian Stable with Linux gir 2.6.26-2-amd64 #1 SMP Wed Aug 19 22:33:18 UTC 2009 x86_64 GNU/Linux.

The server basically runs Lighttpd, several FastCGI PHP processes and a MySQL database. Typical webserver tasks.

The CPU is never really fully used up and memory is mainly used for buffers and cache which is fine. I tried to restart the various services to see if one of them would decrease the load again, but without luck.

Here are graphics showing load, CPU and IOStat:

So, question is: What could cause a slowly but ever increasing load? And how do I find out what's responsible?

Update: I forgot to mention, when I reboot the server, the load will be down to around 0.3 to 0.6 and will start to climb up again slowly over the next weeks.

Each zombie process adds 1.0 to the load. You might be seeing an accumulation of zombies.

I found an excellent hint in answer to a different question.

Looking for processes in state 'D' shows four PHP processes that seem to hang for quite a while corresponding to the "steps" in the load curve:

#> ps aux | awk '$8 ~ /D/  { print $0 }'
wiki      6651  0.0  0.0      0     0 ?        D    Oct04   0:41 [php-cgi]
bugs      6731  0.0  0.0      0     0 ?        D    Oct27   0:14 [php-cgi]
manpages  7536  0.0  0.0      0     0 ?        D    Oct30   0:21 [php5-cgi]
wiki     23847  0.0  0.0      0     0 ?        D    Oct06   1:32 [php-cgi]

So these seem to be the problem. I now need to find out while those processes hang and how to fix it. Thanks everyone.

My guess is that server is IO starved , maybe you should add the iotop stats to the graphs

I wonder if you can have an per application io activity that is also a factor for server load

http://rt.wiki.kernel.org/index.php/I/Otop_utility

other tool is dstat

If it were I/O, then he would see the iowait (pink) on the cpu graphs.