Apache gets "clogged" with certain requests
I have a very odd intermittent issue. It occurs on a shared hosting server running CentOS and Apache/2.2.8, serving mostly PHP based websites at a rate of between 10 - 30 requests/sec. 3.5 GB RAM, kernel 2.6.9-67.0.7ELsmp in case that matters.
What happens is, about once a week (but seemingly randomly with no correlation) apache will get "clogged" handling requests for only ONE domain, and normally most of the retests are all for the same URL. I know the problem occurs because load goes through the roof. (Load reaches 50 or 100, if I don't stop it load hits 200!) Only fix I've found is to: killall -9 httpd; /etc/init.d/httpd restart.
If I check /server-status when this is happening, the output is similar to the following. Note it doesn't always happen on this site but about 75% of the time this site's the culprit: EDIT: This is now happening on a completely different server, so it's not specific to a given script.
UPDATE: We have moved all the sites on this server to a brand new virtual machine on brand new hardware. Faster disks, more RAM. The issue persists, in fact it's even worse! Any ideas? I'd gladly offer a gigantic bounty to the correct answer, but I am reluctant to actually offer one as this question has gone years with no correct answer in sight.
Apache Server Status for localhost
Server Version: Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.7a DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.6 mod_perl/2.0.4 Perl/v5.8.8
Server Built: Jun 7 2008 09:35:37
Current Time: Friday, 17-Jul-2009 10:15:40 EDT
Restart Time: Friday, 03-Jul-2009 12:36:59 EDT
Parent Server Generation: 3
Server uptime: 13 days 21 hours 38 minutes 40 seconds
Total accesses: 10343186 - Total Traffic: 111.6 GB
CPU Usage: u1710.63 s2129.19 cu.41 cs0 - .32% CPU load
8.61 requests/sec - 97.4 kB/second - 11.3 kB/request
250 requests currently being processed, 0 idle workers
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWW......
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-3 3282 0/52/316180 W 1.47 41802 0 0.0 0.54 3849.21 209.249.53.104 zachdotsey.com GET /wordpress/2007/08/18/the-body-exhibition/ HTTP/1.0
1-3 30061 0/191/313974 W 34.46 44051 0 0.0 2.62 3400.19 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
2-3 2720 1/23/307812 W 0.13 42298 0 0.5 0.01 3479.29 68.118.8.157 zachdotsey.com GET /wordpress/ HTTP/1.1
3-3 1733 0/382/306392 W 3.59 41744 0 0.0 3.67 3383.63 75.101.200.204 zachdotsey.com GET /wordpress/2007/08/18/the-body-exhibition/ HTTP/1.0
4-3 3141 0/132/305071 W 3.52 41776 0 0.0 2.20 3335.79 209.249.53.236 zachdotsey.com GET /wordpress/2007/08/18/the-body-exhibition/ HTTP/1.0
5-3 31819 0/73/300070 W 0.79 43932 0 0.0 2.05 3349.65 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
6-3 6799 0/13/294060 W 0.16 40354 0 0.0 0.25 3261.88 76.23.230.184 zachdotsey.com GET /wordpress/?p=111 HTTP/1.1
7-3 4334 0/92/289873 W 0.42 40277 0 0.0 0.36 3211.94 72.233.56.144 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
8-3 18326 0/11/294687 W 0.14 32476 0 0.0 0.06 3217.41 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
9-3 2451 0/56/291730 W 0.23 42381 0 0.0 0.32 3113.19 68.118.8.157 zachdotsey.com GET /wordpress/index.php HTTP/1.1
10-3 31211 0/59/280899 W 6.04 44051 0 0.0 0.77 3128.00 72.52.238.128 zachdotsey.com POST /wordpress/wp-cron.php?doing_wp_cron HTTP/1.0
11-3 3823 0/4/277229 W 0.17 41746 0 0.0 0.04 3014.66 68.118.8.157 zachdotsey.com GET /wordpress/index.php HTTP/1.1
12-3 4473 0/9/277074 W 0.08 38453 0 0.0 0.00 3313.94 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
13-3 4242 1/14/272166 W 0.10 41659 0 0.5 0.05 2977.05 68.118.8.157 zachdotsey.com GET /wordpress/ HTTP/1.1
14-3 2323 0/234/269513 W 0.84 42136 0 0.0 1.02 2946.48 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
15-3 32051 0/6/257740 W 0.08 43872 0 0.0 0.58 2866.34 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
16-3 16986 0/271/259463 W 18.13 16761 0 0.0 4.12 2896.05 74.6.18.233 zachdotsey.com GET /wordpress/tag/xbox/ HTTP/1.0
17-3 2297 1/8/252501 W 0.54 42502 0 0.5 0.18 2797.73 68.118.8.157 zachdotsey.com GET /wordpress/ HTTP/1.1
18-3 9476 1/145/247116 W 0.61 40095 0 0.4 1.53 2703.11 64.237.119.18 zachdotsey.com GET /wordpress/ HTTP/1.1
19-3 2494 0/214/244830 W 3.16 42017 0 0.0 2.40 2618.41 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
20-3 5526 0/118/238426 W 3.50 37656 0 0.0 1.73 2544.06 68.10.212.178 zachdotsey.com GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
21-3 31901 0/28/228757 W 0.14 43992 0 0.0 0.04 2522.36 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
22-3 12398 0/25/216141 W 0.40 39998 0 0.0 4.72 2398.82 38.117.108.131 zachdotsey.com GET /wordpress/ HTTP/1.1
23-3 2300 0/187/222141 W 0.68 42315 0 0.0 0.67 2503.15 65.184.161.115 zachdotsey.com GET /wordpress/ HTTP/1.1
24-3 31969 0/25/211848 W 0.26 43810 0 0.0 0.38 2326.01 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
25-3 31582 0/25/198495 W 0.17 69010 0 0.0 0.25 2139.13 70.63.195.121 myvenue.venuecom.com POST /fastmail/ HTTP/1.1
26-3 28115 1/1/197089 W 0.04 40155 0 0.4 0.00 2126.75 64.237.119.18 zachdotsey.com GET /wordpress/ HTTP/1.1
27-3 12988 1/42/186898 W 0.55 35055 0 2.1 0.36 2037.54 71.193.156.250 zachdotsey.com GET /wordpress/ HTTP/1.1
28-3 3780 1/26/181659 W 0.34 41662 0 0.5 0.05 1918.00 68.118.8.157 zachdotsey.com GET /wordpress/ HTTP/1.1
29-3 1002 0/22/169331 W 0.18 39885 0 0.0 0.03 1888.59 64.40.121.190 zachdotsey.com GET /robots.txt HTTP/1.0
30-3 4493 0/6/163101 W 0.06 41465 0 0.0 0.18 1732.58 209.131.41.49 zachdotsey.com GET /wordpress/feed/ HTTP/1.0
31-3 32458 0/1/162852 W 0.04 43750 0 0.0 0.01 1758.80 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
32-3 14198 0/571/157504 W 47.36 32680 0 0.0 8.64 1733.04 74.6.18.233 zachdotsey.com GET /wordpress/2008/12/ HTTP/1.0
33-3 2213 0/9/152830 W 0.32 42624 0 0.0 0.00 1782.36 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
34-3 2308 0/1/137395 W 0.06 39398 0 0.0 0.00 1433.25 64.40.121.190 zachdotsey.com GET /wordpress/ HTTP/1.0
35-3 6984 0/123/127498 W 1.88 40241 0 0.0 0.18 1350.38 206.196.125.113 zachdotsey.com GET /wordpress/?feed=rss2 HTTP/1.1
36-3 1003 0/21/116316 W 0.81 39874 0 0.0 1.15 1216.35 67.202.60.246 zachdotsey.com GET /wordpress/ HTTP/1.1
37-3 3025 0/19/106816 W 0.16 42196 0 0.0 0.28 1096.76 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
38-3 4250 1/15/101047 W 0.09 41660 0 0.5 0.01 1198.01 68.118.8.157 zachdotsey.com GET /wordpress/ HTTP/1.1
39-3 3073 0/73/91871 W 2.32 41891 0 0.0 1.81 924.03 68.118.8.157 zachdotsey.com GET /wordpress/index.php HTTP/1.1
40-3 2133 0/15/86368 W 0.20 39445 0 0.0 0.09 888.41 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
41-3 2803 0/93/79858 W 0.67 39020 0 0.0 0.69 877.38 64.40.121.190 zachdotsey.com GET /wordpress/ HTTP/1.0
42-3 3030 0/165/72532 W 35.08 41532 0 0.0 2.17 777.91 65.184.161.115 zachdotsey.com GET /wordpress/?feed=rss2 HTTP/1.1
43-3 13566 0/427/68292 W 14.81 33263 0 0.0 6.28 682.48 174.129.193.94 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
44-3 31951 0/6/65523 W 0.39 43980 0 0.0 0.15 624.37 78.31.110.126 zachdotsey.com GET /wordpress/feed HTTP/1.1
45-3 17775 0/4/64507 W 0.11 32768 0 0.0 0.00 666.08 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
46-3 17485 0/280/61533 W 30.17 16250 0 0.0 6.34 655.87 77.88.42.26 zachdotsey.com GET /robots.txt HTTP/1.1
47-3 1379 0/305/51058 W 2.50 42256 0 0.0 5.02 691.25 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
48-3 15264 0/28/49621 W 0.11 33958 0 0.0 0.02 527.41 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
49-3 20544 1/28/48765 W 1.55 40125 0 0.4 0.17 516.42 64.237.119.18 zachdotsey.com GET /wordpress/ HTTP/1.1
50-3 14318 0/408/50772 W 26.77 32722 0 0.0 7.28 509.18 74.6.18.233 zachdotsey.com GET /wordpress/2008/12/ HTTP/1.0
51-3 2258 0/18/45699 W 0.08 39388 0 0.0 0.01 456.81 64.40.121.190 zachdotsey.com GET /robots.txt HTTP/1.0
52-3 3039 0/21/37675 W 0.38 42077 0 0.0 1.58 442.82 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
53-3 2747 0/3/35053 W 0.06 39268 0 0.0 0.22 371.79 64.40.121.190 zachdotsey.com GET /wordpress/ HTTP/1.0
54-3 17408 0/22/35784 W 0.26 32871 0 0.0 0.42 367.04 74.6.18.233 zachdotsey.com GET /robots.txt HTTP/1.0
55-3 4511 0/6/29200 W 0.40 41444 0 0.0 0.23 375.89 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
56-3 5534 0/54/27466 W 2.63 37695 0 0.0 0.38 305.27 68.10.212.178 zachdotsey.com GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
57-3 23760 0/41/20806 W 0.23 40553 0 0.0 0.33 219.16 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
58-3 5638 0/247/17586 W 2.79 40174 0 0.0 2.66 182.34 64.237.119.18 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
59-3 31783 0/5/18112 W 0.05 40067 0 0.0 0.06 182.11 65.113.57.7 zachdotsey.com GET /wordpress/feed/ HTTP/1.0
60-3 2319 0/1/13920 W 0.03 39391 0 0.0 0.00 145.61 38.117.108.131 zachdotsey.com GET /wordpress/ HTTP/1.1
61-3 23730 0/155/12935 W 29.86 28181 0 0.0 3.25 153.35 72.30.161.222 zachdotsey.com GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
62-3 12586 0/25/12506 W 0.14 40271 0 0.0 0.09 135.23 72.233.56.156 zachdotsey.com GET /wordpress/ HTTP/1.1
63-3 13567 0/15/10439 W 0.54 34975 0 0.0 0.10 137.24 84.217.7.221 zachdotsey.com GET /wordpress/2009/06/06/emergency-baby-call-tests/ HTTP/1.1
64-3 17776 1/161/10534 W 1.65 31758 0 0.4 2.14 128.33 208.92.218.250 zachdotsey.com GET /wordpress/ HTTP/1.1
65-3 3053 0/159/7939 W 7.25 41485 0 0.0 2.64 74.76 209.131.41.48 zachdotsey.com GET /robots.txt HTTP/1.0
66-3 5056 0/115/6934 W 1.56 37848 0 0.0 2.43 81.76 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
67-3 16535 0/160/6199 W 19.57 39641 0 0.0 2.57 67.22 64.40.121.190 zachdotsey.com GET /wordpress/ HTTP/1.0
68-3 17777 0/37/7722 W 0.24 32569 0 0.0 0.15 93.77 74.6.18.233 zachdotsey.com GET /wordpress/2008/12/ HTTP/1.0
69-3 2355 0/2/5737 W 0.15 39346 0 0.0 0.03 73.54 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
70-3 10432 0/24/5080 W 0.33 36392 0 0.0 0.47 45.89 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
71-3 10433 0/12/4099 W 0.93 36365 0 0.0 0.65 41.80 68.7.141.144 zachdotsey.com GET /wordpress/2006/12/18/the-plastic-coating-on-the-end-of-a-s
72-3 6145 0/114/4835 W 12.61 37285 0 0.0 1.49 57.55 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
73-3 5539 0/71/4717 W 5.37 37680 0 0.0 1.63 62.39 68.10.212.178 zachdotsey.com GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
74-3 17690 0/38/5004 W 0.12 32810 0 0.0 0.25 46.34 74.6.18.233 zachdotsey.com GET /wordpress/2008/12/ HTTP/1.0
75-3 1528 0/38/3991 W 0.14 39764 0 0.0 0.08 41.91 64.40.121.190 zachdotsey.com GET /wordpress/ HTTP/1.0
76-3 2751 0/26/2886 W 0.22 39144 0 0.0 0.63 25.56 64.40.121.190 zachdotsey.com GET /wordpress/ HTTP/1.0
77-3 5453 0/95/6939 W 1.95 21796 0 0.0 1.09 87.49 69.147.79.37 zachdotsey.com GET /robots.txt HTTP/1.0
78-3 5061 0/192/3238 W 9.87 37110 0 0.0 2.13 39.70 78.31.110.126 zachdotsey.com GET /wordpress/feed HTTP/1.1
79-3 9955 0/143/4659 W 11.55 35985 0 0.0 7.35 64.53 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
80-3 17779 0/29/4181 W 0.33 32548 0 0.0 0.47 48.11 74.6.18.233 zachdotsey.com GET /wordpress/2008/12/ HTTP/1.0
81-3 1192 0/74/2861 W 0.54 39651 0 0.0 0.42 27.09 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
82-3 13108 0/42/4116 W 0.13 35144 0 0.0 0.10 37.39 71.193.156.250 zachdotsey.com GET /wordpress/wp-login.php?loggedout=true HTTP/1.1
83-3 32180 0/0/6535 W 0.98 10753 0 0.0 0.00 105.04 142.166.3.122 zachdotsey.com GET /robots.txt HTTP/1.1
84-3 24479 0/148/4080 W 98.02 28139 0 0.0 3.46 60.62 72.30.161.222 zachdotsey.com GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
85-3 776 1/204/8349 W 40.83 9214 0 0.4 3.57 99.11 64.237.119.18 zachdotsey.com GET /wordpress/ HTTP/1.1
86-3 13569 0/25/3218 W 10.53 34863 0 0.0 0.29 28.96 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
87-3 13110 0/2/2535 W 0.09 35187 0 0.0 0.00 26.16 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
88-3 4716 0/157/2237 W 3.75 37444 0 0.0 3.58 25.28 206.196.125.113 zachdotsey.com GET /wordpress/?feed=rss2 HTTP/1.1
89-3 19878 0/110/2928 W 10.39 31357 0 0.0 0.54 26.66 58.61.164.39 zachdotsey.com GET /robots.txt HTTP/1.1
90-3 2003 0/28/1847 W 1.81 39519 0 0.0 0.39 19.10 64.40.121.190 zachdotsey.com GET /wordpress/ HTTP/1.0
91-3 13570 0/0/1928 W 10.59 35116 0 0.0 0.00 23.59 71.193.156.250 zachdotsey.com GET /wordpress/ HTTP/1.1
92-3 16358 0/282/1910 W 33.11 32280 0 0.0 5.57 26.53 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
93-3 16367 0/50/1664 W 10.03 33243 0 0.0 0.88 17.62 174.129.193.94 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
94-3 25642 0/82/2391 W 19.24 28268 0 0.0 2.07 34.95 72.30.161.222 zachdotsey.com GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
95-3 24480 0/261/2711 W 31.22 27528 0 0.0 3.14 43.29 75.219.249.154 zachdotsey.com GET /wordpress/tag/missy-warkentien/ HTTP/1.1
96-3 17692 0/514/1024 W 46.55 30965 0 0.0 6.50 14.37 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
97-3 10442 0/126/1041 W 1.25 35891 0 0.0 1.20 8.47 203.171.91.144 zachdotsey.com GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
98-3 25643 0/103/1705 W 0.67 28008 0 0.0 2.83 20.89 74.6.18.233 zachdotsey.com GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
99-3 22619 0/338/1657 W 27.38 28329 0 0.0 4.00 27.55 72.30.161.222 zachdotsey.com GET /robots.txt HTTP/1.0
100-3 19714 0/0/1845 W 12.66 31778 0 0.0 0.00 22.14 208.92.218.250 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
101-3 19367 1/155/1983 W 10.76 31728 0 0.4 0.15 23.66 208.92.218.250 zachdotsey.com GET /wordpress/ HTTP/1.1
102-3 26652 0/122/1187 W 41.42 26872 0 0.0 3.00 14.95 75.219.249.154 zachdotsey.com GET /wordpress/tag/missy-warkentien/ HTTP/1.1
103-3 7280 3/361/1639 W 12.44 35165 0 8.9 4.80 17.06 71.193.156.250 zachdotsey.com GET /wordpress/ HTTP/1.1
104-3 20843 0/507/949 W 99.96 28160 0 0.0 7.70 11.64 72.30.161.222 zachdotsey.com GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
105-3 20844 0/308/1868 W 23.03 30024 0 0.0 3.00 28.11 78.31.110.126 zachdotsey.com GET /wordpress/feed HTTP/1.1
106-3 17791 1/278/1026 W 26.89 31698 0 0.4 4.18 13.29 208.92.218.250 zachdotsey.com GET /wordpress/ HTTP/1.1
107-3 17792 0/304/642 W 12.27 31388 0 0.0 2.72 8.79 58.61.164.39 zachdotsey.com GET /robots.txt HTTP/1.1
108-3 17697 0/57/515 W 0.30 32611 0 0.0 1.78 8.41 74.6.18.233 zachdotsey.com GET /wordpress/2008/12/ HTTP/1.0
109-3 19715 0/611/783 W 60.52 29153 0 0.0 5.94 7.20 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
110-3 13952 0/345/614 W 5.95 33656 0 0.0 3.47 8.18 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
111-3 32009 0/75/1959 W 5.25 24778 0 0.0 1.06 22.60 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
112-3 17793 0/35/178 W 11.11 32590 0 0.0 0.44 1.92 74.6.18.233 zachdotsey.com GET /wordpress/2008/12/ HTTP/1.0
113-3 19533 0/8/2004 W 0.16 16705 0 0.0 1.32 34.66 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
114-3 22622 0/304/925 W 181.63 28201 0 0.0 5.05 12.30 72.30.161.222 zachdotsey.com GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
115-3 26656 0/38/763 W 0.55 27724 0 0.0 0.40 6.22 75.219.249.154 zachdotsey.com GET /wordpress/tag/missy-warkentien/ HTTP/1.1
116-3 31804 0/57/3039 W 2.17 10748 0 0.0 1.27 44.70 142.166.3.122 zachdotsey.com GET /wordpress/2009/07/13/millys-first-time-in-a-restaurant/ HT
117-3 26658 0/0/866 W 14.05 28069 0 0.0 0.00 11.50 74.6.18.233 zachdotsey.com GET /robots.txt HTTP/1.0
118-3 26659 0/46/646 W 4.59 27608 0 0.0 0.47 4.90 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
119-3 5457 0/86/998 W 21.35 21916 0 0.0 1.59 14.79 69.147.79.37 zachdotsey.com GET /robots.txt HTTP/1.0
120-3 9855 0/486/1470 W 58.03 18535 0 0.0 8.31 25.07 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
121-3 17802 0/8/206 W 0.05 32743 0 0.0 0.00 1.67 74.6.18.233 zachdotsey.com GET /wordpress/2008/12/ HTTP/1.0
122-3 17803 0/10/141 W 0.19 32701 0 0.0 0.00 1.08 74.6.18.233 zachdotsey.com GET /wordpress/2008/12/ HTTP/1.0
123-3 19888 0/69/120 W 0.37 31389 0 0.0 0.97 1.10 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
124-3 25184 0/160/689 W 33.97 27701 0 0.0 5.24 9.16 75.219.249.154 zachdotsey.com GET /wordpress/tag/missy-warkentien/ HTTP/1.1
125-3 27208 0/615/2460 W 28.73 10736 0 0.0 7.88 44.54 142.166.3.122 zachdotsey.com GET /wordpress/2009/07/14/millys-smile/ HTTP/1.1
126-3 21945 0/281/410 W 4.56 29359 0 0.0 3.82 7.16 74.6.18.233 zachdotsey.com GET /robots.txt HTTP/1.0
127-3 25187 0/392/429 W 58.81 25928 0 0.0 7.26 7.47 75.219.249.154 zachdotsey.com GET /wordpress/ HTTP/1.1
128-3 21948 0/223/299 W 2.32 30044 0 0.0 1.29 1.77 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
129-3 27517 0/211/551 W 60.71 25918 0 0.0 4.73 10.25 75.219.249.154 zachdotsey.com GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
130-3 9078 1/150/753 W 13.68 21327 0 0.4 2.78 11.88 208.92.218.250 zachdotsey.com GET /wordpress/ HTTP/1.1
131-3 20742 0/97/2004 W 1.96 15475 0 0.0 2.14 35.30 78.31.110.126 zachdotsey.com GET /wordpress/feed HTTP/1.1
132-3 12265 0/658/1664 W 62.77 16822 0 0.0 11.80 27.61 74.6.18.233 zachdotsey.com GET /robots.txt HTTP/1.0
133-3 31859 0/72/2354 W 12.54 10604 0 0.0 1.14 54.86 174.129.193.99 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
134-3 4105 0/60/658 W 12.23 22622 0 0.0 1.46 9.19 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
135-3 4106 0/195/679 W 15.18 21676 0 0.0 3.77 14.09 69.147.79.37 zachdotsey.com GET /robots.txt HTTP/1.0
136-3 25192 0/8/8 W 0.21 28835 0 0.0 0.01 0.01 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
137-3 25193 0/103/103 W 15.55 27940 0 0.0 1.20 1.20 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
138-3 25194 0/37/37 W 11.09 28465 0 0.0 1.07 1.07 38.117.108.35 zachdotsey.com GET /wordpress/feed HTTP/1.1
139-3 28509 0/64/106 W 1.12 26708 0 0.0 0.82 1.54 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
140-3 28955 0/68/68 W 2.49 26135 0 0.0 1.37 1.37 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
141-3 28957 0/59/59 W 8.77 26104 0 0.0 1.60 1.60 75.219.249.154 zachdotsey.com GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
142-3 28958 0/126/126 W 4.49 26058 0 0.0 1.75 1.75 75.219.249.154 zachdotsey.com GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
143-3 3101 0/26/1994 W 1.88 9152 0 0.0 0.16 32.99 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
144-3 21433 0/29/1004 W 1.06 15819 0 0.0 0.21 16.72 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
145-3 12273 0/336/786 W 29.77 18507 0 0.0 7.08 15.99 58.61.164.141 zachdotsey.com GET /wordpress/2008/11/01/early-voting/ HTTP/1.1
146-3 4109 0/65/232 W 1.64 22700 0 0.0 0.52 2.96 78.31.110.126 zachdotsey.com GET /wordpress/feed HTTP/1.1
147-3 9081 0/250/582 W 61.84 20036 0 0.0 4.35 10.27 125.244.77.2 zachdotsey.com POST /_vti_bin/_vti_aut/author.dll HTTP/1.1
148-3 21639 0/216/1433 W 8.19 14597 0 0.0 1.97 25.39 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
149-3 31137 0/1/1657 W 0.08 11318 0 0.0 0.00 29.24 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
150-3 31210 0/227/227 W 86.27 23567 0 0.0 8.50 8.50 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
151-3 4071 0/42/2081 W 12.87 8452 0 0.0 0.74 32.38 76.13.20.61 zachdotsey.com GET /robots.txt HTTP/1.0
152-3 9904 0/31/445 W 1.53 21377 0 0.0 0.71 10.45 208.92.218.250 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
153-3 25865 0/8/1236 W 0.20 13614 0 0.0 0.13 14.50 76.13.20.61 zachdotsey.com GET /robots.txt HTTP/1.0
154-3 9905 0/78/294 W 15.40 20614 0 0.0 1.30 4.53 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
155-3 9085 0/75/78 W 11.31 21562 0 0.0 0.88 0.88 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
156-3 18546 0/51/192 W 3.91 17077 0 0.0 0.22 4.83 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
157-3 32217 0/0/1740 W 0.24 10744 0 0.0 0.00 28.44 142.166.3.122 zachdotsey.com GET /wordpress/2009/07/11/millys-anticipation/ HTTP/1.1
158-3 14856 0/264/641 W 3.65 17963 0 0.0 2.75 9.95 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
159-3 18153 0/92/3545 W 4.82 2279 0 0.0 1.50 45.82 74.6.18.233 zachdotsey.com GET /wordpress/2008/10/24/ HTTP/1.0
160-3 31679 0/56/56 W 0.54 25195 0 0.0 0.61 0.61 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
161-3 32316 0/1/1628 W 0.05 10733 0 0.0 0.00 27.14 142.166.3.122 zachdotsey.com GET /wordpress/2009/07/12/millys-first-poop-on-daddy/ HTTP/1.1
162-3 25433 0/89/590 W 1.86 13495 0 0.0 1.50 9.58 76.13.20.61 zachdotsey.com GET /robots.txt HTTP/1.0
163-3 19960 0/278/338 W 26.51 14965 0 0.0 3.34 3.96 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
164-3 20271 0/111/598 W 3.85 15553 0 0.0 3.49 10.91 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
165-3 13282 0/113/3121 W 3.01 4585 0 0.0 0.52 37.39 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
166-3 25641 0/227/658 W 27.64 12512 0 0.0 2.51 8.38 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
167-3 26850 0/303/816 W 6.27 12241 0 0.0 2.71 12.95 77.88.42.26 zachdotsey.com GET /robots.txt HTTP/1.1
168-3 25966 0/413/898 W 16.14 12226 0 0.0 3.66 10.86 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
169-3 25967 0/35/276 W 0.84 13409 0 0.0 1.19 3.09 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
170-3 28752 0/426/4119 W 31.27 134 0 0.0 4.26 65.53 71.127.124.128 northwoodcinemagrill.com GET / HTTP/1.1
171-3 10540 0/16/1765 W 0.22 5732 0 0.0 0.23 20.56 92.51.157.168 zachdotsey.com GET /robots.txt HTTP/1.1
172-3 4297 0/26/1208 W 0.43 8572 0 0.0 0.02 22.46 76.13.20.61 zachdotsey.com GET /robots.txt HTTP/1.0
173-3 10221 0/140/140 W 63.93 19817 0 0.0 4.69 4.69 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
174-3 26697 0/529/845 W 24.04 11002 0 0.0 6.17 13.25 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
175-3 32336 0/224/1243 W 15.09 9411 0 0.0 2.93 20.51 207.179.181.130 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
176-3 25871 0/1/189 W 0.08 13705 0 0.0 0.01 1.38 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
177-3 25648 0/57/166 W 3.24 13460 0 0.0 0.38 3.02 58.61.164.39 zachdotsey.com GET /wordpress/2008/10/25/big-day-for-duke-sports/ HTTP/1.1
178-3 31175 0/93/975 W 26.19 10624 0 0.0 3.61 13.03 174.129.193.99 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
179-3 10228 1/34/34 W 0.26 21297 0 0.4 1.60 1.60 208.92.218.250 zachdotsey.com GET /wordpress/ HTTP/1.1
180-3 10229 0/235/235 W 11.14 19431 0 0.0 6.96 6.96 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
181-3 14002 0/180/2169 W 10.99 4030 0 0.0 1.11 18.94 65.184.161.115 zachdotsey.com GET /wordpress/ HTTP/1.1
182-3 31180 0/307/460 W 38.99 9753 0 0.0 6.36 7.88 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
183-3 10232 1/28/28 W 1.84 21357 0 0.4 0.26 0.26 208.92.218.250 zachdotsey.com GET /wordpress/ HTTP/1.1
184-3 31929 0/46/714 W 0.38 10749 0 0.0 0.27 10.75 142.166.3.122 zachdotsey.com GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
185-3 8930 0/201/1413 W 11.72 6142 0 0.0 0.90 18.43 92.51.157.168 zachdotsey.com GET /robots.txt HTTP/1.1
186-3 31931 0/314/471 W 47.33 9256 0 0.0 9.23 13.30 76.13.20.61 zachdotsey.com GET /robots.txt HTTP/1.0
187-3 1832 0/78/850 W 21.91 9234 0 0.0 1.14 8.29 64.237.119.18 zachdotsey.com GET /wordpress/feed/ HTTP/1.1
188-3 31181 0/121/556 W 5.97 10756 0 0.0 2.10 13.38 142.166.3.122 zachdotsey.com GET /robots.txt HTTP/1.1
189-3 9444 0/251/1730 W 5.98 5671 0 0.0 2.12 28.18 65.184.161.115 zachdotsey.com POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
190-3 31932 0/25/169 W 2.85 10769 0 0.0 0.73 2.87 142.166.3.122 zachdotsey.com GET /robots.txt HTTP/1.1
191-3 25658 0/6/6 W 0.10 13734 0 0.0 0.04 0.04 76.13.20.61 zachdotsey.com GET /robots.txt HTTP/1.0
192-3 9884 0/136/1248 W 1.37 5630 0 0.0 1.32 14.36 65.184.161.115 zachdotsey.com GET /wordpress/ HTTP/1.1
193-3 324 0/37/116 W 5.95 10106 0 0.0 1.27 2.86 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
194-3 19087 0/756/2584 W 67.99 1279 0 0.0 9.26 35.86 78.31.110.126 zachdotsey.com GET /wordpress/feed HTTP/1.1
195-3 20058 0/722/2684 W 62.21 1123 0 0.0 7.96 30.71 206.196.125.113 zachdotsey.com GET /wordpress/?feed=rss2 HTTP/1.1
196-3 2425 1/38/56 W 2.18 9154 0 0.4 0.47 0.59 64.237.119.18 zachdotsey.com GET /wordpress/ HTTP/1.1
197-3 28767 0/341/3091 W 12.60 64 0 0.0 2.89 33.12 65.184.161.115 myvenue.venuecom.com GET / HTTP/1.1
198-3 4816 0/260/942 W 26.04 7324 0 0.0 2.25 10.97 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
199-3 21139 0/567/2438 W 121.57 18 0 0.0 10.32 43.01 208.59.151.137 webdav.venuecom.com GET /calendar/dnewcorn/calendars/Web%20Demos.ics HTTP/1.1
200-3 4311 0/55/188 W 4.84 8571 0 0.0 1.31 3.77 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
201-3 5891 0/15/205 W 0.11 7957 0 0.0 1.61 6.07 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
202-3 30984 0/0/2581 W 42.72 40 0 0.0 0.00 31.17 207.210.148.62 marketplace-mail.com GET /checklist.php HTTP/1.1
203-3 2435 1/40/42 W 19.37 9184 0 0.4 2.43 2.44 64.237.119.18 zachdotsey.com GET /wordpress/ HTTP/1.1
204-3 29989 0/26/2777 W 30.30 78 0 0.0 1.51 38.95 198.178.147.1 raleighwoodmovies.com GET / HTTP/1.1
205-3 19694 0/0/1370 W 0.14 2633 0 0.0 0.00 19.70 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
206-3 19406 0/202/1756 W 10.98 2238 0 0.0 1.83 20.94 74.6.18.233 zachdotsey.com GET /wordpress/2008/10/24/ HTTP/1.0
207-3 19695 0/937/2326 W 42.78 482 0 0.0 8.58 23.44 76.182.6.152 raleighwoodmovies.com GET / HTTP/1.1
208-3 11763 0/129/802 W 14.71 4766 0 0.0 2.16 9.27 65.184.161.115 zachdotsey.com GET /wordpress/?feed=rss2 HTTP/1.1
209-3 2442 0/26/27 W 9.27 9236 0 0.0 0.54 0.54 77.88.42.26 zachdotsey.com GET /robots.txt HTTP/1.1
210-3 2447 0/94/95 W 7.65 9136 0 0.0 0.73 0.73 76.13.20.61 zachdotsey.com GET /robots.txt HTTP/1.0
211-3 3914 0/80/100 W 6.41 8330 0 0.0 0.48 0.61 78.31.110.126 zachdotsey.com GET /wordpress/feed HTTP/1.1
212-3 11674 0/0/519 W 14.19 5522 0 0.0 0.00 7.40 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
213-3 7627 0/70/211 W 2.47 6989 0 0.0 1.77 3.11 77.88.42.26 zachdotsey.com GET /robots.txt HTTP/1.1
214-3 29724 0/557/557 W 80.97 9376 0 0.0 7.93 7.93 76.13.20.61 zachdotsey.com GET /robots.txt HTTP/1.0
215-3 19410 0/980/2490 W 162.44 138 0 0.0 7.87 26.41 198.178.147.1 raleighwoodmovies.com GET /printable.php HTTP/1.1
216-3 9451 0/9/405 W 0.16 6415 0 0.0 0.06 2.35 66.249.66.2 zachdotsey.com GET /robots.txt HTTP/1.1
217-3 11764 0/9/695 W 0.13 5407 0 0.0 0.08 9.91 98.122.166.13 zachdotsey.com GET /wordpress/ HTTP/1.1
218-3 20889 0/46/1701 W 0.18 2217 0 0.0 0.37 26.16 74.6.18.233 zachdotsey.com GET /wordpress/2008/10/24/ HTTP/1.0
219-3 17642 0/511/1352 W 8.05 2259 0 0.0 2.89 13.17 74.6.18.233 zachdotsey.com GET /wordpress/2008/10/24/ HTTP/1.0
220-3 19411 0/757/1910 W 29.09 1126 0 0.0 5.47 18.72 65.184.161.115 zachdotsey.com GET /wordpress/ HTTP/1.1
221-3 19412 0/284/1153 W 5.34 2085 0 0.0 3.74 22.78 74.6.18.233 zachdotsey.com GET /wordpress/2008/10/24/ HTTP/1.0
222-3 19696 0/885/1794 W 54.08 237 0 0.0 15.96 25.04 69.86.60.67 webdav.venuecom.com GET /calendar/lswuser/private/calendars/Prospecting.ics HTTP/1.
223-3 19697 0/57/1272 W 0.58 2409 0 0.0 0.38 11.08 74.6.18.233 zachdotsey.com GET /robots.txt HTTP/1.0
224-3 20500 0/226/1395 W 0.82 2147 0 0.0 2.12 11.41 74.6.18.233 zachdotsey.com GET /wordpress/2008/10/24/ HTTP/1.0
225-3 21060 0/776/1869 W 36.25 475 0 0.0 10.92 24.86 78.31.110.127 allnews.allchoiceinsurance.com GET /feed/ HTTP/1.1
226-3 15942 8/360/708 W 10.60 2428 0 29.1 3.08 6.61 65.184.161.115 zachdotsey.com GET /wordpress/ HTTP/1.1
227-3 18059 1/9/402 W 2.48 3120 0 419.1 0.83 4.29 24.199.245.190 zachdotsey.com GET /wordpress/ HTTP/1.1
228-3 3929 0/102/102 W 16.70 8332 0 0.0 1.24 1.24 76.13.20.61 zachdotsey.com GET /robots.txt HTTP/1.0
229-3 7814 0/343/396 W 2.93 6380 0 0.0 1.71 1.86 98.122.166.13 zachdotsey.com GET /wordpress/ HTTP/1.1
230-3 20958 0/746/2021 W 86.23 121 0 0.0 11.92 25.76 72.30.87.105 baucomsbest.com GET /index.php HTTP/1.0
231-3 20897 1/765/1673 K 43.84 4 15 10.6 7.17 17.99 24.163.58.115 myvenue.venuecom.com GET /editor/images/progress-bar-back.gif HTTP/1.1
...
After you saying what modules you had, I noticed you said EAccelerator, which has a bug (Bug tracker link here) that causes Apache to lockup in never ending loop, and bring the server to a near standstill, try disabling that and see what happens.
Do you see many requests for POST /wordpress/wp-admin/admin-ajax.php at the problem times? If so then it could be caused by either a passing spammer trying to brute force something, by a deliberate DoS attempt, or just because the site is getting posted somewhere busy every now and again (do your logs show many requests coming from the same off-site referrer?).
It could also be a database locking issue, i.e. one request has asked a long operation to be performed with full deny-read-until-done locks on a frequently accessed table and the other requests are just stuck waiting for the lock to be released (which may happen naturally, or as a result of you killing Apache and therefore the connection that is holding the lock(s). If the site is busy enough then this can create significant load as lots of Apache processes get created which then sit consuming memory until they can continue past the lock - if your Apache config is such that a relatively unconstrained number of child processes can be created then the system could easily start thrashing swap at this point.
To see if the load is being caused by CPU use, IO due to DB access and such, or IO due to swap thrashing, you'll need to investigate the output of htop (or just plain top), free, and the other usual system load monitoring utils. This may give more clues as to where the root of the issue might be found.