Is Apache stalled? /server-status shows over 240 requests like "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"

Some details:

• Webserver: Apache/2.2.13 (FreeBSD) mod_ssl/2.2.13 OpenSSL/0.9.8e
• OS: FreeBSD 7.2-RELEASE
• This is a FreeBSD Jail.
• I believe I use the Apache 'prefork' MPM (I run the default for FreeBSD).
• I use the default values for MaxClients (256)

I have enabled mod_status, with "ExtendedStatus On". When I view /server-status , I see a handful of regular requests. I also see over 240 requests from the 'localhost', like these.

37-0    -   0/0/1   .   0.00    1510    0   0.0 0.00    0.00    127.0.0.2   www.example.gov OPTIONS * HTTP/1.0
38-0    -   0/0/1   .   0.00    1509    0   0.0 0.00    0.00    127.0.0.2   www.example.gov OPTIONS * HTTP/1.0
39-0    -   0/0/3   .   0.00    1482    0   0.0 0.00    0.00    127.0.0.2   www.example.gov OPTIONS * HTTP/1.0
40-0    -   0/0/6   .   0.00    1445    0   0.0 0.00    0.00    127.0.0.2   www.example.gov OPTIONS * HTTP/1.0

I also see about 2417 requests yesterday from the localhost, like these:

Apr 14 11:16:40 192.168.16.127 httpd[431]: www.example.gov 127.0.0.2 - - [15/Apr/2010:11:16:40 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"

The page at http://wiki.apache.org/httpd/InternalDummyConnection says "These requests are perfectly normal and you do not, in general, need to worry about them", but I'm not so sure.

Why are there over 230 of these? Are these active connections? If I have "MaxClients 256", and over 230 of these connections, it seems that my webserver is dangerously close to running out of available connections. It also seems like Apache should only need a handful of these "internal dummy connections"

We actually had two unexplained outages last night, and I am wondering if these "internal dummy connection" caused us to run out of available connections.

UPDATE 2010/04/16

It is 8 hours later. The /server-status page still shows that there are 243 lines which say "www.example.gov OPTION *". I believe these connections are not active. The server is mostly idle (1 requests currently being processed, 9 idle workers). There are only 18 active httpd processes on the Unix host.

If these connections are not active, why do they show up under /server-status? I would have expected them to expire a few minutes after they were initialized.

Apache handles a thundering herd a little differently than you might imagine. When you get a burst of inbound traffic, it spawns a number of child processes, if it determines it needs more, it spawns twice as many in the next interval until it finally has enough processes to serve the requests or hits maxclients.

If you see these, it means that apache is just checking the children, and whatever caused apache to fork that many processes is probably gone. Yes they do take up client connections, but, whatever event caused things to spool up is probably gone.

First thing I would check in your logs would be a bunch of 302s prior to the event.

If you had something like

<?php include("http://www.oursite.com/header.php");?>   

where header.php was missing and

ErrorDocument 404 /404.php 

where 404.php included header.php, you would get a recursive loop and a hit on that page would immediately cause apache to use all available connections.

My understanding here is that, given that these are connections from the parent to the child process, they’re merely Apache keeping track of what the children are doing. Bear in mind that:

• children can hang around for quite a while after they’ve processed a request
• the internal dummy connections occur regularly
• if the child hasn’t done anything else (because the server’s mostly idle), the dummy connection will be the most recent thing it’s processed

it’s not the case, as far as I know, that the dummy connections “use up” children. Apache is checking what the status of its children is, rather than exercising them to test whether they work or not.