Why does apache fail to start and only say "Configuration Failed" in error log?

Apache 2.2.23 fails to start for me, and the only entries in the log file are:

[Fri Feb 22 02:54:24 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Feb 22 02:54:24 2013] [notice] Digest: generating secret for digest authentication ...
[Fri Feb 22 02:54:24 2013] [notice] Digest: done
Configuration Failed
[Fri Feb 22 02:54:57 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Feb 22 02:54:57 2013] [notice] Digest: generating secret for digest authentication ...
[Fri Feb 22 02:54:57 2013] [notice] Digest: done
Configuration Failed

This has been driving me crazy. The network is fine, the disks are fine, the config files pass syntax checks, nothing has been changed since it previously worked. It just refuses to start, usually after some load spike.

I'd also mention that the restart mechanism includes doing kill -9 if graceful shutdown doesn't succeed.

I can't find anything on google or here about this.

Solution 1:

I finally found the solution to this, by removing all site configs, modules, and config directives.

Then, finally, the error log produced this gem:

[Fri Feb 22 03:04:22 2013] [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.10752) (5)
[Fri Feb 22 03:04:24 2013] [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?

Now this, there were answers for on google. As mentioned here and other places, this error was caused by too many open semaphores from apache, as the disk was empty and writeable.

I solved this by running:

ipcs -s | grep www-data | awk '{print $2;}' | while read -r line; do ipcrm sem "$line"; done

and adding that to the restart script. Now, everything is fine. I assume there's a correlation between this and using kill -9, but without that I'm left with hung, unresponsive apache processes.

Solution 2:

I would suggest you to run an strace to see what's going on:

strace -f -o apache.trace /usr/sbin/httpd

Maybe the disk is full?

Solution 3:

Check the permissions of the location your logs are being written to, including the files themselves. Apache needs to be able to write to the log when it drops permissions from root, and you'll get little of use from the logs if Apache is bailing out because it can no longer write to them. It's one of the most simple, yet most infuriatingly elusive problems you can run into.

If this is not the case, @quanta's suggestion is probably the best way to identify what was happening when Apache bailed out.