Find out what high CPU usage apache process is actually doing?

Currently having a few issues with our server where, intermittently, we seem to get apache processes which just run and run, taking up 100% CPU.

When running top, we see the following:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20788 www-data  20   0  318m  18m 3984 R  100  0.0  40:29.21 /usr/sbin/apache2 -k start
23523 www-data  20   0  319m  20m 4684 R  100  0.0   4:12.36 /usr/sbin/apache2 -k start

I want to try and find out what script (or whatever it is) is causing this, so I tried:

 strace -p 20788

But that doesn't show any output at all (I've left it for about 10 minutes, and it shows nothing). From my understanding, this could mean it's stuck in an infinite loop, and there aren't any "system calls" to show.

Is there anything else I can do to show what's going on?

Thanks

Edit - Forgot to mention, this is a live server with a few hundred users at any one time! So I can't really just freely try changing config options and restart apache.

Edit 2 - The backtrace (bt) from gdb doesn't seem to be all that useful when PHP isn't configured with --enable-debug - it only shows "execute()", but I need to know what PHP script is actually running.. is there any other way?

#0  0x00007f6c143fb0c5 in ?? () from /usr/lib/apache2/modules/libphp5.so
#1  0x00007f6c143b040b in execute () from /usr/lib/apache2/modules/libphp5.so
#2  0x00007f6c1438b970 in zend_execute_scripts () from     /usr/lib/apache2/modules/libphp5.so
#3  0x00007f6c14337fe3 in php_execute_script () from     /usr/lib/apache2/modules/libphp5.so
#4  0x00007f6c1441ae7d in ?? () from /usr/lib/apache2/modules/libphp5.so
#5  0x00007f6c18912508 in ap_run_handler ()
#6  0x00007f6c1891297e in ap_invoke_handler ()
#7  0x00007f6c18922570 in ap_process_request ()
#8  0x00007f6c1891f398 in ?? ()
#9  0x00007f6c18918fa8 in ap_run_process_connection ()
#10 0x00007f6c189271d0 in ?? ()
#11 0x00007f6c1892793a in ?? ()
#12 0x00007f6c189284e7 in ap_mpm_run ()
#13 0x00007f6c188fd4a4 in main ()

Well, in case you're feeling brave:

gdb -p 20788

then issue bt to see the stack-frame, for e. g.

And BTW, there's also ltrace to mention — try it as well.

UPD.: well, ok, since now we have an idea that Apache is really running something, why wouldn't ya look at mod_status output — Extended one?

An very easy approach is to use htop. You can sort for the high CPU processes and then use

• s for strace a process
• l for lsof to see the open files of a processes
• L to ltrace.

I found that at least one of that options finds the script that generates the load and you can of course use this on a production web server to debug.

You could try:

• iotop (showing I/O on the system)
• netstat -t (showing connections)
• Take a look at the apache logfiles and find out what the server did last
• set some RLimits for the apache process. When these limits are reached the process will be killed, giving you some more information