Apache is not responding and nothing is logged after short, strong "traffic wave"

I think that when your connections spike at more than 450 per second it may relate to the fact that you're running out of ephemeral ports in Linux.

Check out this previously answered question

Small abstract from the answer:


sysctl net.ipv4.ip_local_port_range
sysctl net.ipv4.tcp_fin_timeout

The ephermal port range defines the maximum number of outbound sockets a host can create from a particular I.P. address. The fin_timeout defines the minimum time these sockets will stay in TIME_WAIT state (unusable after being used once). Usual system defaults are:

net.ipv4.ip_local_port_range = 32768 61000
net.ipv4.tcp_fin_timeout = 60 

This basically means your system cannot guarantee more than (61000 - 32768) / 60 = 470 sockets at any given time. If you are not happy with that, you could begin with increasing the port_range. Setting the range to 15000 61000 is pretty common these days. You could further increase the availability by decreasing the fin_timeout. Suppose you do both, you should see over 1500 outbound connections, more readily.


Can you attach to the running unresponsive process and see what happens? Might be easier if you run prefork.

Attaching to the process using trace

strace -p <pid> -o /tmp/somefile

You might want to play with -s

-s strsize Specify the maximum string size to print (the default is 32). Note that filenames are not considered strings and are always printed in full.


I agree with 3molo, strace can give you a hint of what is going on i.e. if there are system calls that are hanging. The one thing I haven't found strace to be helpful with are slow io issues. Running

sudo iotop

and

sudo top

Can give a bit of insight as to what sort of IO activity is taking place. Slow IO has caused similar behavior for me, in the past; such as having to read many very small files from a slow NAS. If top reports a high 'wait' and iotop shows a high percentage of bandwidth, you may need to apply a different storage solution.