Varnish running out of open ports, lots of SYN_SENT connections

Your problem is probably with the sysctl on the Apache servers.

Some assumptions: Typically Varnish is vastly faster at processing each connection than a webserver (unless perhaps your Varnish servers have much less CPU, and your Apache servers are only serving static files cached in memory.) I'm going to assume your connections process faster in Varnish than Apache.

Therefore, resources on your Apache servers may be ample, but requests will have to queue up somewhere, if only very briefly. Right now they are not queuing up in a healthy way where they eventually get processed.

It seems like your requests are getting stuck in Varnish and not making it to the Apache servers.

There is some evidence for this:

Notice in your munin graph, before the SYN_SENTs get backed up, requests in TIME_WAIT increase, then after a point, they start to pile up as SYN_SENTS. This indicates that requests are beginning to be answered more slowly, then the queue backs up and requests don't get answered at all.

This indicates to me that your Apache server isn't accepting enough connections (where they can then sit and queue up for Apache to process them.)

I see several possible limits in your config file:

When you have spike, you have approximately 30000 connections in the SYN_SENT state on your Varnish server.

However, on the Apache server your max_syn_backlog is only 16384. Your somaxconn is only 2048.

Notice also the size of your network memory buffers on the Apache servers is very low. You have adjusted them on the Varnish server to 16MB. But on the Apache server your net.ipv4.tcp_rmem is only 524KB to match your net.core.rmem_max.

I recommend raising all of these parameters on the Apache server.

You're going to need to focus more on the diagnostics on the Apache server to find out exactly what is going on, but you might not need to if you raise these values.

You should probably not adjust net.ipv4.tcp_mem. Notice the unit for this parameter is in pages, not bytes, thus copying the same value from net.ipv4.tcp_rmem or net.ipv4.tcp_wmem (both in bytes) doesn't make any sense. It is auto tuned by linux based on your amount of memory so it rarely needs adjustment. In fact this may be your problem, arbitrarily limiting the memory available for overall connection queuing.

See: http://russ.garrett.co.uk/2009/01/01/linux-kernel-tuning/

Also notice your "vm.swappiness = 0" is set twice, once as 10, and again at the bottom as 0, which is the effective value.