Running out of swap space on web servers, what to do?

I have 2 LAMP web servers that are routinely running out of swap space (see attached top screenshot).

Apache's settings are as follows:

<IfModule prefork.c>
StartServers       64
MinSpareServers    64
MaxSpareServers    128
ServerLimit        256
MaxClients         256
MaxRequestsPerChild  4096
</IfModule>

The resource limits on PHP are:

max_execution_time = 30
max_input_time = 30
memory_limit = 80M

I'm new to server admin stuff like this (I'm a developer); What can I adjust so that I can prevent swapping?

Here is a screenshot of top on one of the servers when crashing:

Solution 1:

You're running out of swap because you're using all your RAM and then some. You have a serious problem that you need to rectify right now.

You have two choices: ignore cause and just add more RAM, or target the problem of what's actually munching on your memory.

Adding RAM is fairly cheap and fairly easy if it's your server but it's a temporary fix and if it's a VPS or a rented server, it's not so cheap. Let's have a go at fixing the root problem instead. What's sucking in that much memory? Here are a few tips:

• Turn off InnoDB (unless you need it) in MySQL
• Beat Apache (and MySQL) with the Stick of Configuration +5
• Consider a smaller httpd like cherokee, lighttpd or nginx (they're really fast and eat almost no RAM). Main downside is you can't use .htaccess files but you can hard-code in their functionality.
• Are you using an OP-code cache for PHP? Try turning it off or switching to another, more efficient one.

In terms of just getting swap items back into real RAM, you can do that by:

swapoff -a && swapon -a

But don't try then when you've got less free memory than you have things in swap. It'll crash your server.

Solution 2:

For a quick change do:

"MaxRequestsPerChild 4096" to something like: 700 will help. The longer an apache process lives the more resident memory it's going to consume due to mod_php and the like. Also, enable keepalive and place aggresive timeout settings for it:

KeepAlive On
MaxKeepAliveRequests 200
KeepAliveTimeout 5

This will allow each client page request to use one apache process to handle all it's requests where it would otherwise use multiple apache processes. This will cut down on the amount of apache processes running at any given time.

For optimal memory and requests per sec:

Move away from mod_php and use fastcgi, or another app server, instead. Apache processes consume a negligible amount of memory when php pages are served by fastcgi. Not to mention fastcgi can keep persisten connections to your dbase server amongst other things.