How can I monitor what process is swapping?

Solution 1:

iotop is probably what you're after. You'll need Python 2.5 or greater installed, which is likely. You'll also need a couple of kernel (2.6.20 or later) options enabled (TASK_DELAY_ACCT and TASK_IO_ACCOUNTING) so the right metrics are exposed in /proc for iotop to use them.

Solution 2:

You should definitely remove all unnecessary modules from apache. It matters a lot. Turn off extended status, auto indexing, etc.

setting AllowOverride as "None" and doing any htaccess stuff in the server config will help with I/O and reduce response time, and that will result in less memory usage per minute.

See apache performance guide.

Also look for postgresql documentation for improving performance and reducing memory usage.

Probably your biggest memory hog is the python program, as python is not very optimized for low-memory usage. You can try a profiler on the application.

Solution 3:

If the activity is while performing many HTTP requests, and not at other times, then it will be the Apache and python processes that are actively causing swap activity.

There is not much more than this we can say without a little more info.

• How are you running the python scripts? (CGI, FCGI, mod_python, other?)
• How many concurrent requests was ab told to attempt?
• How much memory is in the machine? (it could just be that the machine is undersized for the job)
• How many Apache processes are running?
• If FGCI, how many Python processes do you have running?

In general to reduce Apache memory use you need to reduce the number of child processes it will start by altering the config files, the same for the number of python processes if you are using FastCGI.