Degraded httpd performance after moving from RHEL 5.6 to 6.1

We have recently replaced an RHEL 5.6 web server with RHEL 6.1. For both environments, the (stock) Redhat shipped Apache httpd has been used (i.e. yum install httpd). The server is serving PHP content and is busy (serving approximately 2500 - 4000 page requests per minute). The specifications of both servers are identical in terms of memory, storage and network connection. What we are seeing is significantly higher load averages on the RHEL 6.1 box - where the load average will (at times) spike to over 40 (all httpd processes) resulting in substantially degraded performance of the site. We have monitored the RHEL 5.6 environment and the load average does not exceed approximately 5 concurrent httpds. How can we investigate this issue ? Please bear in mind that this is a production environment, but we can compare "apples with apples" by switching between the 5.6 and 6.1 server.

Solution 1:

Your 2500 - 4000 requests per minute translates to 40-60 requests per second. That kind of load hardly needs kernel level tuning, but more likely there is something wrong with your Apache or PHP setup. Some typical reasons include

• Long TimeOut values in httpd.conf
• KeepAlive on and/or long KeepAliveTimeOut values in httpd.conf (can lead to extra httpd processes)
• Some unnecessary httpd module loaded
• Some unnecessary PHP module loaded
• Misconfigured memcached setup (if used)
• Database connectivity/configuration problem

You need to figure out what's different compared to your old server. Did you copy the configuration values from the old server to new one? Was the old server fine-tuned years ago and you have forgotten something vital?

What does Apache's server-status page show you?

If everything else fails, you can always use PHP's XDebug module. That means you perform page loads at your server, and let XDebug generate a Valgrind compatible report for you. Then you can analyze that file with KCacheGrind or some other analyzer and see where the precious CPU time gets consumed. That can give you a clue about what's wrong.

Solution 2:

Upgrading from 5.6 to 6.1 is a big step - you're probably running different versions of both apache, php, and perhaps a bytecode cache as well.

If you're using a bytecode cache (which I believe you should, with the kind of load you're describing), you should verify that it's working (I know apc has changed some configuration syntax between versions, for example). Which one are you in that case using?

You should have some metrics explaining what kind of cpu time is spent - if it's IOWait, System or User time. The same applies to memory usage, etc. How does these things look?

I will again recommend some kind of monitoring tool, like for example Munin.

Solution 3:

Perhaps you can compare the output of sysctl -A on both systems? You may have tuned your RHEL5 kernel. I've also found oprofile to be useful to determine where the kernel and processes are spending time.

Edit: I assume apache and php are configured the same way.

Solution 4:

Question: Is the both apache configurations the same i.e. are you using thread invokations or process invokations for each request?

If you are using process invokations, then a process is created for each request (with a few remaining behind) and thus a higher load factor.

If you are using thread invokations, then a number of requests will be handled by threads within a far fewer number of processes.

Thus, there might be nothing wrong.