How can I keep Apache from falling over?

Solution 1:

Magento and Zend Framework are quite CPU-heavy, as you noticed. The best way to avoid the CPU load is simply but rendering any content only once, until it changes. Most parts of your catalog don't change that often, and often only the shopping cart block on your page, or the 'most popular items' block are the only dynamic parts.

I would suggest putting a Varnish cache in front of Apache. This gives you high-performance page-caching that can seriously offload your LAMP stack. We recently survived a very public launch of a website thanks to Varnish and I was seriously impressed by the speed and low cpu-load. Varnish is free, and flexible enough to cache entire pages, or cache only the relatively static parts and include the cart dynamically.

However, Varnish will not cache much on a default Magento installation, since there's a lot of per-user dynamic content, cookies, etc. A Magento module such as 'PageCache powered by Varnish' modifies Magento to work well with Varnish. It also provides a Varnish configuration file that matches the Magento setup. These two together make for a very efficient setup. It's a commercial module, but much more affordable than a more powerful server would be.

The parts your offloading to a CDN or Nginx are not your real problem, although it does help. Even Apache can handle quite a number of static requests. You need to cache the stuff that takes effort to generate again and again, i.e. your dynamic parts.

Solution 2:

I normally setup the MaxRequestsPerChild to be in the thousands - usually, nearer 10,000.

You say that you have "the recommended PHP caching" - but do you have APC installed? Finally, how many users do you see hitting the website at the same time. If you have Apache extended stats, you will be able to see how many of the Apache processes are actually in the Running state at a time.

800 APC file hits per second, and another 200 user-cache is a lot. If that is a dual or quad-core, I'd expect it to be keeping up OK though. If the database is genuinely keeping up, then getting a bigger machine - and more CPUs, may be the best thing for it, at least right now.

Solution 3:

Your average load is entirely too high for a dual core VPS. 8 should be the max.

I've had good success with using mod_pagespeed and event MPM for Magento. I would recommending switching to using event MPM, and installing mod_pagespeed.

More info about Event MPM: Apache event MPM documentation

And mod_pagespeed: Google Code: mod_pagespeed

If you continue to have load issues even after making the above changes, you may want to consider switching to a different, better VPS plan.

Solution 4:

As Alister hints, a MaxRequestsPerChild value of 400 is absurdly low.

The load average is very high - but 60k page views per day is not a lot of traffic.

how many processes do you normally have serving requests?

I'm not familiar with Magento but it looks like there's something wrong with this setup. I would expect that you could get significantly more throughput at a lower load level.

Go get a copy of Steve Souders book and read it. Enable compression for all outgoing HTML content (static and dynamic). And make sure you've got a good caching config. Start logging %D in your access_log file and build some tools for analysing the data / isolating the slowness. Similar for MySQL.

Try mysqltuner.pl and see if it flags up any problems.