Apache stability on a VERY high traffic server
So I have a client with a very high bandwidth server. The hits its serving are pretty light in terms of demand - static jpeg and text.
SERVER SPECS:
- Dual Quad-Core L5520 2.26gHz (3.2 Turbo & HT) 8MB cache
- 72GB Ram
- 2 300GB SSD's (1 dedicated to /usr to keep up with logging)
- 100Mbps (unmetered) port at a very reputable datacenter
Ive currently got apache configured as follows:
- Start Servers 512
- Minimum Spare Servers 256
- Maximum Spare Serves 512
- Server Limit 7500
- Max clients 7500
- Max Requests per Child 100
- Keep-Alive Timeout 5
- Max Keep Alive requests 768
- Timeout 30
I was seeing about 2000-3000 apache access's per second last week with no issues. That translated to 80-90Mbps up and a load around 1 with peaks around 5 and momentary extreme peaks of 30-50 when apache would restart and all the processes re-spawn. Memory usage was about 40GB. Traffic has increased and I'm now seeing 3000-4000 apache access's per second and the http is unstable. Although apache is occasionally restarting, I am getting periods of no http connectivity while apache is running and there are idle workers and open slots. Load is around 2 when stable and frequently spiking which no doubt is my connection issues, but I'm not sure why. RAM usage is 50-60GB. Throughput is about 130Mbps. (I'm not sure how I'm getting over 100Mbps when my package is a 100Mbps but I've got a ticket in with the DC now to inquire if perhaps they are dropping packets as a result of my usage, but that doesn't explain my stability issues unless ack's are dropping and leaving connections to time out... maybe... not sure... never thought about the theory behind that thought till now)
I'm looking for opinions on if I have reached the operational max of what apache can handle or is there some other system limit that I may be exceeding.
Some people have told me LiteSpeed is the solution but I have no experience with it, and my client has heard bad things so opinions there are appreciated as well.
Thanks All!
First off, Apache, nginx, or LightSpeed aside - if you're running a server with 2000-3000 requests/second it's time to start thinking about dual servers and load balancing. Depending on what you're serving you can easily get more out of any of those servers, but at those rates you're serving something important (or at least high-traffic), so you want redundancy in addition to the ability to handle momentary load spikes.
Start seriously considering a load balancing infrastructure (there are plenty of questions/answers about HAProxy and the like here, and Google can help you as well).
That said, I don't believe you've hit the operational limits of Apache.
You can certainly consider other high-performance web servers (nginx is very popular), or you can consider tuning your Apache configuration for better performance.
Some Apache suggestions based on what I see in your question:
Before doing anything else, read the Apache performance tuning documentation.
MaxRequestsPerChild
is really only useful for containing resource leaks.
100 (your current value) is absolutely insane. You're churning processes which kills performance.
0 (Never kill a child) is certainly viable if all you're serving are static resources.
10000 (ten thousand, the default) is fine in almost all circumstances. 50000 (fifty thousand) is what I use for pure static HTML sites.StartServers
,MinSpareServers
andMaxSpareServers
can be tuned.
I generally setStartServers
andMinSpareServers
to the same value.
If there is a specific minimum number of spare servers you want to keep around, that is the number you should start with. A good value for this is your low-water-mark of simultaneous active connections.MaxSpareServers
should be set to 75-80% of your high-water-mark of simultaneous active connections.ServerLimit
andMaxClients
can possibly be increased.
If you have lots of free RAM and lots of free CPU, increase these numbers.
If you're running close to resource saturation, leave them as-is.Use graceful restarts
You say you are seeing "momentary extreme peaks" in your load when Apache restarts.
This tells me you're probably not using graceful restarts.
Whatever is causing Apache to restart, have it sendSIGUSR1
to Apache rather thanSIGHUP
(or heaven forbid, actually stopping and starting the entire server). This is far less abusive and disruptive to the system than a regular restart of a full stop/start.Consider other MPMs
You are almost certainly using theprefork
MPM if you're on a Unix system.
Consider the Worker MPM instead.
Tuning for the Worker MPM is a little differentSpend some cache
Apache has caching modules which can be used to hold frequently accessed data in RAM. This avoids a round-trip to the disk (or at least the filesystem layer) for frequently accessed data.
Configuring memory backed caching can give you a pretty big performance boost for a relatively small amount of memory.