How to configure Apache "workers" for maximum concurrency

Hoping this is a good question and can be useful for anyone in a similar situation. I recently took over from a very senior Linux administrator in my company, and for the first time I've had to set up a new server to serve our content to our customers. My predecessor left limited documentation on this matter as other things were of higher priority at the time of his departure. I've referred extensively to what documentation I had, plus configurations from previous servers and a lot of online reading to set everything up.

The "stack" of a server I am building is as follows: CentOS 7, Apache 2.4.6, MariaDB 5.X, Tomcat 7 serving JSP/Servlets. Previous installations for production used CentOS 5, older version of Apache and MySQL, and Tomcat 5.X.

Apache is the front end, it uses AJP to connect to a Tomcat backend. Most traffic will be HTTPS.

The server is, I think a Gen 9 HP server, it has 32GB RAM and SSDs in a RAID 1 array. After getting it to work, I am now concerned with maximising its ability to serve users simultaneously, I expect a burst of several hundred users would not be unusual. To that end, I've looked to prevent bottlenecks at any point in the stack: starting with Tomcat and MariaDB and taking measures specific to those, measures with which I am satisfied. Moving on to Apache however I find it more difficult to understand what I should be doing.

There are 3 differnt types of "workers" and I have 3 major problems optimising these:
1. There are three options, prefork, mpm_worker and mpm_event. I don't really understand how to configure them though I do have a general understanding of what each is used for.
2. Unlike Apache in previous servers, the options for StartServers, ServerLimit, MaxRequestsPerChild etc are not listed in /etc/httpd/conf/httpd.conf. Could there be default settings that I have to manually override? Is there any reason other than "they were added manually last time" that settings would be found in an older httpd.conf but not a new one?

I refer to this, in an older httpd.conf, I cannot see anything similar anywhere in the new configuration files, only a specification:

// prefork MPM
// StartServers: number of server processes to start
// MinSpareServers: minimum number of server processes which are kept spare
// MaxSpareServers: maximum number of server processes which are kept spare
// ServerLimit: maximum value for MaxClients for the lifetime of the server
// MaxClients: maximum number of server processes allowed to start
// MaxRequestsPerChild: maximum number of requests a server process serves

StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000

// worker MPM
// StartServers: initial number of server processes to start
// MaxClients: maximum number of simultaneous client connections
// MinSpareThreads: minimum number of worker threads which are kept spare
// MaxSpareThreads: maximum number of worker threads which are kept spare
// ThreadsPerChild: constant number of worker threads in each server process
// MaxRequestsPerChild: maximum number of requests a server process serves

StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0

Note that I replaced the hash signs with // to signify comments in the httpd.conf, because the hash sign causes the text to appear as a heading.

In a reversal of the problem I have on the new server, I cannot tell which worker is being used on the old one, there are settings for both prefork and mpm_worker, but no actual specification of which is to be used that I can find.

3. After a little tweaking, I can start the server with mpm_worker if I want (have to disable a php.conf file first) but I'm not sure if modules controlling everything are thread safe (modules for virtual hosts, openssl, mod_proxy_ajp). mod_proxy_ajp is the only thing I'm using that would be in any sense unusual.

The server isn't in production yet, so I don't have the option of getting average memory usage over large numbers of current clients to help determine the "perfect" number of maximum clients, additionally, I'm not concerned with getting a perfect number, just an adequate one.

If I want to make sure that roughly ~400 people maybe a little more can use the server at a given time, without anyone seeing http 500 type errors, what are the settings that I should use for Apache?

Thanks in advance, hope this isn't too nooby.

There seems to be a lot of possible questions here, a bit unstructured for a "response". Let me see if I can help answer some.

• Determine which Multi-Processing Modules (MPMs) is in use.

  The "easiest" way is to dump any currently running modules and select out the mpm ones to see what's running. Here are two methods:

  # apachectl -M | grep mpm
  mpm_worker_module (shared)
  # httpd -V | grep MPM
  Server MPM:     worker
  

• Change which MPM is in use

  On CentOS7, this is now controlled in a config file named /etc/httpd/conf.modules.d/00-mpm.conf. It's a short file, and only one LoadModule should be uncommented at any time.

  LoadModule mpm_worker_module modules/mod_mpm_worker.so

  A service restart is required post-change to activate the desired MPM.

• Set your desired configuration values

  Many of the module's config directives have defaults when unspecified. StartServers using worker will default to 3, when using prefork = 5. See more here: https://httpd.apache.org/docs/current/mod/mpm_common.html#startservers . I would recommend creating a new file to contain your overrides instead of editing an existing one.

  Something like /etc/httpd/conf.modules.d/10-worker.conf:

  <IfModule mpm_worker_module>
      ServerLimit             250
      StartServers             10
      MinSpareThreads          75
      MaxSpareThreads         250
      ThreadLimit              64
      ThreadsPerChild          32
      MaxClients             8000
      MaxRequestsPerChild   10000
  </IfModule>
  

As with anything, tuning the exact numbers is left up to you to test with your workloads and applications - tools like ab and siege can help.

• https://httpd.apache.org/docs/2.4/programs/ab.html
• https://www.joedog.org/siege-home/