How to limit number of sessions?

The problem you are ultimately trying to solve is with the capacity of the application - and that's where you should be solving the problem. None of the components you mention has anything to do with session management for an HTTP application.

There are some tricks you can apply with the recent module in iptables or using fail2ban in the opposite way to the purpose it was designed for - but these both require a very detailled understanding of the tools and the problem domain. You could implement access control at the level of these components but driven by published state information from the application on the number of sessions.

I also need a way to track the current number of sessions for monitoring purposes

Assuming, for the time being, that the application is a black box with no scope for modification / instrumentation (which is highly improbable) you can get this information from your apache logs by including the session cookie - filter or tail the logs to maintain a list of active cookies - and remove entries from the list when they coincide with the logout URL or have not been seen for the TTL.

This is not exactly what you are asking for, but I've already done the following with F5 load balancers :

• Count the number of post request on the login page.
• Delay users if logins per second is above a first limit
• Send users to a maintenance page if logins per second is above a second limit

As the website was sometimes under heavy load (horse races), it helped.

This problem is solveable both by using the RadWare AppDirector, and (for completeness) likely also by using Apache mod_security as per your excellent finding in the comment below.

For an AppDirector solution I believe it is possible to create two farms mapping to the same backend server(s). These farms can have different criteria and operating conditions applied to them. One farm would be the "default" and the other would answer to URI:s which you define as being "a session". The latter would get a limit to the amount of sessions it accepts in the load balancer.

I am from now on going to substitute your "session" term for "logged in" for two reasons:

• It avoids ambiguity as it clearly defines the desired state in that the user is authenticated.
• The AppDirector User Guide and GUI redefines the term "connection" to have a meaning for all practical purposes identical to "session", see below. This adds confusion which we try to avoid.

It is also possible to show a sorry page if the "logged in" farm has reached the chosen connection limit.

Before getting to the how, I must clearly state I have no operating experience of the AppDirector product, but do administer a competing and slightly less advanced load balancer on a daily basis. The product I use can do this scenario right off the bat. I have found information through the AppDirector User Guide and what online documentation is available which suggests that the same is true for the AppDirector. However whilst concepts are similar, the terminology is different. I am simply doing a when-in-rome act with regards to wording, hoping to get it fairly right without being too obviously a clueless moron.

The greatest roadblock was getting access to a manual, which is not made available unless one is an active customer. Through some googling it was possible to find an old version which I hope is not too out of date, I also found a couple of knowledgebase articles and this link: Radware AppDirector – Configuration: Basic Application.

Here is a solution draft, as interpreted mainly through the User Guide:

Client entry to the load balancer is done through a VIP which is used to connect both the "default" sessions and the "logged in sessions". This is achieved through a L4 policy as per p.99 in the User Guide:

"When AppDirector receives the first packet of a session destined to a
Virtual IP address, it searches for a Layer 4 Policy that matches the
Layer 4 Protocol, Destination port, Source IP, etc. Then, based on this
information, AppDirector selects the farm allocated to this service and
the best server for the task from that farm, and forwards the packet to
that server.

The L4 policy can be tied to L7 policies which are used to select a suitable farm. The L7 policy process is described thus in the User Guide p.104:

"The Layer 7 content aware decision making mechanism allows you to have
a single point of entry to the site, and provides differentiated service
for different user groups.

A Layer 7 decision is made using a mechanism called Delayed Binding.
When Delayed Binding is used, AppDirector first performs a TCP handshake
with the client to receive the HTTP request. AppDirector parses the HTTP
request’s data, usually HTTP headers, and performs the load balancing
decision. Only after that, does AppDirector select a farm and a server.
Lastly, AppDirector initiates a TCP handshake with the server and
forwards the traffic to it
[...]
When Layer 7 Policies are used, farm selection is based on matching the
request data with a list of Layer 7 Policies defining the Layer 7
parameters differentiating the service. The process of server selection
within the farm can also be content-based, using a third Layer 7
parameter."

The methods available to define an L7 behaviour are decribed on p.106, of which you could pick a suitable method to choose routing to your "logged in" Farm rather than to the "default" Farm:

"Methods are the basic building blocks for Layer 7 service selection.
They define content by which traffic is differentiated. You can use
the same Method to select one or more services. The following Method
Types are available:

- URL: Looks for a specified host name and/or path in the HTTP request.
- File Type: Looks for a specified File Type in the HTTP request.
- Header Field: Looks for a specified Header Field in the HTTP request.
- Cookie: Looks for a specified Cookie in the HTTP request.
- Regular Expression: Looks for a regular expression anywhere in the
HTTP request. AppDirector supports Posix 1002.3 regular expressions;
the string can be up to 80 characters.
- Text: Looks for a text string anywhere in the HTTP request."

As seen in the Basic Application link, one could for instance create an L7 policy evaluating URI patterns for routing to different farms. The made up URI patterns '^/login?=true' and '^/loggedin' could be routed to your "logged in" farm. The made up pattern '^/logout' (and all other URI:s) could similarly be routed to a "default" farm.

A Farm is defined by the User Guide p.121 thus: "An AppDirector farm is a group of networked servers that provide the same service [...] A server that provides multiple services can be used in multiple farms."

A server is further differentiated through separating the definition of a backend server into two layers, the 'Physical Server' object layer which represents the ip address of a server and the 'Farm Server' object layer which represent services running on one or more Physical Servers.

Session limiting on a farm can according to the 'AppDirector User Guide' be done per each Farm Server object defined for a farm (as well as through other means) in addition to per Physical Server object. This is described amongst other places on p.137:

"The Connection Limit is the maximum number of users that can be directed
to a server for a service provided by the farm. The number of users allowed
depends on the Sessions mode selected because it determines the number of
active entries in the Client Table for sessions destined to the specific server.

When the Entry Per Session or Server Per Session modes are selected, the number
of active entries destined to the same server is higher than in the Regular
mode (see Regular, page 153).

When the Regular mode is selected, all requests from a single client IP destined
to the same server are reflected by a single entry in the Client Table (see
Client Table Views, page 164).

The default value for the Connection Limit parameter is 0. When it is configured
to 0, it is disabled for this server and there is no user number limit."

The Client Table and its 'Regular mode' is defined on p.153:

"The Layer 3 Client Table is always used when Entry Per Session is used.
AppDirector uses the Layer 3 Client Table to ensure Layer 3 persistency.

This table contains information about the server selected for each client
(Source IP address) in each farm, and it allows AppDirector to select a
server for a new session.
[...]
In the Regular mode, AppDirector maintains Layer 3 persistency. In this mode,
each entry is identified by the following parameters:
• Layer 4 Policy VIP Address
• Client IP Address
• Destination TCP/UDP Port Used from the Client to the Server"

In a screenshot of a server definition window on the Basic Application page, the server connection limit box is seen right beside the bandwith limit box.

So a bit depending on configuration but for the purposes of this answer, a 'connection' as defined through the Client Table and a 'session' as defined by you essentially ends up being the same thing. And a limit to that effect can be imposed per server object in a farm.

As the AppDirector differentiates between physical servers and farm servers, it would be possible to define two farm servers mapping to your Apache physical server object, one having a low connection limit.

However, Apache also needs to answer calls from both farm server objects, for instance through being called on two separate ports or ip addresses - one being used by each (farm/farm server) combo. The question then becomes, are you able to define two application server entry points? i.e. are you able to equip your Apache front end application(/vhost?) to answer on two ports or IP addresses (one per farm)? This is through a bit of guess work as I do not wish to spend too much time with the manual, but I'm sure you could solve this fairly elegantly when actually looking at the AppDirector GUI and the Apache.

Setting the connection limit has a little quirk. From Physical Servers, Connection Limit p.140:

"Connection Limit

Maximum number of Client Table entries that can run simultaneously on 
the physical server. This depends on the farm’s Sessions mode (see 
Sessions Modes, page 150). When the limit is reached, new requests are 
no longer directed to this server. All open sessions are continued.

When the Connection Limit parameter is configured to 0 (default), this 
mechanism is disabled for this physical server and there is no user 
number limit.

Note: When configuring the physical server, ensure that the Connection 
Limit in the farm servers with the same Server Name is lower than or 
equal to the Connection Limit in the physical server. Total number of 
active sessions that run simultaneously on the farm servers must not 
be higher than the Connection Limit value defined on the physical server."

You would therefore need to define a very high Connection Limit (with a wide margin to the max number possible through your user base) for the unrestricted, "default" farm server, and set the Connection Limit for the "logged in" farm server as low as you have to. The physical server definition would need to have the sum of the two as its Connection Limit, as a precondition to activating the desired session limit.

You also have this requirement in your question:

After the specified session limit has been reached, the next user should be
directed to a custom error page.

This is termed a 'No HTTP Service Page' in the User Guide, p.134:

When all servers belonging to a farm cannot be used for a specific
session, AppDirector can reply to a Web request (destined to port 80)
with a simple Web page, indicating that the service is currently not
available. Servers that cannot be used for a session include servers
in Not In Service or in No New Sessions mode. No HTTP Service Page is
configured for each farm. Each Web page is limited to 1K of HTML code.

For the monitoring part I have not done as thorough research but here is what I think:

track the current number of sessions for monitoring purposes

AppDirector seems to have MIBs. Probably a pain to find the right OID as it usually is, but you can probably snmp it to your tool of choice.

whitelist the monitoring server (which is issuing queries to the webapp
periodically) and exempt it from the limit.

This one could require some creative thinking. Assuming the AppDirector doesn't include a template for this right out of the box, how about:

• URIs outside of the "logged in" farm would not be affected by the session limit. So monitor away, it's the same backend server(s) anyway.
• Use the AppDirector health checks instead, they will likely not count toward the session limit you impose. Find a way to pass alerts to your monitoring server though :-)
• Set up a third farm, through which you pass health checks. Messy, but it would work.