Apache - keep-alive or not keep-alive?

Solution 1:

The benefit from having keep-alive on is that a client will be able to request more than one entity from your server without having to create another TCP connection (3-way handshake with its round-trips included). The problem with this is that if, say, you have a connection limit in apache set to 300, if there's 300 active connections all the others will have to wait until the first 300 clients are done and/or the timeout expires.

Disabling keep-alive will force the clients to create 1 connection per request. When the socket is properly closed on both ends, it goes into the TIME_WAIT status, as you've noticed. This happens to ensure that the port used in that connection does not receive data from a previous connection for a while before being available (see this, but there's much more out there). In my linux system, /proc/sys/net/ipv4/tcp_fin_timeout is set to 60 seconds. You can try reducing that, but don't go too far down. How far? It depends on how many connections per second you are getting. For 100-200req/s, don't bother changing the default.

Solution 2:

On a low-memory server, you're going to run into issues with keepalives turned on at any sort of scale. This is because apache creates one thread or process per connection, allocating by default 8 MB of stack on Linux (which can be adjusted down via Apache config, to what degree is application-dependent). On your 360 MB server, that is likely to be an issue if every client is holding open at least one and possibly two connections.

This is, in fact, a major reason that event-based web servers like nginx and lighttpd were created - to allow systems to handle tens of thousands of connections using sane amounts of memory.

So, I'd turn off keppalives in your situation and live with the performance impact on clients. Or perhaps use something like nginx in front of apache to handle user-facing connections, proxying requests to Apache. This is a very common model, effectively treating Apache as "thread pool" for potentially blocking operations like disk or database access.