Apache: Force HTTP 1.1 or Persistent/KeepAlive connections for HTTP 1.0 requests

I want to force Keep Alive or Persistent connections for all HTTP requests on my Apache 2.2.3 server running on RHEL 5.8. A lot of web crawlers are using HTTP 1.0 for some reason, and I would like to either force persistent connections, or somehow force those connections to use HTTP 1.1 so that the Keep Alive On setting in the Apache config will cause persistent HTTP. This is because I want to reduce the number of TCP connections being opened. How can I accomplish this?

Solution 1:

HTTP 1.0 doesn't support keep-alive (A.K.A persistent connections) and there is no system to ask them to use HTTP 1.1 you could return them a 400x but they would have still already created a connection and there is no standard way to tell them to try HTTP 1.1 instead. Even if all your clients do work out they need HTTP1.1 there is nothing you can do on the service to stop them sending the "close" in keep-alive header or dropping the connection and still do a single connection per request.

There are some Linux kernel options (specifically DEFER_ACCEPT andTIME_WAIT) that can help with reducing the TCP connection table but I wouldn't recommend going down that route unless you know what you are doing as its really easy to make things worse rather than better.

But thinking more broadly I think your trying to solve the problem in the wrong way. The internet is dark and full of terrors, If random bots are causing you TCP connection limit issues then you are hopeless vulnerable to SYN floods or other DDOS type events and bots are just a signal that your webserver is either underscaled, under attack or misconfigured. Either way something id wrong.

If you really want to control what kind of requests arrive at apache you need a WAF(web application firewall) or Load balancer infront of apache to filter out bad requests before they consume expensive apache connections. In many cases you can also offload the CPU intensive work of doing SSL/TLS encryption.

I know this isn't a quick fix or the answer your looking for but whatever the issue is your very unlikekly to solve it going down this path.