Increased number of 408 Request Timeouts for null on a webserver

In my log watch records there has been a steadily growing progression of these:

  408 Request Timeout
      null: 694 Time(s)

On my webserver.

Here are some of what look like the contributing requests from the /var/log/apache2/access.log access log:

ip - - date requestsfor"-"? httpcode bytes referrer useragent
75.149.117.146 - - [28/Jan/2013:17:49:47 -0500] "-" 408 0 "-" "-"
65.55.215.247 - - [28/Jan/2013:17:57:40 -0500] "-" 408 0 "-" "-"
205.157.206.75 - - [28/Jan/2013:18:00:21 -0500] "-" 408 0 "-" "-"

Normal access request examples of course have a lot more relevant info like this:

ip - - date request-for httpcode bytes referrer useragent
66.251.23.171 - - [28/Jan/2013:17:45:41 -0500] "GET /images/al/al-mb0608tn.jpg HTTP/1.1" 200 4085 "http://example.com/brands.php?F=S&BrandCode=AL" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

See a larger sampling of my access log here (with a few normal get requests that 408 in with the rest)

I did reverse IP lookups on the ips, and they seem to come from diverse locations in the US and Canada. That could just mean that there is a proxy involved, I suppose? There is a large block from:

96.42.74.117 - - [18/Feb/2013:02:55:58 -0500] "-" 408 0 "-" "-"

That repeats frequently.

I hesitate to jump to conclusions that this is an attack as opposed to a fault, but the number of probes recorded has been steadily increasing at around the same time, e.g. logwatch also says

 A total of 125 sites probed the server
    107.22.9.89
    108.132.76.100
    108.172.60.59
    108.226.133.142
    12.166.56.82
    12.54.94.24

.... on and on with a list of various ips that used detected probes against the server. There is some overlap of the ips listing as "probing" the server and the ips that hit the access log with nulls, so that may suggest an attack, but since the server has to serve requests, it'll be hard to tell a legit timeout from a DOS request timeout attack if that's what is going on here.

How do I debug this problem, or if it is an attack, deal with this attack?

Solution 1:

From some researching I did, these are some possibilities leading to 408 message. You can test each to identify the relevant one for your case.

1) Very low Apache TIMEOUT value - your web server could be ending the session before the client even get the chance to send a request.

too many 408 error codes in access log

2) browser predictive optimization - you can easily test this with different browsers. Simply do tail -f /var/log/apache2/access.log, while using relevant key words which would bring up your site on first search page, hover over the link pointing to your site, check if the site preview appears.....all these without clicking on the link which opens your website.

http://forum.linode.com/viewtopic.php?f=10&t=8048

3) denial of service attack - ddos warms such as slow loris can open too many connections to the server without sending a single bye tying up all the Apache processes.

http://blog.spiderlabs.com/2011/07/advanced-topic-of-the-week-mitigating-slow-http-dos-attacks.html

A quote from above linke -

"The trick is to open a connection to the server but not send a single byte. Opening the connection and waiting requires almost no resources by the attacker, but it permanently ties up one Apache process to wait patiently for a request. Apache will wait until the timeout expires, and then close the connection. As of Apache 1.3.31, request-line timeouts are logged to the access log (with status code 408). Request line timeout messages appear in the error log with the level info. Apache 2 does not log such messages to the error log, but efforts are underway to add the same functionality as is present in the 1.x branch."

Solution 2:

The majority of the 408 code server responses seem to be due to client timeouts rather than look aheads. I'm ruling out denial of service in the cases mentioned, my servers, and the referenced write-ups because the number of hits is low. I have tried but haven't been able to reproduce any 408 errors through look aheads in Safari, Chrome or Firefox under OS X. I also do not see them in most months on a local (non public) server which I would expect if they were normal browser behavior. It could be browsers looking ahead by opening connections and never using them for a request but I don't see that behavior with enough prevelance to account for the numbers of 408s on these public servers. While opening a connection with an empty request does get the connection started it makes more sense for a look ahead to be grabbing a resource or using OPTIONS (Which I also only see only in rare cross site and spider situations).

I did a DNS host lookup on a list of 47 IPs receiving 408 responses without an identified resource, agent or request sizes in the log from a lightly used public server. I ignored an additional 67 IPs where they differed by only the last tuple and represented them with the first in each set. So this is 47 different client networks/ISPs. Apache's TimeOut is the default 60 seconds. Of the 47, 20 produced machine names. Of these machines 10 (50%) were in mainland China, 5 in the US (25% and all OH, OK and NH) and 5 between Brazil, UK, Italy and Taiwan (ROC). I also suspect that a good proportion of the 27 IPs that did not result in a machine name are also in China but can't show that.

Of the 67 IPs ignored as their block was already represented 63 were Baidu spiders: baiduspider-*.crawl.baidu.com (If that seems excessive bear in mind that it is now larger than Yahoo or Bing in terms of searches per month - Second only to Google).

If it were modern browser behavior causing them I would have expected a higher proportion in the US. The server I logged is in The Bay Area. Instead I think this is primarily network congestion causing connections to take too long to setup and allow the full request to be sent and received before the connection is dropped or timed out by Apache. In particular the extreme congestion due to demand growth, inadequate cross border bandwidth and national firewall infrastructure in China. It could also be deliberate forged connection resets but there's no reason to expect that for this. The remainder being home users with poor connections or unfortunate equipment situations elsewhere.

I think the problem is primarily a client end connectivity issue but you may want to have a conciliatory, and non graphical, 408 error page. Though judging by the 0 size shown in the log for the outbound response, Apache isn't using the error page.