investigating apache benchmark failed request

This is an issue with dynamic pages, it happens because the Content-Length can vary between the requests. When using ab with such pages you need to use the -l option.

-l              Accept variable document length (use this for dynamic pages)

The failed requests are all based on length - that is the number of responses that didn't match exactly the byte count stated. This will be due to dynamic content such as adverts etc that are served differently each time so this is nothing to worry about.

Martin is correct. More helpfully, I ran a quick curl against our production servers:

{user@staging:~}$ for i in `seq 1 10`; do curl -sk https://app.copperegg.com/login >    /tmp/lb$i.txt ; done
{user@staging:~}$ wc /tmp/lb*
   74   239  3316 /tmp/lb1.txt
   74   239  3324 /tmp/lb10.txt
   74   239  3320 /tmp/lb2.txt
   74   239  3316 /tmp/lb3.txt
   74   239  3316 /tmp/lb4.txt
   74   239  3316 /tmp/lb5.txt
   74   239  3320 /tmp/lb6.txt
   74   239  3316 /tmp/lb7.txt
   74   239  3316 /tmp/lb8.txt
   74   239  3316 /tmp/lb9.txt
  740  2390 33176 total
{user@staging:~}$ diff /tmp/lb1.txt /tmp/lb10.txt  
7c7
<   var g_time_offset = new Date().getTime() - 1318965621000;
---
>   var g_time_offset = new Date().getTime() - 1318965622000;
15c15
< <meta name="csrf-token" content="bi9pgbUoUQLOwB3V8fT1E40sV06x4914ybLSvnfEeeg="/>
---
> <meta name="csrf-token" content="GL&#47;RRZCf2Zk&#47;AQzRgEW2U4Iv3htD1hodt2qfp4jwIxQ="/>
23c23
<   <form accept-charset="UTF-8" action="/authenticate" id="loginForm" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;" /><input name="authenticity_token" type="hidden" value="bi9pgbUoUQLOwB3V8fT1E40sV06x4914ybLSvnfEeeg=" /></div>
---
>   <form accept-charset="UTF-8" action="/authenticate" id="loginForm" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;" /><input name="authenticity_token" type="hidden" value="GL/RRZCf2Zk/AQzRgEW2U4Iv3htD1hodt2qfp4jwIxQ=" /></div>
{user@staging:~}$ diff /tmp/lb1.txt /tmp/lb2.txt 
7c7
<   var g_time_offset = new Date().getTime() - 1318965621000;
---
>   var g_time_offset = new Date().getTime() - 1318965622000;
9,10c9,10
< <link href="/stylesheets/application.css?1318747862" media="screen" rel="stylesheet" type="text/css" />
< <script src="/javascripts/cache/application.js?1318747811" type="text/javascript"></script>
---
> <link href="/stylesheets/application.css?1318747582" media="screen" rel="stylesheet" type="text/css" />
> <script src="/javascripts/cache/application.js?1318747448" type="text/javascript"></script>
15c15
< <meta name="csrf-token" content="bi9pgbUoUQLOwB3V8fT1E40sV06x4914ybLSvnfEeeg="/>
---
> <meta name="csrf-token" content="BMZKKUZ3WFhQrCIewQ81VuArEtUp8gc6ccr0Wi3&#47;sqE="/>
23c23
<   <form accept-charset="UTF-8" action="/authenticate" id="loginForm" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;" /><input name="authenticity_token" type="hidden" value="bi9pgbUoUQLOwB3V8fT1E40sV06x4914ybLSvnfEeeg=" /></div>
---
>   <form accept-charset="UTF-8" action="/authenticate" id="loginForm" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;" /><input name="authenticity_token" type="hidden" value="BMZKKUZ3WFhQrCIewQ81VuArEtUp8gc6ccr0Wi3/sqE=" /></div>
{user@staging:~}$ 

Notice that we're seeing a "content" string that has a nice ugly string of characters. Notice that the '/' character is in the "form" line, but in the "meta" line, it's replaced with "/". This accounts for the 4 or 8 character difference between my request lengths.

It's irritating that apachebench isn't smart able to account for this, but at least we can see the cause.