HAProxy check does not check content on IIS

iis haproxy healthcheck

I am new to HAproxy and I have read the docs googled every possible phrase I could think of but I cannot get haproxy to read the content my test page returns.

Here's the setup: Single arm config - 1x CentOS 7 haproxy 1.5.14 - 2x Windows Server 2012r2 IIS 8.5

Backend Server config:

backend mt-http
balance     roundrobin
mode http
option httpchk /check.aspx?appserver=dev-cluster.xxxx.com&databaseserver=test.xxxx.com&database=######dev
http-check expect string 200\ OK
server  WebLB-test2 xx.xx.xx.xx:80 check
server  WebLB-test1 xx.xx.xx.xx:80 check

check.aspx connects to a specific database through a named app server cluster to confirm end to end connectivity from the web server through database. If the test is successful the check returns 200 OK in a web page as text. If one of the components is unavailable check.aspx returns a 500 error. Success:

 <!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml">
 <head><title>
        Mediatools Check
 </title></head>
 <body id="bodyID">200 OK</body>
 </html>

Fail:

 <html xmlns="http://www.w3.org/1999/xhtml">
 <head><title>
     Mediatools Check
 </title></head>
 <body id="bodyID">500 Internal Server Error</body>
 </html>

When I use the option "http-check expect string 200\ OK" (I have tried with and with out the "\" escape character in the string) the servers show down and the error is "Layer 7 Invalid Response: http content check found empty response body."

When I use the option "http-check expect status 200 OK", the return page is successful whether the status text is 200 or not (because I assume, the return of check.aspx is successful even if returns 500 code).

Any help would be greatly appreciated.

Here is the output from a curl -v: Looks like maybe my request is too long?

 curl -v xx.xx.xx.xx/yourDB/check.aspx?appserver=dev-cluster.yourdomain.com&databaserver=test.yourdomain.com&database=yourDBdev
 [1] 16077
 [2] 16078
 [root@dev-cluster log]# * About to connect() to xx.xx.xx.xx port 80 (#0)
 *   Trying xx.xx.xx.xx...
 * Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 80 (#0)
 > GET /customer/check.aspx?appserver=dev-cluster.yourdomain.com HTTP/1.1
 > User-Agent: curl/7.29.0
 > Host: xx.xx.xx.xx
 > Accept: */*
 > 
 < HTTP/1.1 500 Internal Server Error
 < Cache-Control: private
 < Content-Type: text/html; charset=utf-8
 < Server: Microsoft-IIS/8.5
 < X-AspNet-Version: 4.0.30319
 < X-UA-Compatible: IE=EmulateIE7
 < Date: Wed, 17 Feb 2016 22:16:59 GMT
 < Content-Length: 3428
 < 
 <!DOCTYPE html>
 <html>
  <head>
    <title>Runtime Error</title>
    <meta name="viewport" content="width=device-width" />
    <style>
     body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} 
     p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
     b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
     H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
     H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
     pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
     .marker {font-weight: bold; color: black;text-decoration: none;}
     .version {color: gray;}
     .error {margin-bottom: 10px;}
     .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
     @media screen and (max-width: 639px) {
      pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
     }
     @media screen and (max-width: 479px) {
      pre { width: 280px; }
     }
    </style>
</head>

<body bgcolor="white">

        <span><H1>Server Error in '/customer' Application.<hr width=100% size=1 color=silver></H1>

        <h2> <i>Runtime Error</i> </h2></span>

        <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">

        <b> Description: </b>An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine.
        <br><br>

        <b>Details:</b> To enable the details of this specific error message to be viewable on remote machines, please create a &lt;customErrors&gt; tag within a &quot;web.config&quot; configuration file located in the root directory of the current web application. This &lt;customErrors&gt; tag should then have its &quot;mode&quot; attribute set to &quot;Off&quot;.<br><br>

        <table width=100% bgcolor="#ffffcc">
           <tr>
              <td>
                  <code><pre>

 &lt;!-- Web.Config Configuration File --&gt;

  &lt;configuration&gt;
  &lt;system.web&gt;
    &lt;customErrors mode=&quot;Off&quot;/&gt;
&lt;/system.web&gt;
 &lt;/configuration&gt;</pre></code>

              </td>
           </tr>
        </table>

        <br>

        <b>Notes:</b> The current error page you are seeing can be replaced by a custom error page by modifying the &quot;defaultRedirect&quot; attribute of the application&#39;s &lt;customErrors&gt; configuration tag to point to a custom error page URL.<br><br>

        <table width=100% bgcolor="#ffffcc">
           <tr>
              <td>
                  <code><pre>

 &lt;!-- Web.Config Configuration File --&gt;

 &lt;configuration&gt;
 &lt;system.web&gt;
    &lt;customErrors mode=&quot;RemoteOnly&quot; defaultRedirect=&quot;mycustompage.htm&quot;/&gt;
&lt;/system.web&gt;
 &lt;/configuration&gt;</pre></code>

              </td>
           </tr>
        </table>

        <br>

</body>
 </html>
  * Connection #0 to host xx.xx.xx.xx left intact

 [1]-  Done                    curl -v xx.xx.xx.xx/customer/check.aspx?appserver=dev-cluster.yourdomain.com
 [2]+  Done                    databaserver=test.yourdomain.com

Okay one more time! I had the Web developer pull the appserver, databaseserver, and database parameter from the web.config instead of sending them to the check.aspx

The new config looks like this:

 backend mt-http
 balance     roundrobin
 mode http
 option httpchk GET /customer/check.aspx
 http-check expect status 200 OK
 server  WebLB-test2 xx.xx.xx.xx:80 check
 server  WebLB-test1 xx.xx.xx.xx:80 check

New Return from Curl:

 curl -v xx.xx.xx.xx/pgglobal/check.aspx
 * About to connect() to xx.xx.xx.xx port 80 (#0)
 *   Trying xx.xx.xx.xx...
 * Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 80 (#0)
 > GET /customer/check.aspx HTTP/1.1
 > User-Agent: curl/7.29.0
 > Host: xx.xx.xx.xx
 > Accept: */*
 > 
 < HTTP/1.1 200 OK
 < Cache-Control: private
 < Content-Type: text/html; charset=utf-8
 < Server: Microsoft-IIS/8.5
 < Set-Cookie: ASP.NET_SessionId=whvmaboyg03lsl3rd1gcsbxl; path=/; secure; HttpOnly
 < X-AspNet-Version: 4.0.30319
 < X-UA-Compatible: IE=EmulateIE7
 < Date: Wed, 17 Feb 2016 23:00:07 GMT
 < Content-Length: 162
 < 


 <!DOCTYPE html>

 <html xmlns="http://www.w3.org/1999/xhtml">
 <head><title>
Happiness Check
 </title></head>
 <body id="bodyID">Success</body>
 </html>
 * Connection #0 to host xx.xx.xx.xx left intact

haproxy still reports "Layer 7 wrong status: HTTP status check returned <500>"

Here is the curl output on a failed check.aspx (one of the components checked is powered off to force the error page).

 curl -v xx.xx.xx.xx/Customer/check.aspx
 * About to connect() to xx.xx.xx.xx port 80 (#0)
 *   Trying xx.xx.xx.xx...
 * Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 80 (#0)
 > GET /Customer/check.aspx HTTP/1.1
 > User-Agent: curl/7.29.0
 > Host: xx.xx.xx.xx
 > Accept: */*
 > 
 < HTTP/1.1 500 Internal Server Error
 < Cache-Control: private
 < Content-Type: text/html; charset=utf-8
 < Server: Microsoft-IIS/8.5
 < X-AspNet-Version: 4.0.30319
 < X-UA-Compatible: IE=EmulateIE7
 < Date: Thu, 18 Feb 2016 17:17:01 GMT
 < Content-Length: 3428
 < 
 <!DOCTYPE html>
 <html>
  <head>
    <title>Runtime Error</title>
    <meta name="viewport" content="width=device-width" />
    <style>
     body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} 
     p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
     b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
     H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
     H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
     pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
     .marker {font-weight: bold; color: black;text-decoration: none;}
     .version {color: gray;}
     .error {margin-bottom: 10px;}
     .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
     @media screen and (max-width: 639px) {
      pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
     }
     @media screen and (max-width: 479px) {
      pre { width: 280px; }
     }
    </style>
 </head>

 <body bgcolor="white">

        <span><H1>Server Error in '/Customer' Application.<hr width=100% size=1 color=silver></H1>

        <h2> <i>Runtime Error</i> </h2></span>

        <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">

        <b> Description: </b>An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine.
        <br><br>

        <b>Details:</b> To enable the details of this specific error message to be viewable on remote machines, please create a &lt;customErrors&gt; tag within a &quot;web.config&quot; configuration file located in the root directory of the current web application. This &lt;customErrors&gt; tag should then have its &quot;mode&quot; attribute set to &quot;Off&quot;.<br><br>

        <table width=100% bgcolor="#ffffcc">
           <tr>
              <td>
                  <code><pre>

 &lt;!-- Web.Config Configuration File --&gt;

 &lt;configuration&gt;
  &lt;system.web&gt;
    &lt;customErrors mode=&quot;Off&quot;/&gt;
  &lt;/system.web&gt;
 &lt;/configuration&gt;</pre></code>

              </td>
           </tr>
        </table>

        <br>

        <b>Notes:</b> The current error page you are seeing can be replaced by a custom error page by modifying the &quot;defaultRedirect&quot; attribute of the application&#39;s &lt;customErrors&gt; configuration tag to point to a custom error page URL.<br><br>

        <table width=100% bgcolor="#ffffcc">
           <tr>
              <td>
                  <code><pre>

 &lt;!-- Web.Config Configuration File --&gt;

 &lt;configuration&gt;
  &lt;system.web&gt;
    &lt;customErrors mode=&quot;RemoteOnly&quot; defaultRedirect=&quot;mycustompage.htm&quot;/&gt;
  &lt;/system.web&gt;
 &lt;/configuration&gt;</pre></code>

              </td>
           </tr>
        </table>

        <br>

  </body>
 </html>
 * Connection #0 to host xx.xx.xx.xx left intact

Your config is perfect, except for one little thing. You haven't specified that the health check should actually fetch the page it's requesting, and as a result the contents will be blank.

According to the HAProxy docs, option httpchk by default uses the OPTIONS method, which doesn't get the body of the page.

option httpchk <uri>
option httpchk <method> <uri>
option httpchk <method> <uri> <version>

Enable HTTP protocol to check on the servers health

<method>
is the optional HTTP method used with the requests. When not set, the "OPTIONS" method is used, as it generally requires low server processing and is easy to filter out from the logs. Any method may be used, though it is not recommended to invent non-standard ones.

<uri>
is the URI referenced in the HTTP requests. It defaults to " / " which is accessible by default on almost any server, but may be changed to any other URI. Query strings are permitted.

<version>
is the optional HTTP version string. It defaults to "HTTP/1.0" but some servers might behave incorrectly in HTTP 1.0, so turning it to HTTP/1.1 may sometimes help. Note that the Host field is mandatory in HTTP/1.1, and as a trick, it is possible to pass it after "\r\n" following the version string.

You have a couple of options to solve this.

  1. You can change your method to GET:

    backend mt-http
  balance roundrobin
  mode http
  option httpchk GET /check.aspx?appserver=dev-cluster.xxxx.com&databaseserver=test.xxxx.com&database=######dev
  http-check expect string 200\ OK
  server  WebLB-test2 xx.xx.xx.xx:80 check
  server  WebLB-test1 xx.xx.xx.xx:80 check

    I've tried this on my test load-balancer and it works as expected.

  2. You can change check.aspx to emit something other than a 200 OK HTTP status when the checks fail.
    I had once had a need to check whether a particular service was running, because IIS would always return 200 OK even if the actual application was down on a backend. So I wrote a simple C# script to do just that:

    <%@ Page Language="C#"%>
<%@ Import Namespace="System" %>
<%@ Import Namespace="System.ServiceProcess" %>
<%@ Import Namespace="System.Net" %>
<%  

ServiceController sc = new ServiceController(Request.QueryString["service"]);

switch (sc.Status)
{
  case ServiceControllerStatus.Running:
    Response.StatusCode = (int)HttpStatusCode.OK;
    Response.ContentType = "text/plain";
    Response.Write("Running");
    break;

  default:
    Response.StatusCode = (int)HttpStatusCode.ServiceUnavailable;
    Response.ContentType = "text/plain";
    Response.Write("Failed");
    break;
}
%>

    You'd use it thusly:

    backend FooBar
  balance roundrobin
  mode http
  option httpchk GET /ServiceCheck/check.aspx?service=ServiceName
  http-check expect status 200
  server  WebLB-test2 xx.xx.xx.xx:80 check
  server  WebLB-test1 xx.xx.xx.xx:80 check

Related

Configure apache2 virtual host subdomain
Need a resource that lists WMI perfmon classes
Xen in routed mode: start fails
HP DL380p Gen8 and PCIe SSD? [duplicate]
Trouble with interactive service on Windows Server 2012
Quick Linux Mail Server Setup For Programming
Python cos(90) and cos(270) not 0
Gathering a sequence of unknown length in dask
Prevent user from going back to login screen after logging in successfully in React Native
Is there any equivalent to String.strip() in android
different python versions from the same command prompt in windows
How do I get around the limitation of having to place super constructor in the first line?