HAproxy 503 Service Unavailable No server is available to handle this request

I'm new to this load balancing scenario and I'm being tasked to figure out how to make this load balancing works.

My environment:

Centos 6.4 64 Bit
Webserver: Lighttpd
All running in ESXI
virtual IP: 192.168.1.6
LB1: 192.168.1.4
LB2: 192.168.1.5
Webserver 1: 192.168.1.12
Webserver 2: 192.168.1.13
Gateway: 192.168.1.1

Trying to run a test in the lab prior production with HAproxy and keepalived. Here is what I have on my keepalived setting:

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.1.4
   smtp_connect_timeout 30
   router_id 192.168.1.1
}

vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 1                     # check every second
weight 2                       # add 2 points of prio if OK
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101  #priority 101 for master
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.6

    }

    track_script {
    chk_haproxy
            }

            }

and here is my setting for HAproxy

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main *:80
#    acl url_static       path_beg       -i /static /images /javascript /stylesheets
#    acl url_static       path_end       -i .jpg .gif .png .css .js

#    use_backend static          if url_static
#    default_backend             view

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
#backend static
#    balance     roundrobin
#    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
mode tcp
    balance     roundrobin
    server  server1 192.168.1.12:80 check inter 2000 rise 2 fall 5
    server  server2 192.168.1.13:80 check inter 2000 rise 2 fall 5

When I started the HAproxy, I got thiss error and I'm not quite sure where to start looking to fix it. Perhaps someone who has done this many times can help me shed some light?

503 Service Unavailable No server is available to handle this request. 

However manually connecting to the webserver1 and webserver2 works just fine.

All I want is just a simple load balancing for the webserver that sit behind the HAproxy. Any advise or suggestion is absolutely appreciated. Please help? Many thanks.


Solution 1:

I've never used HAproxy but a quick search leads me to think you need to add default_backend app immediately below frontend main *:80. I see nowhere in that configuration connecting the backend and frontend together.

Solution 2:

Problem is in your HAProxy configuration. When I remove all comments from your config, I will get this:

global
  log         127.0.0.1 local2

  chroot      /var/lib/haproxy
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  user        haproxy
  group       haproxy
  daemon

  stats socket /var/lib/haproxy/stats

defaults
  mode                    http
  log                     global
  option                  httplog
  option                  dontlognull
  option http-server-close
  option forwardfor       except 127.0.0.0/8
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000

frontend  main *:80

backend app
  mode tcp
  balance roundrobin
  server  server1 192.168.1.12:80 check inter 2000 rise 2 fall 5
  server  server2 192.168.1.13:80 check inter 2000 rise 2 fall 5

And now you can clearly see that there is no configuration of frontend at all. Requests comes to HAProxy via frontend main but HAProxy doesn't know which servers are reliable to handle it, so will return 503.

You have to link backend to frontend with default_backend or with acl.

You should use stats too, not only with socket but with protected web interface too. I can shows you information about clusters behind haproxy, which servers are offline, which has any problems, about response times and so on. Very usefull for debugging.