How to make redundant load balancers?

I understand that the purpose of load balancers is to balance load between your servers and keep track of instance health, etc. But what if load balancer itself fails? How do you set up redundant load balancers? (load balancing load balancers?)

I could see how DNS health checks could be useful, but there's obviously major latency issues, isn't there?

This is assuming that you're not using any third party services like AWS ELB or anything similar. What to do if you're just using say Nginx?

There are couple of ways to achieve HA (high availability) of a Load Balancer - or in that regards any service. Lets assume you have two machines, with IP addresses:

• 192.168.100.101
• 192.168.100.102

Users connect to an IP, so what you want to do is separate IP from specific box - eg create virtual IP. That IP will be 192.168.100.100.

Now, you can choose HA service which will take care of automatic failover/failback of IP address. Some of the simplest services for unix are (u)carp and keepalived, some of the more complex ones are for example RedHat Cluster Suite or Pacemaker.

Lets take keepalived as an example - two keepalived services - each running on its own box - and they communicate together. That communication is often called heartbeat.

|   VIP   |                           |         |
|  Box A  | ------v^-----------v^---- |  Box B  |
|   IP1   |                           |   IP2   |

If one keepalived stops responding (either service goes down for whatever reason, or the box bounces or shuts down) - keepalived on other box will notice missed heartbeats, and will presume other node is dead, and take failover actions. That action in our case will be bringing up the floating IP.

                                      |   VIP   |
    ------------------ -------------- |  Box B  |
                                      |   IP2   |

Worst case that can happen in this case is the loss of sessions for clients, but they will be able to reconnect. If you want to avoid that, two load balancers have to be able to sync session data between them, and if they can do that, users won't notice anything except maybe broken a short delay.

Another pitfall of this setup is split brain - when both boxes are online but the link is severed, and both boxes bring up the same IP. This is often resolved through some kind of fencing mechanism (SCSI reservation, IPMI restart, smart PDU power cut, ...), or odd number of nodes requiring majority of cluster members to be alive for service to be started.

|   VIP   |                           |   VIP   |
|  Box A  |                           |  Box B  |
|   IP1   |                           |   IP2   |

More complex cluster management software (like Pacemaker) can move whole service (eg.: stop it on one node and start it on another) - and this is the way HA for services like databases can be achieved.

Another possible way - if you are controlling routers near your load balancers, is to utilize ECMP. This approach also enables you to horizontally scale load balancers. This works by each of your two boxes talking BGP to your router(s). Each box has to advertise virtual IP (192.168.100.100) and the the router will load balance traffic via ECMP. If a machine dies, it will stop advertising VIP, which will in turn stop routers from sending traffic to it. Only thing you have to take care of in this setup is to stop advertising IP if the load balancer itself dies.

Using Nginx as your load balancer should allow you to follow the redirect detailed in this post by altering your config to detect a no response timeout:

nginx automatic failover load balancing

In theory if you have a HA environment, multiple load balancers clustered should allow service to be maintained if one was to fail.

Hope this helps.

Hardware load balancers have supported "active/passive" or "active/active" setups for years, in both cases they are then set up in parallel from a layer 1/2 perspective... active/passive uses monitoring/keepalive mechanisms as described, active/active can be implemented in numerous ways. To appear as a single IP at the frontend, two or more balancers might, as long as they are all/both on line, do things like:

• selectively answer ARP requests to the shared IP based on a has of the source MAC or IP address when clients are on the same network
• negotiate between each other who handles the traffic of a given new TCP connection
• let duplicated or erroneous layer 3-7 traffic happen recklessly and rely on client/router TCP stacks to sort it out

And then change their mode to accepting all or more traffic when communication with the/a partner device is lost.

at the backend side:

• each of the balancers might, in normal operation, only use a given sub-pool of application servers
• or, duplicated requests might simply be generated here too...
• or, negotiation between balancers might be done