Rolling deploys with Tomcats behind an HAProxy instance
I have three Tomcat instances running behind an HAProxy. When I roll out changes to my web applications, I'd like to do a rolling deploy (i.e. bounce one Tomcat at a time) so users don't see any downtime.
How do I do this? I see that a running instance of haproxy can be hot-reconfigured (which is good for adding or removing new pool servers), but how does HAProxy react when one of its target servers is temporarily down?
If there's a better solution than HAProxy, I'd be willing to look at that too.
Thanks!
I sent an e-mail to Willy Tarreau and got back the following options:
You can use hot reconfiguration or socat commands to the stats server to set the weight of your target server to 0. This will prevent new sessions from being balanced to that server, but won't affect existing connections.
You can set the http-check disable-on-404 option in combination with "option httpcheck /myurl" and then have your target servers respond to /myurl such that they send a 200 status if everything is fine, a 404 if the server should stop receiving new requests, and a 500 if the server should receive nothing (i.e. when you're ready to bounce the server). haproxy will re-check on the server at the interval specified in your server line.
Being as that HAProxy does not support removal without reconfiguration according to @Ernest Muller in his answer, I will provide you an alternative as you requested other scenarios as well.
I use LVS, which is one of my favorite solutions for load balancing, as it can be used for more than HTTP.
With LVS, you can use ipvsadm to manually add and remove servers. An example of removal is the following command:
/sbin/ipvsadm -e -t VIP:443 -r SERVERIP:443 -g -w WEIGHT
As opposed to manually adding and removing by interacting with LVS, I often prefer to use the request option with ldirectord. ldirector is a daemon that polls and manages your ipvs configuration. You can specify a file with a URI path using that parameter. During rollout, you remove the file and wait for the server to stop being accessed. At that point, you can deploy the code without impacting production clients.