Amazon EC2 Elastic Load Balancing - strategy for zero downtime server restart
I run apache/mod_perl as load-balanced EC2 instances, and do code upgrades regularly just as you say. My process is:
- take an instance or two out of rotation
- shut down apache on that instance
- upgrade the instance(s)
- return to rotation, and remove the others
The AWS documentation goes over how to add and remove instance from rotation using either the API or the Console, your choice. You'll notice that with my approach, webservers go out of rotation gracefully, so I'm not worrying about whether a particular user request gets killed. As @cyberx86 mentioned, you can use the command apachectl -k graceful
to shut down your apache server after each request is processed.
I have another way to achieve this, If you have have several things to upgrade on each server. That'll be very wasting to update 5 server for the same operations.
Create a Load Balancing & Scaling Group.Ensure the connection draining is enabled.
There's my upgrade step.
- Remove 1 instance from your load balancer. Update everything you need. And create an AMI from you production instance. This will cause the restart of your instance.
- Configure the Scaling group with new AMI.
- Remove instance from load balancer and the stop the instance. The new instance will start automatically
- Continue to shutdown other instance and after all instance re-created. The upgrade is completed.
Some guides:
Load Balance Your Auto Scaling Group
Connection Draining
If your deployment can be modified to support newly created instances coming online with new code - you can actually remove the instance from an ELB, wait 60 seconds (after which Amazon's ELB would close the connection to backend and client anyway) and then terminate the instance - and rely on Amazon's Auto Scaling Group to bring a new instance up for you. I've created an open-source tool that automates this process - available at awsmissingtools.com - look for the tool called "AWS-HA-Release."