Drain connections on restart of NGINX process?

I'm using NGINX 1.8.1 on CentOS 6 and 7. During our deploys, it is required that NGINX is stopped, the deploy executes, and NGINX starts again. Unfortunately, NGINX during a stop seems to just 500 all connections. As I'm using Ansible to update 2/10 hosts at a time, it's likely therefore that many of our clients will get 500'd throughout our deploy process.

Is there a way via NGINX settings to have the NGINX process drain all live connections (ie wait for them to complete) while rejecting all new connections during a stop?

I know that Amazon's Elastic Load Balancers do have this functionality, is there an equivalent for NGINX?

Solution 1:

According to the comments, you could try to solve this using a firewall:

• Task: Block incoming new connections, but keep related and established ones

• Try: iptables -I INPUT -j DROP -p tcp --syn --destination-port 80 (or 443, whatever applies to your setup) to set the rule, thus blocking new traffic

• Do your deployment...

• Try: iptables -D INPUT -j DROP -p tcp --syn --destination-port 80 (or 443, whatever applies to your setup) to revert the rule, thus allowing new traffic

• Note: I've just tested this shortly, it worked as expected. But: Please don't take this for granted, and test for yourself as well. Feedback welcome...

Edit: As @Guntram Blohm pointed out in the comments, better to use iptables -I instead of iptables -A. Changed this appropriately in the above code.

Solution 2:

Send nginx the "QUIT" signal to perform a graceful shutdown, which stops listening for new connections but allows workers to continue to serve active connections.

It's worth distinguishing between graceful shutdown and connection draining; graceful shutdown affects how NGINX deals with incoming connections while the server is shutting down, but connection draining (which as Martijn Heemels mentioned is a feature of NGINX Plus) deals with how NGINX removes a backend server from service when acting as a load balancer. From your question, it sounds like you're more interested in the former.

Solution 3:

As far as I can tell, connection draining is only available in the commercial version, Nginx Plus.

There's a description of the feature here: https://www.nginx.com/blog/nginx-plus-r5-released/

It's part of the ngx_http_upstream_conf_module module.