Restarting nginx backends without losing requests
I'm sure it's been asked before in different words but I run several Django sites via uwsgi (emporer mode) behind nginx. It's all a fairly standard configuration but I find that if I restart the central uwsgi process, nginx just bombs out 502s rather than waiting for the socket to become available.
I recognise that most of this is probably for a reason but people seeing 502 errors really stings me. It's certainly not something I want a client to see. So...
- Can I beg nginx to wait/retry backends? Or,
- Is there anything (other than the obvious) I can do to minimise commercial damage from uwsgi restarts?
One idea is to replace nginx's default 502 template with a page that automatically refreshes the client. You essentially just need to make a new file that has <meta http-equiv="refresh" content="5">
in the header. Give it some friendly text, explaining that the site is currently undergoing maintenance (or some equiv BS) and link to it from your nginx config:
error_page 502 /502.html;
location = /502.html {
root /var/www/502.html;
}
You'll need that in all your sites (there may be a way of doing that globally) but the result is anybody who would see a gateway timeout will now see a page that doesn't look particularly odd and will, within five seconds, land them on the page they originally wanted.
That all assumes the backend will come back up. If there's a chance it will be off indefinitely, you might want to write something in JS that checks the URL itself and have a retry-counter. All fairly simple but it may appease clients who are getting annoyed with a site being down.