nginx proxy retry while backend is restarting
Nginx returns 502 while the backend is restarting. How can I make nginx proxy retry N times with M seconds delay? Is there a plugin or something else?
Solution 1:
I found that Nginx took ~2 seconds to exhaust attempts if you specified hundreds of instances of the same backend:
server localhost:8080 max_fails=0;
server localhost:8080 max_fails=0;
server localhost:8080 max_fails=0;
server localhost:8080 max_fails=0;
(.... er, repeat as required!)
Yes, a horrendous kludge - but it does add a degree of tenacity...
Even worse than that, you could use:
server localhost:8080 max_fails=0;
server localhost:80 backup;
Assuming Nginx is running on port 80, this would try to continually loop the request around Nginx until localhost:8080 responds. I.e. retry an infinite(?) number of times with zero seconds delay.
I'll get back to my padded cell now....
Solution 2:
If your nginx installation has Lua support then you can hold on the client for some time with sleep method. The operation is non-blocking and doesn't lock the worker. Keep in mind that user can't be held infinitely as some other network related socket/firewall timeouts may occur finally.
server {
listen 8502;
location / {
#25 seconds sleep
content_by_lua_block {
ngx.sleep(25);
ngx.exit(ngx.HTTP_BAD_GATEWAY);
}
}
}
Then in your upstream list you need to add above server as backup to hold a client.
upstream backend {
server 127.0.0.1:3001 fail_timeout=2s; #The backend
server 127.0.0.1:8502 backup; #Lua holding server in the event backend is restarting
}
And this should be included in your proxied location specification:
proxy_read_timeout 30; #Value must be higher than sleep in Lua
proxy_next_upstream error timeout http_502 http_504;
Solution 3:
Closest thing I know is this project, but I never used. I know that varnish has backend checking, maybe you want to take a look on it.