Nginx Retry of Requests ( Nginx - Haproxy Combination ) [duplicate]

I wanted to ask about Nginx Retry of Requests. I have a Nginx running at the backend which then sends the requests to HaProxy which then passes it on the web server and the request is processed. I am reloading my Haproxy config dynamically to provide elasticity. The problem is that the requests are dropped when I reload Haproxy. So I wanted to have a solution where I can just retry that from Nginx. I looked through the proxy_connect_timeout, proxy_next_upstream in http module and max_fails and fail_timeout in server module. I initially only had 1 server in the upstream connections so I just that up twice now and less requests are getting dropped ( only when ) have say the same server twice in upstream , if I have same server 3-4 times drops increase ).

So , firstly I wanted to now , that when a request is not able to establish connection from Nginx to Haproxy so while reloading it seems that conneciton is seen as error and straightway the request is dropped .

So how can I either specify the time after the failure I want to retry the request from Nginx to upstream or the time before which Nginx treats it as failed request.

( I have tried increaing proxy_connect_timeout - didn't help , mail_retires , fail_timeout and also putting the same upstream server twice ( that gave the best results so far )

Nginx Conf File

upstream gae_sleep {

server 128.111.55.219:10000;

}

server {

listen 8080;
server_name 128.111.55.219;
root /var/apps/sleep/app;
# Uncomment these lines to enable logging, and comment out the following two
#access_log  /var/log/nginx/sleep.access.log upstream;
error_log  /var/log/nginx/sleep.error.log;
access_log off;
#error_log /dev/null crit;

rewrite_log off;
error_page 404 = /404.html;
set $cache_dir /var/apps/sleep/cache;



location / {
  proxy_set_header  X-Real-IP  $remote_addr;
  proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_redirect off;
  proxy_pass http://gae_sleep;
  client_max_body_size 2G;
  proxy_connect_timeout 30;
  client_body_timeout 30;
  proxy_read_timeout 30;
}

location /404.html {
  root /var/apps/sleep;
}

location /reserved-channel-appscale-path {
  proxy_buffering off;
  tcp_nodelay on;
  keepalive_timeout 55;
  proxy_pass http://128.111.55.219:5280/http-bind;
}

}

So after trying to find the answer to retrying requests at nginx, I haven't found a clean way of retrying requests but have come up with sort of a hacky way for it. So within the upstream section in the nginx conf , you should put multiple copies of the same upstream server as retrying in nginx is at upstream server level. If one upstream server fails then nginx tries the request at another upstream server. If you have only 1 upstream server like I have it wont retry the request. So to overcome that I put multiple copies of the same upstream server , so that by the time nginx goes through the list of servers and sends requests , the upstream server ( haproxy in this case ) would have reloaded and request will go to. It is also essential to go through the various timeouts that nginx provides at http module and server module. "fail_timeout" - says that if a upstream server is not available decommission it for x secs , but if all of them are not available then it doesn't decommission ( I am mentioning this as by the time nginx goes through the entire list haproxy may not have come up but this wont be a problem because of this property ) PS : this is a hacky solution and i had to have some 100 - 150 entries of upstream in my nginx file for reducing errors to in significant number. Better solutions are welcome :)

Nginx Retry of Requests ( Nginx - Haproxy Combination ) [duplicate]

Related

Recent Posts