Docker Nginx stopped: [emerg] 1#1: host not found in upstream

I am running docker-nginx on ECS server. My nginx service is suddenly stopped because the proxy_pass of one of the servers got unreachable. The error is as follows:

[emerg] 1#1: host not found in upstream "dev-example.io" in /etc/nginx/conf.d/default.conf:988

My config file is as below:

 server {
       listen      80;
       server_name     test.com;
       location / {
          proxy_pass         http://dev-exapmle.io:5016/;
          proxy_redirect     off;

          ##proxy_set_header   Host             $host;
          proxy_set_header   X-Real-IP        $remote_addr;
          proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

          client_max_body_size       10m;
          client_body_buffer_size    128k;

          proxy_connect_timeout      90;
          proxy_send_timeout         90;
          proxy_read_timeout         90;

          proxy_buffer_size          4k;
          proxy_buffers              4 32k;
          proxy_busy_buffers_size    64k;
          proxy_temp_file_write_size 64k;
       }
}

server {
   listen       80 default_server;
   server_name  localhost;

   #charset koi8-r;
   #access_log  /var/log/nginx/log/host.access.log  main;

   location / {
      root   /usr/share/nginx/html;
      index  index.html index.htm;
   }

   #error_page  404              /404.html;

   # redirect server error pages to the static page /50x.html
   #
   error_page   500 502 503 504  /50x.html;
   location = /50x.html {
      root   /usr/share/nginx/html;
   }
}

I have many servers in the config file, even if one server was down, I need to have running nginx. Is there any way to fix it?

Any suggestion to fix this issue would be appreciated.

Solution 1:

Just adding a resolver did not resolve the issue in my case. But I was able to work around it by using a variable for the host. Also, I guess it makes more sense to use Docker's DNS at 127.0.0.11 (this is a fixed IP).

Example:

server {
  listen 80;
  server_name test.com;
  
  location / {
    resolver 127.0.0.11;
    set $example dev-example.io:5016;
    proxy_pass http://$example;
  }
}

I found the variable workaround on this page.

Solution 2:

Include to prevent Nginx from crashing if your site is down, include a resolver directive, as follows:

 server {
       listen                80;
       server_name           test.com;
       location / {
          resolver           8.8.8.8;
          proxy_pass         http://dev-exapmle.io:5016/;
          proxy_redirect     off;
 ...

WARNING! Using a public DNS create a security risk in your backend since your DNS requests can be spoofed. If this is an issue, you should point the resolver to a secure DNS server.