Unexpected 301 redirects from Nginx when behind Nginx reverse proxy

A solution, and possibly the solution, came after multiple hard hours of trying to understand the problem and exact behavior of Nginx's redirection mechanism. The solution was rather simple really:

Remove the proxy_set_header Host $host from the reverse proxy config:

server {
  ...

  location / {
    proxy_pass http://12.12.12.12:8000/demo/;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Or alternatively, use proper proxy_redirect value:

server {
  ...
  location / {
    proxy_pass http://12.12.12.12:8000/demo/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_redirect http://demo.example.com:8000/demo/ http://demo.example.com/;
  }
}

The reason behind this is that proxy_set_header Host $host delegates the upstream server the original hostname. The nginx at the upstream server applies this host to redirections. Therefore the redirection location is already http://demo.example.com:8000/demo/items/ when dispatched by the upstream server and before the reverse proxy modifies it in any way. Understanding this was the first half of the solution.

The second half deals with how proxy_pass and proxy_redirect affect the redirection. If proxy_redirect is not defined it still has default value default. Its default and rather hidden behavior is to take the redirections from the upstream and replace exact matches of the value of proxy_pass with the original host. In other words, given proxy_pass http://abc, only those redirects where Location header contains http://abc are modified. Finally, if no match is found, the Location header is left unaltered.

Therefore, the original issue was caused because the upstream server already replaced its IP address with the host header provided by the reverse proxy. That caused mismatch between the value of proxy_pass and the redirection location, thus preventing the activation of the default proxy_redirect directive. Furthermore, the unaltered redirect was passed to client and bounced the client to the invalid URL.

I would say this was a rather complex interaction between Host header, proxy_pass, default proxy_redirect, and built-in 301 redirections. Fortunately, it got solved!