Getting "real" IP in Nginx behind AWS network load balancer

I have a network load balancer which is forwarding traffic to an Nginx docker container running in ECS (using awsvpc network mode). My nginx config is as follows:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log info;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    server {

        allow all;
        listen 443 default ssl;
        ssl_certificate /etc/nginx/ssl/cert.pem;
        ssl_certificate_key /etc/nginx/ssl/privkey.pem;
        ssl_trusted_certificate /etc/nginx/ssl/chain.pem;

        ssl_prefer_server_ciphers on;
        ssl_protocols TLSv1.2;

        add_header Strict-Transport-Security "max-age=31536000";
        access_log /var/log/nginx/access.log;

        location / {
            proxy_pass http://blahblahblah;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr, $proxy_add_x_forwarded_for;
        }
    }
}

However when looking at the logs the X-Forwarded-For header doesn't contain the client's "real" IP, it just contains a series of internal IPs (one of which is the network load balancer's internal IP).

My understanding of the Network Load Balancer was that the client's "real" IP was supposed to be preserved as the incoming IP, why is is not showing in this header?

Solution 1:

https://aws.amazon.com/blogs/aws/new-network-load-balancer-effortless-scaling-to-millions-of-requests-per-second/

Source Address Preservation – With Network Load Balancer, the original source IP address and source ports for the incoming connections remain unmodified, so application software need not support X-Forwarded-For, proxy protocol, or other workarounds. This also means that normal firewall rules, including VPC Security Groups, can be used on targets.

Solution 2:

nginx started supporting PROXY protocol v2 since 1.13.11, which is about a month after you asked this question. I encountered this problem today, so in case this is useful for anyone else:

http {
    #...
    server {
        listen 80   proxy_protocol;
        listen 443  ssl proxy_protocol;
        #...
    }
}
   
stream {
    #...
    server {
        listen 12345 proxy_protocol;
        #...
    }
}

Add proxy_protocol to your listener. This will allow your nginx to accept Proxy Protocol V2 connections.

If you're using AWS LB, you need to enable Proxy protocol v2 on your target group:

("Preserve client IP addresses" option doesn't seem to be necessary, $proxy_protocol_addr appears correctly in logs).