Scaling nginx beyond 45k connections

I have a situation that means my nginx server has many long running open connections, however after a certain amount of connections it starts throwing an error 'Cannot assign requested address for upstream'.

I've already gone ahead and increased the underlying system limits to the maximum (increasing open file limits, usable port ranges etc), and in terms of CPU/RAM the system is performing absolutely fine, it seems like the bottleneck is just the number of open connections/ports.

Is there really no way to have a single nginx server handle say 100k, or 200k concurrent open connections? I know more machines can be added to the equation, but then that disrupts things such as nginx rate limiting.

Solution 1:

The error message you quoted indicates that the problem is not with the number of incoming connections, but the number of outgoing connections, from nginx to your backend web application.

You will have done something like this somewhere in your configuration:

proxy_pass http://127.0.0.1:8000;

Remember that a TCP connection is uniquely identified by source address and port and destination address and port. In this case it appears you have run out of source port numbers to make unique connections.

The obvious immediate fix is to switch to UNIX domain sockets. Your application must also listen on a UNIX domain socket. You might also need to arrange to have nginx's number of open files limit raised (e.g. raise LimitNOFILE= in its systemd unit) as well as that of your web app. The number of connections you can make this way is essentially unlimited.

proxy_pass unix:/run/myapp/myapp.sock;

A less obvious fix, if you can't make your app use UNIX domain sockets, is to have your web app listen on more than one address/port pair, and to have nginx rotate between them. This will double the number of connections you can make. If your web app already listens for both IPv4 and IPv6 connections, then a very quick and cheap fix is to have nginx alternate between them. You could also modify your app to listen on multiple ports.

upstream myapp {
   server 127.0.0.1:8000;
   server [::1]:8000;
}

or

upstream myapp {
    server 127.0.0.1:8000;
    server 127.0.0.1:8001;
    server 127.0.0.1:8002;
}

Then

proxy_pass http://myapp;