nginx 500 (24: too many open files)

[I posted this on the Nginx forums, but had no responses a week later, so trying here]

I am a Linux and Nginx novice, but have learnt enough to get it installed and running and working as a simple reverse proxy for two internal webservers. This has been running fine for months, but I recently started getting 500 errors.

Here is the recent output of /var/log/nginx/error.log (I have replaced our company name with "companyname.com" and replaced our public WAN IP address with

2020/02/10 15:17:49 [alert] 1069#1069: *1011 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web1.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web1.companyname.com"

2020/02/10 15:21:41 [alert] 1069#1069: *2022 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web2.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web2.companyname.com"

2020/02/10 15:33:28 [alert] 1084#1084: *19987 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web2.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web2.companyname.com"

2020/02/10 15:34:16 [alert] 1084#1084: *39974 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web1.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web1.companyname.com"

2020/02/10 15:50:30 [error] 1086#1086: *1 client intended to send too large body: 4294967295 bytes, client: 176.58.124.134, server: london.companyname.com, request: "GET /msdn.cpp HTTP/1.1", host: "<WANIP>"

2020/02/10 16:32:56 [alert] 1086#1086: *19989 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web1.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web1.companyname.com"

I have added the following to the end of /etc/security/limits.conf

nginx soft nofile 10000
nginx hard nofile 30000

I have added the following to /etc/sysctl.conf

fs.file-max=70000

...And rebooted. However, I'm getting the same problem immediately after a reboot.

Interestingly the IP address that appears in the log "176.58.124.134" I don't recognise and a quick google search suggests this is an abusive IP address. I can block at the firewall, but I'm not sure that's the problem.

Any tips, suggestions are grealy appreciated. Thanks.

Your nginx server is not keeping up with the number of requests arriving. More precisely it lacks available file descriptors to open connections to the upstream.

These are regulated by three parameters:

The per-system limit in /proc/sys/fs/file-max, which limits the maximum number of fds for the whole system. Don't change it, the default is high enough (800000 on my small server).
The per-process hard limit (RLIMIT_NOFILE), which can be only set by root (or a process having CAP_SYS_RESOURCES). This is usually quite high (ulimit -Hn, around 1000000), so no need to increase it. If you want to increase it the pam_limit.so configuration /etc/security/limits.conf will not help you, as nginx is started by systemd (I have to guess, since you don't mention your distribution), which does not use PAM. You'll need to edit the nginx.service file instead:
```
systemctl edit --full nginx.service
```
and add the following line to the [Service] section:
```
LimitNOFILE=your_limit
```
The per-process soft limit. nginx can increase it itself with the directive mentioned by Romeo:
```
worker_rlimit_nofile = your_limit;
```

Each limit can not be higher, than the ones in the previous points.

However, unless your website became extremely popular overnight, it is much more probable that it is experiencing a DDOS attack. You can mitigate it by limiting the number of connections per client using the http_limit_conn module. The example configuration in the documentation should directly apply to your case:

http {
    limit_conn_zone $binary_remote_addr zone=addr:10m;
    limit_conn addr 10;
    ...

which will limit the number of connections per IP address to 10. On most distributions you can put the two limit_* directives in a separate file (e.g. /etc/nginx/conf.d/limit.conf) without modifying the main nginx.conf.

nginx 500 (24: too many open files)

Related

Recent Posts