Why is my web server dropping connections with a TCP reset at high load?

Solution 1:

To set max number of open files ( if that is causing your issue ) you need to add "fs.file-max = 64000" to /etc/sysctl.conf

Solution 2:

There could be any number of sources of the connection resets. The load tester could be out of available ephemeral ports from which to initiate a connection, a device along the way (such as a firewall doing NAT) might have its NAT pool exhausted and is unable to provide a source port for the connection, is there a load balancer or firewall at your end that might have reached a connection limit? And if doing source NAT on the inbound traffic, that could also experience port exhaustion.

One would really need a pcap file from both ends. What you want to look for is if a connection attempt is sent but never reaches the server but still appears as if it was reset by the server. If that is the case then something along the line had to reset the connection. NAT pool exhaustion is a common source of these kinds of problems.

Also, netstat -st might give you some additional information.

Solution 3:

Some ideas to try, based on my own recent similar tuning experiences. With references:

You say it's a static text file. Just in case there's any upstream processing going on, apparently domain sockets improve TCP throughput over a TC port based connection:

https://rtcamp.com/tutorials/php/fpm-sysctl-tweaking/ https://engineering.gosquared.com/optimising-nginx-node-js-and-networking-for-heavy-workloads

Regardless of upstream termination:

Enable multi_accept and tcp_nodelay: http://tweaked.io/guide/nginx/

Disable TCP Slow Start: https://stackoverflow.com/questions/17015611/disable-tcp-slow-start http://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum-performance/

Optimize TCP Congestion Window (initcwnd): http://www.nateware.com/linux-network-tuning-for-2013.html