worker_connections are not enough while have idle workers
Arriving late here but for Googlers, try increasing the worker processes limit (or use automatic):
worker_processes auto;
As per the Nginx documentation, they directly affect each other:
It should be kept in mind that this number includes all connections (e.g. connections with proxied servers, among others), not only connections with clients. Another consideration is that the actual number of simultaneous connections cannot exceed the current limit on the maximum number of open files, which can be changed by worker_rlimit_nofile.