Permission denied while reading upstream

nginx

We have deployed our rails application on on Nginx and passenger. Intermittently, pages of application get loaded partially. There is no error in application log, but the Nginx error log shows the following:

2011/02/14 05:49:34 [crit] 25389#0: *645 open() "/opt/nginx/proxy_temp/2/02/0000000022" 
  failed (13: Permission denied) while reading upstream, client: x.x.x.x, 
  server: y.y.y.y, request: "GET /signup/procedures?count=0 HTTP/1.1", 
  upstream: "passenger:unix:/passenger_helper_server:", host: "y.y.y.y", 
  referrer: "http://y.y.y.y/signup/procedures"

Solution 1:

I had the same problem on an NGINX/PHP-FPM setup (php-fpm=improved fcgi for php).

You can find out which user the nginx processes are running as

ps aux | grep "nginx: worker process"

And then check out if the permissions in your proxy files are correct

ls -l /opt/nginx/proxy_temp/

In my case, nginx was running as www-data and two of the directories in my proxy directory belonged to root.

I don't know how it happened yet, but I fixed it by doing (as root)

chown www-data.www-data /opt/nginx/proxy_temp

Solution 2:

You probably started with user root, then changed it. Now the problem is that the cache folders, i.e.

/var/cache/nginx/client_temp
/var/cache/nginx/fastcgi_temp
/var/cache/nginx/proxy_temp
/var/cache/nginx/scgi_temp
/var/cache/nginx/uwsgi_temp

are already owned by root, so your nginx (www-data or whatever you're trying to switch to) user can't access them because they have a permission of 700.

So the solution is easy. Stop nginx, then:

rm -rf /var/cache/nginx/*

or whatever the path is on your distribution and release. Then restart nginx which will re-create these folders with the appropriate permissions.

Solution 3:

Also check the nginx.conf file to make sure you are specifying the correct user AND group.

I had a problem where the permissions on the directory were setup for username/nginx, but the nginx.conf user only specified username. By default, if no group is given to the user directive, it uses the same name as user. So, username/username was trying to access a directory instead of username/nginx. Updating the config fixed my problems.

See: http://nginx.org/en/docs/ngx_core_module.html#user

Solution 4:

So I did all of the above and unfortunately for me it was giving me the same error. I am running a rails app packaged into a jar file with torquebox on a centos 6.7 machine with nginx. I battled this for about 3 hours until I found another solution and I hope it helps someone else. According to this article nginx may run on enforcing mode. I just simply changed nginx to permissive mode with

setenforce 0

With that, the error was gone and I was able to run my application on a staging/production environment.

I was clueless until I found the error on the audit.log

type=AVC msg=audit(1444454198.438:466): avc:  denied  { name_connect } for  pid=3201 comm="nginx" dest=8080 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket

I really hope this saves someone the 3 hours I just lost.

Solution 5:

When starting nginx from an unprivileged account the use_temp_path=off.

proxy_cache_path ... use_temp_path=off;

This needed to avoid nginx trying to put the files into the default proxy_temp_path. From the nginx docs:

The directory for temporary files is set based on the use_temp_path parameter (1.7.10). If this parameter is omitted or set to the value on, the directory set by the proxy_temp_path directive for the given location will be used. If the value is set to off, temporary files will be put directly in the cache directory.