Nginx - Unable to upload files to /tmp

I'm unable to upload files to /tmp using Nginx, php-fpm and Ubuntu 18.04. When the application tries to upload to /tmp/upload, it looks like nothing actually happens and no attempt to upload to that location has happened.

I've checked for permission issues and anything in the logs but there was nothing. I've tried setting upload_store value to a folder outside of /tmp (/var/www/upload) and repeated the upload process. This worked as expected and upload has succeeded. I then tried to see if I wasn't getting permission errors in the logs by setting that folder to unreadable but when I've tried to upload to it I've got a regular Permission denied error.

I've tried setting up a watcher on the folder:

inotifywait -m --format "%e %f" /tmp/upload

Which reported something like this when manually touching a file but nothing during upload:

CREATE test_file
OPEN test_file
ATTRIB test_file
CLOSE_WRITE,CLOSE test_file

And this when I tried uploading to /var/www/upload:

CREATE 0000000173
OPEN 0000000173
MODIFY 0000000173
CLOSE_WRITE,CLOSE 0000000173
OPEN 0000000173
ACCESS 0000000173
CLOSE_NOWRITE,CLOSE 0000000173
OPEN 0000000173
ACCESS 0000000173
CLOSE_NOWRITE,CLOSE 0000000173
DELETE 0000000173

However, when watching /tmp/upload, I've got nothing. I've also tried uploading directly to /tmp with the same results. /tmp and /tmp/upload both have 777 permissions.

Here's my nginx config for that location:

location = /setup/images/saveimages {
    upload_pass @accept;
    upload_store /tmp/upload;
    upload_set_form_field "$upload_field_name.name"         "$upload_file_name";
    upload_set_form_field "$upload_field_name.content_type" "$upload_content_type";
    upload_set_form_field "$upload_field_name.path"         "$upload_tmp_path";
    upload_pass_form_field "^photo[0-9]_label$";
}

Weird thing is, this works flawlessly on local development and staging but not on production, so seems like a configuration issue. However, I'm unable to find any differences in configuration.

Any ideas?

After digging deeper, it turned out that the issue was with systemd and PrivateTmp variable which was set to true. What this does is it makes sure that the /tmp directory for application in different namespaces are not shared. See http://manpages.ubuntu.com/manpages/bionic/man5/systemd.exec.5.html for more detail.

This means that the nginx uploads to /tmp php-fpm reads from /tmp are essentially looking at different directories.

The easiest solution is to turn off that feature within nginx systemd configuration was was located /lib/systemd/system/nginx.service in my case. You can do that by setting variable PrivateTmp=false.

However, PrivateTmp=true is the default as a security feature. So if you don't want to completely turn if off, you can make php and nginx share the same namespace through JoinsNamespaceOf variable.

I ended up modifying both php7.2-fpm and nginx services to achieve this.

nginx.service
[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target
JoinsNamespaceOf=php7.2-fpm.service
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target

php7.2-fpm.service
[Unit]
Description=The PHP 7.2 FastCGI Process Manager
Documentation=man:php-fpm7.2(8)
After=network.target
[Service] 
Type=notify
PIDFile=/run/php/php7.2-fpm.pid
ExecStart=/usr/sbin/php-fpm7.2 --nodaemonize --fpm-config /etc/php/7.2/fpm/php-fpm.conf
ExecReload=/bin/kill -USR2 $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target