nginx redirect or return single location
This happens because you put return 404; in the server block.
Rewrites at the server level are processed before rewrites at the location level.
To fix the problem, put it in a location. (As a general rule you should always process requests in a location.)
location / {
return 404;
}