Nginx - Allow requests without a Host header

From here: http://nginx.org/en/docs/http/server_names.html

Miscellaneous names

There are some server names that are treated specially.

If it is required to process requests without the “Host” header field in a server block which is not the default, an empty name should be specified:

server {
    listen       80;
    server_name  example.org  www.example.org  "";
    ...
}

If no server_name is defined in a server block then nginx uses the empty name as the server name.

So adding "" to your server_name seems to do what you want.