Ubuntu 12.04 open port 80 inside VLAN
This is not really a solution, but comments are gettting too big.
Now you might want to know if the problem comes from os/network or from nginx.
One way to achieve this is :
Shut down nginx and check there is no remaining process (I guess your server is not already in production).
Start netcat on port 80. Can be
nc -l -p 80
but depends which netcat you useFrom another box type
telnet 192.168.0.11 80
and start typing ramdom stuff. The random stuff should appear in the terminal where you started netcat. If not, try capturing traffic with tcpdump again.
If you can't reproduce your RST, ACK problem with netcat, nginx is likely to be the problem.