Long running requests with gunicorn + nginx

I've put together an integration server for our Django-powered application. A few of the features are still experimental, and result in overly long requests.

I'm okay with the poor performance, for now, but I need to be able to integrate. Whenever we use the feature that leads to a long request, the app hangs (as expected) and then, after maybe a minute and a half, returns a '502 - Bad Gateway'. The rest of the app works fine.

I checked the gunicorn log, and whenever this happens I get a line like

2012-01-20 17:30:13 [23128] [DEBUG] GET /results/
2012-01-20 17:30:43 [23125] [ERROR] WORKER TIMEOUT (pid:23128)
Traceback (most recent call last):
  File "/home/demo/python_envs/frontend/lib/python2.6/site-packages/gunicorn/app/base.py", line 111, in run
    os.setpgrp()
OSError: [Errno 1] Operation not permitted

however, this happens long before the actual worker timeout, which I've set to 10 minutes just to make sure. Here's part of the upstart script that runs gunicorn.

description "..."

start on runlevel [2345]
stop on runlevel [!2345]
#Send KILL after 5 seconds
kill timeout 5
respawn

env VENV="/path/to/a/virtual/env/"

#how to know the pid
pid file $VENV/run/guniconr-8080.pid

script
exec sudo -u demo $VENV/bin/gunicorn_django --preload --daemon -w 4 -t 600 --log-level debug --log-file $VENV/run/gunicorn-8080.log -p $VENV/run/gunicorn-8080.pid -b localhost:8080 /path/to/settings.py
end script

I'm running gunicorn version 0.13.4. Any help would be greatly appreciated.

Solution 1:

502 Bad Gateway means your gunicorn worker is timeout. You can add --timeout option to the gnuicorn command. The default is 30s.

As @greg-k notes, proxy_read_timeout option of Nginx controls yet another timeout. If you see 504 Gateway Timeout error, you can adjust this option.

Solution 2:

This is maybe an issue with your nginx configuration. Did you configure the proxy_read_timeout value?

This directive sets the read timeout for the response of the proxied server. It determines how long nginx will wait to get the response to a request. The timeout is established not for entire response, but only between two operations of reading.

Source: http://wiki.nginx.org/HttpProxyModule#proxy_read_timeout