Plain uWSGI error: uwsgi_response_write_body_do() TIMEOUT

This looks like a well-known issue with Nginx+uWSGI timeouts, but my setup is different and I couldn't find information about it.

I run a Flask app on uWSGI, on http protocol (not sockets). When I query my API, sometimes I receive an error:

uwsgi_response_write_body_do() TIMEOUT !!!
IOError: : write error

I receive it more often during "business hour", i.e. not so often at night. I'm querying for long JSON data (geo coordinates, up to 50 MBytes) and sometimes I get no response at all, and sometimes I get part of the response. These are errors from Python's requests request:

requests.exceptions.ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))

requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

ValueError: Expecting object: line 1 column 5590371 (char 5590370)

ValueError: Unterminated string starting at: line 1 column 20998536 (char 20998535)

ValueError: end is out of bounds

# when I set `timeout=None` in Python
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='mydomain.com', port=80): Read timed out. (read timeout=None)

The connection is over plain HTTP (no S). My uWSGI config below:

[uwsgi]
module = run:app
pidfile = /tmp/app.pid
logto = /var/log/uwsgi/%n.log

master = true
processes = 2

http-socket = 0.0.0.0:someport
http-timeout = 300

die-on-term = true

And I run this using uwsgi /path/to/inifile. There is no Nginx, but actually before, with Nginx, I was receiving same problems. So I ruled out this one.

How can I avoid these timeouts?

I also use nginx + uwsgi to produce a large file, and meet the same problem. I solved this problem by disable uwsgi temp-file buffer in nginx use directive uwsgi_max_temp_file_size 0;.

I think the transmission speed from nginx to client is far slower to the speed from uwsgi to nginx. And nginx's buffer is full, so uwsgi wait much time before there are space for nginx to receive new data and this make uwsgi's write request timeout.

I post a blog to analysis this problem in my blog.

There are only two places in the uwsgi source code where that error string occurs. The surrounding code looks similar for both, so let’s focus on one of them:

            ret = uwsgi_wait_write_req(wsgi_req);
            if (ret < 0) { wsgi_req->write_errors++; return -1;}
            if (ret == 0) {
        // here we use the parent name
                    uwsgi_log("uwsgi_response_write_body_do() TIMEOUT !!!\n");
                    wsgi_req->write_errors++;
                    return -1;
            }

It looks like if the return value of uwsgi_wait_write_req is zero, that’ll trigger the TIMEOUT !!! error. uwsgi_wait_write_req is defined in uwsgi.h as:

#define uwsgi_wait_write_req(x) uwsgi.wait_write_hook( \
    x->fd, uwsgi.socket_timeout) ; x->switches++

Without digging deeper, I cannot be 100% sure that the socket_timeout here is the same as the uwsgi --socket-timeout option.

However, I can say that adding --socket-timeout 1200 to my uwsgi invocation did get fix a quite reliable way to reproduce the TIMEOUT !!! error when downloading a fairly large file from a Python→uwsgi→nginx setup over a slow connection.

So I think increasing --socket-timeout is the best option, though I can’t be sure if some other buffering settings in the nginx config might also be required.

Plain uWSGI error: uwsgi_response_write_body_do() TIMEOUT

Related

Recent Posts