Nginx as a load balancer. Frequent upstream timed out (110: Connection timed out) while connecting to upstream

Solution 1:

I ran into something like this in a nginx<->apache2 setup. It was apache taking too long under load due to MySQL bogging down. To find out how long apache was taking I changed the log format to:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %DµSEC" timed

and the nginx log to:

log_format timed_combined '$remote_addr - $remote_user [$time_local]  '

Then it became easier to see that although apache was completing all the requests, it was very late (by many seconds)in delivering data back to nginx.

I'm not sure why haproxy has helped your situation unless one apache server is much slower than the others. That can happen with identical machines when one machine is having recoverable disk errors. The errors should show up in the syslogs.