Proxy Error 502 "Reason: Error reading from remote server" with Apache 2.2.3 (Debian) mod_proxy and Jetty 6.1.18

Apache is receiving requests at port :80 and proxying them to Jetty at port :8080

The proxy server received an invalid response from an upstream server
The proxy server could not handle the request GET /.

My dilemma: Everything works fine normally (fast requests, few seconds or few tens of seconds long requests are processed ok). Problems occur when request processing takes long (few minutes?).

If I issue request instead directly to Jetty at port :8080 the request is processed OK. So problem is likely to sit somewhere between Apache and Jetty where I am using mod_proxy. How to solve this?

I have already tried some "tricks" related to KeepAlive settings, without luck. Here is my current configuration, any suggestions?

#keepalive Off                     ## I have tried this, does not help
#SetEnv force-proxy-request-1.0 1  ## I have tried this, does not help
#SetEnv proxy-nokeepalive 1        ## I have tried this, does not help
#SetEnv proxy-initial-not-pooled 1 ## I have tried this, does not help
KeepAlive 20                       ## I have tried this, does not help
KeepAliveTimeout 600               ## I have tried this, does not help
ProxyTimeout 600                   ## I have tried this, does not help

NameVirtualHost *:80
<VirtualHost _default_:80>
    ServerAdmin [email protected]

    ServerName www.mydomain.fi

    ServerAlias mydomain.fi mydomain.com mydomain www.mydomain.com

    ProxyRequests On
    ProxyVia On
    <Proxy *>
            Order deny,allow
            Allow from all
    </Proxy>

    ProxyRequests Off
    ProxyPass / http://www.mydomain.fi:8080/ retry=1 acquire=3000 timeout=600
    ProxyPassReverse / http://www.mydomain.fi:8080/

    RewriteEngine On
    RewriteCond %{SERVER_NAME} !^www\.mydomain\.fi
    RewriteRule /(.*) http://www.mydomain.fi/$1 [redirect=301L]

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog /var/log/apache2/access.log combined
    ServerSignature On

</VirtualHost>

Here is also the debug log from a failing request:

74.125.43.99 - - [29/Sep/2010:20:15:40 +0300] "GET /?wicket:bookmarkablePage=newWindow:com.mydomain.view.application.reports.SaveReportPage HTTP/1.1" 502 355 "https://www.mydomain.fi/?wicket:interface=:0:2:::" "Mozilla/5.0 (Windows; U; Windows NT 6.1; fi; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10"
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: error reading status line from remote server www.mydomain.fi, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: Error reading from remote server returned by /, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::

Solution 1:

I have solved the problem. The Keepalive=On should be inserted into ProxyPass config line:

ProxyPass / http://www.dom.fi:8080/ retry=1 acquire=3000 timeout=600 Keepalive=On

See that

Keepalive=On

there? It is critical ;)

Solution 2:

This error can also occur if you don't end your proxy url with a /. Either both paths should end with a / or neither.

Solution 3:

Have you tried setting setenv proxy-initial-not-pooled 1?

Reference here

Solution 4:

Looking at the log, there's something that times out at 5 minutes (=300 seconds). That's a pretty long time to wait for a response. When you access the Jetty server directly, does this resource really take that long to produce a response?

If the five minutes really is within possible response times, you might try tweaking the ProxyTimeout configuration directive.

Depending on your network set-up, it might well be that there's no reason to even try to use any keepalive system (is there a firewall between the app server and proxy which might be configured to drop sessions that are idle for too long?), but the ProxyTimeout would affect the behaviour of the proxy itself.

If the same proxy also serves other backends, it would be better to keep the current ProxyTimeout, and configure the timeout in the ProxyPass directive (see mod_proxy documentation).

If, however, the responses without the proxy are consistently something much less than the five minutes see here as the cut-off limit, then there might really be some odd interference between the proxy and app server, but you're not providing anything of value for identifying what it might be.