Why am I getting an Apache Proxy 503 error?

Apache will respond with 503's for at least 60 seconds any time it detects that the backend server is down. This is the default behavior. As in your example, if you restart your backend server (Rails in this example) and someone tries to access it through the Apache proxy before Rails is ready then Apache will return 503's for the next 60 seconds regardless if your backend is now 'up'. Please see the apache docs on ProxyPass where it states:

retry 60

Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance, and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.

So if you set your Proxy Pass to include retry=0 you won't see the 503's when you restart your backend service. This is also useful when using Apache as reverse proxy during development! For example:

ProxyPass / http://localhost:8000 retry=0

Run following command

# /usr/sbin/setsebool httpd_can_network_connect 1

OR

# /usr/sbin/setsebool httpd_can_network_connect true

and after that restart httpd

# service httpd restart

Are you sure they're restarting in the correct order? I've had weird issues where Apache starts, then Mongrel starts and although Mongrel is running, Apache still throws the proxy error.

I've solved this in the past with various incantations and restarts of Apache and eventually the gods are happy. It seems that sometimes the Mongrel processes don't properly shut down so you have to manually kill them. Here's a link with some [possible] help.

I ended up adding a "kill" option to my /etc/init.d/ mongrel script because it happened so much. It stop Mongrel, killed all Mongrel sessions, started Mongrel and restarted Apache.

<snip>
    kill)
      echo "Stopping, killing, starting, and restarting Apache..."
      mongrel_cluster_ctl stop -c $CONF_DIR --clean
      killall -u mongrel
      mongrel_cluster_ctl start -c $CONF_DIR --clean
      /etc/init.d/httpd restart
      RETVAL=$?
  ;;
</snip>

Probably not a very good solution but the evil went away.

Try running monit to monitor your mongrels behind Apache, and that way it can restart mongrels for you if they die or get too hungry for memory. If for any reason Apache still gets confused you may just have to gracefully restart apache and it should resolve itself, but for 99% of cases having monit watch over your mongrels should avoid this happening again. The other option is look into Phusion Passenger.