How to set up ELB health checks with multiple applications running on each EC2 instance?

Solution 1:

Here are two ways to solve this;

First option is to add another health check on the host that validates the health and returns HTTP 200s to the ELB if the logic says that you want to keep the host online. The logic there is, of course, up to you. The disadvantage here would be that if App 2 deployed successfully on some hosts all hosts would still be 'healthy' and receiving traffic.

Another option is to use an additional ELB for each application. You can point several ELBs to the same backend EC2 instances and the cost is pretty minor to do so. That way you can health check per application and drop hosts with issues at a per-application level rather than an all-or-nothing approach.

Edit: Please note this is an older answer and is specific to ELB not ALB. ALB supports separate targets on one host natively.

Solution 2:

Using one ELB per app is the way to go here.

First, you may need them anyway if each application is on it's own domain and you need to support SSL. Amazon ELBs currently only allow one SSL certificate for each domain, requiring separate ELBs for each SSL-enabled domain. (Wildcard SSL certifications being an exception).

The challenge here is that ELB health checks cannot currently be directed to a particular virtual domain hosted on an EC2 instance. (No "Host:" header is sent). ELB health pings always go to the default domain, as if the you had loaded the IP address for the EC2 instance in your browser. So some glue is required to recieve the health checks on the default domain and then reply with health status of a particular application.

Here is a working example configuration that could be added to an Nginx server directive. It would be installed on each of the EC2 instances being load balanced.

    # This goes in the `server` block noted by 'default_server', often /etc/nginx/sites-enabled/default

    # All AWS Health Checks from the ELBs arrive at the default server.
    # Forward these requests on the appropriate configuration on this host.
    location /health-check/ {
      rewrite ^/health-check/(?<domain>[a-zA-Z0-9\.]+) /api/v1/status break;
      # Lie about incoming protocol, to avoid the backend issuing a 301 redirect from insecure->secure,
      #  which would not be considered successful.
      proxy_set_header X-Forwarded-Proto 'https';
      proxy_set_header "Host" $domain;
      proxy_pass http://127.0.0.1;
    }

In the In the "Health Check" setting of of the ELB for "first-application.com", you would select "HTTP" and Port 80 and enter a path like:

/health-check/first-application.com

With the above Nginx configuration running on the host, the request would be received on the default domain and proxy the response from the Nginx configuration on the same host for https://first-application.com/api/v1/status

With this approach there is no per-app configuration in Nginx. As long as each app has a unique domain name, you just need to make sure you set up an ELB for each app appropriately.

Solution 3:

On 11 August 2016, Amazon introduced Application Load Balancers. These let you specify multiple Target Groups, each with its own type of health check. So this is now possible using a single load balancer!