Apache load balancer with https real servers and client certificates

Our network requirements state that ALL network traffic must be encrypted.

The network configuration looks like this:

                                                              ------------
                                                /-- https --> | server 1 | 
                                               /              ------------
|------------|               |---------------|/               ------------
|   Client   | --- https --> | Load Balancer | ---- https --> | server 2 |
|------------|               |---------------|\               ------------
                                               \              ------------
                                                \-- https --> | server 3 |
                                                              ------------

And it has to pass client certificates.

I've got a config that can do load balancing with in-the-clear real servers:

<VirtualHost *:8666>

    DocumentRoot "/usr/local/apache/ssl_html"
    ServerName vmbigip1
    ServerAdmin [email protected]
    DirectoryIndex index.html


   <Proxy *>
        Order deny,allow
        Allow from all
   </Proxy>

    SSLEngine on
    SSLProxyEngine On
    SSLCertificateFile /usr/local/apache/conf/server.crt
    SSLCertificateKeyFile /usr/local/apache/conf/server.key


   <Proxy balancer://mycluster>
       BalancerMember http://1.2.3.1:80
       BalancerMember http://1.2.3.2:80
       # technically we aren't blocking anyone, but could here
       Order Deny,Allow
       Deny from none
       Allow from all
       # Load Balancer Settings
       # A simple Round Robin load balancer.
       ProxySet lbmethod=byrequests
   </Proxy>

   # balancer-manager
   # This tool is built into the mod_proxy_balancer module allows you
   # to do simple mods to the balanced group via a gui web interface.
   <Location /balancer-manager>
       SetHandler balancer-manager
       Order deny,allow
       Allow from all
   </Location>

    ProxyRequests Off
    ProxyPreserveHost On

    # Point of Balance
    # Allows you to explicitly name the location in the site to be
    # balanced, here we will balance "/" or everything in the site.
    ProxyPass /balancer-manager !
    ProxyPass / balancer://mycluster/ stickysession=JSESSIONID
</VirtualHost>

What I need is for the servers in my load balancer to be

       BalancerMember https://1.2.3.1:443
       BalancerMember https://1.2.3.2:443

But that does not work. I get SSL negotiation errors.

Even when I do get that to work, I will need to pass client certificates.

Any help would be appreciated.

You simply cannot use a client certificate directly with a back-end node that would request the user certificate and where the load-balancer would "terminate" the SSL/TLS connection from the end-user.

The SSL/TLS handshake with a client certificate requires that the client signs all the handshake messages exchanged between the client and the server at the end, which means that the client must be connected directly to the actually SSL/TLS server requesting the client certificate. If the browser's SSL/TLS connection only goes as far as the load balancer, it's the load balancer that is the client to the back-end node. The back-end node will see a different handshake, and this will fail.

There are two possible ways around this:

• Use a DNS or TCP-based load-balancer (e.g. something like ipchains): in this case the SSL/TLS connection from the browser will go directly to the back-end node. Direct client-certificate authentication will be possible.

• Have the load-balancer perform the client-certificate authentication, and simply convey that information to the back-end node. This requires the back-end node to trust the load-balancer to have made the verification properly, but if the back-end node can't trust the load-balancer, there's no point using one.

  mod_proxy_ajp (or mod_jk) can forward the client certificate as part of the AJP protocol, but that's mainly for Java containers and AJP traffic isn't encrypted anyway.

  If you're using mod_proxy_http, you can add an extra header with mod_header to pass the certificate via an HTTP header, using something like RequestHeader set X-ClientCert %{SSL_CLIENT_CERT}s. I can't remember the exact details, but it's important to make sure that this header is cleared so that it never comes from the client's browser (who could forge it otherwise). Whatever application you have running on the back-end node will need to be able to get its authentication information from that header (and trust it as having been verified already by the load-balancer).

  In addition to the SSLVerifyClient and SSLCACertificateFile/Path directives that you need to configure for normal client-certificate authentication on Apache Httpd with mod_ssl, you'll also need to configure SSLProxyCheckPeerCN on and SSLProxyCACertificateFile/Path to configure Apache Httpd as an SSL/TLS client to the back-end nodes (see introduction of the mod_proxy documentation, about the SSLProxy* directives).

  If you want the back-end servers to make sure that the request come from the load-balancer and not a different client (which could make a direct connection), you could make the load-balancer use a client-certificate too (using SSLProxyMachineCertificateFile), to authenticate to the back-end note. Note that this may get the authentication system on the back-end not a bit more complex: although the actual client-certificate authentication they get would be that of the proxy (which they'd need to verify as usual), the applications on those servers would need to be configured to use the header-based client-certificate as far as application users are concerned.

Your SSL configuration looks incomplete. For requiring client certificates, you need to add

SSLVerifyClient require
SSLVerifyDepth  1
SSLCACertificateFile <your CA cert file>

See How can I force clients to authenticate using certificates for details.

Not sure if it's mandatory, but you're also missing a SSLCipherSuite directive. If done like this, you can sort out weak ciphers and do what you can against the BEAST attack:

SSLHonorCipherOrder On
SSLCipherSuite RC4-SHA:ALL:!ADH:!EXPORT:!LOW:!MD5:!SSLV2:!NULL