HAProxy SSL roundrobin not working when SSL terminated and forwarded

I am using the following config to terminate SSL so that I can inspect the request, do URL rewriting, ACL etc, and then forward the SSL traffic back to my backend servers. However, I am not able to get sticky to work. I can get sticky to work when I just use "mode tcp" and do straight forward tcp routing, but sticky stops working as soon as I start terminating SSL at front-end.

Here's my config:

frontend https-forward
  bind *:443 ssl crt /etc/haproxy/certs.d/combo.pem

  option http-server-close
  option forwardfor
  reqadd X-Forwarded-Proto:\ https
  reqadd X-Forwarded-Port:\ 443

  capture request header Referrer len 64
  capture request header Content-Length len 10
  capture request header User-Agent len 64

  # set HTTP Strict Transport Security (HTST) header
  rspadd  Strict-Transport-Security:\ max-age=15768000

  # some ACLs and URL rewrites...

  default_backend backstuff

backend backstuff

  log 127.0.0.1 local2 notice

  balance roundrobin
  option ssl-hello-chk
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello
  stick store-response payload_lv(43,1) if serverhello

  server PO1 10.35.59.160:443 ssl verify none maxconn 5000
  server PO2 10.35.59.161:443 ssl verify none maxconn 5000

Solution 1:

The reason you can't stick on SSL Session ID like you are when using mode http (the default unless you explicitly specify mode tcp) is that you're trying to do it on certain bytes of the packet's payload, when in reality the packets have already been decoded and those offsets might contain totally random data.

You have two options here.

1. Stick based on source IP

   If you're not opposed to sticking on the client's IP, instead of the SSL session ID like you're doing now, then you can change your config to look like this:

   frontend https-forward
     bind *:443 ssl crt /etc/haproxy/certs.d/combo.pem
     mode http
   
     option http-server-close
     option forwardfor
     reqadd X-Forwarded-Proto:\ https
     reqadd X-Forwarded-Port:\ 443
   
     capture request header Referrer len 64
     capture request header Content-Length len 10
     capture request header User-Agent len 64
   
     # set HTTP Strict Transport Security (HTST) header
     rspadd  Strict-Transport-Security:\ max-age=15768000
   
     # some ACLs and URL rewrites...
   
     default_backend backstuff
   
   backend backstuff
     mode http
     log 127.0.0.1 local2 notice
     balance roundrobin
     option ssl-hello-chk
   
     stick-table type ip size 30k expire 30m
     stick on src
   
     server PO1 10.35.59.160:443 ssl verify none maxconn 5000
     server PO2 10.35.59.161:443 ssl verify none maxconn 5000
   

   The key changes are the the stick-table and stick on lines, as well as the explicit use of mode http.

   As you've said, if many clients accessing your site are behind a NAT, they'll all end up on the same server, so it's not the smoothest distribution but it does work and provides the functionality you want.

2. Use the HAProxy-decoded SSL session ID

   Here'd you'd have to leverage HAproxy's knowledge of the connection via ssl_fc_session_id (docs).

   ssl_fc_session_id : binary
   Returns the SSL ID of the front connection when the incoming connection was made over an SSL/TLS transport layer. It is useful to stick a given client to a server. It is important to note that some browsers refresh their session ID every few minutes.

   In this case, you'd use the same config as I've provided above, but with the stick-table and stick lines changed to:

     stick-table type binary len 32 size 30k expire 30m
     stick on ssl_fc_session_id
   

   This is really most akin to what you're trying to achieve.