Architecture question: adding SSL into the mix

Solution 1:

whether the architecture as a whole makes sense (diverting incoming traffic

The diagrams you've provided don't show any branches. I'm a bit confused as to why you've got varnish in front of HA-Proxy instead of the other way around.

but my guts telling me this isn't really a long term solution

The SSL encapsulation should be in front of the HTTP caching (otherwise the content can't be cached).

Certainly it would be nicer to reduce the number of hops, but merging the SSL onto one of the existing layers wouldn't give that much of a performance benefit (at least assuming that at least one end of stunnel is connected via localhost). Its the architecture which Oracle, Cisco, f5 etcetera tend to recommend (that is, with the SSL at the front end, although with the exception that they think you should be running their kit in there somewhere!).

If it were me I'd split the cacheable/non-cacheable content onto different customer facing URLs. (even better use a CDN for all the cacheable content!)

Important questions you've not answered include how many IP addresses you have, how many webservers you have, and the split between cacheable/non-cachable and http/https.

            +--->(cacheable:80)->--------------+
            |                                  |
            +--->(cacheable:443)---> stunnel->-+->Varnish ->-+
 HAProxy ->-+                                                |
            +-->(non-cacheable:443)--> stunnel->-----+-------+---->Apache
            |                                         |
            +--->(non-cacheable:80)->-----------------+

Obviously, if you drop varnish, (optionally using mod_proxy within Apache) this becomes a lot simpler...

            +--->(:80)->------------+
 HAProxy ->-+                       |
            +--->(:443)---> stunnel-+---->Apache

Given the price of hardware, I'm far from convinced that using a caching reverse proxy is a good trade off between price/performance - unless you've got huge amounts of traffic and a large proportion is cacheable. OTOH if you are implementing logic (such as ESI) then its not very practical to not have the proxy, in which case the question becomes can Varnish provide the required load balancing rather than using HAPProxy?

  (:443)-->stunnel--+
                    |
  (:80)-------------+-->varnish-->Apache

Solution 2:

Wow, that stack is getting deep and complicated. Complexity is the enemy of uptime, and also in general the enemy of performance. Every one of those pieces has to manage connections, parse HTTP headers, etc.

I suggest you simplify things. Use nginx for SSL, load balancing, caching, as it supports all three with built-in modules. You can incrementally deploy it into your infrastructure as well, doing SSL only in front at first, then replacing HAproxy for load balancing the services, etc. You could event potentially ditch Apache and have nginx do almost everything if your services are written in a language that has a decent web or FastCGI server.

My small SaaS shop uses a flat nginx SSL/proxy/cache/static tier in front of Tomcat, IIS, and PHP-fastCGI back end services and static web servers. We see 2000 request per second peaks, and nginx isn't even breaking a sweat under that load with just two single-core VMware virtual machines at the front end of everything.