AWS ELB as backend for Varnish Accelerator
I am working on a large deployment on AWS that has high uptime requirements and variable loads throughout the day. Obviously, this is the perfect use case for ELB (Elastic Load Balancer) and autoscaling.
However, we also rely on varnish for caching of API calls. My initial instinct was to structure the stack so that varnish uses ELB as a backend which in turn hits an appGroup.
Varnish -> ELB -> AppServers
However, according to a few sources that isn't possible as ELB constantly changes the IP address of its DNS hostname, which varnish caches on start, meaning changes to the IP won't be picked up by varnish.
Reading around however, it looks like people are doing this so I am wondering what workarounds exist? Perhaps a script to reload the vcl periodically?
In the case of where this is really just not a good idea, any idea of other solutions?
This is absolutely possible, but it takes a few steps to get it working nicely! The way we do it here when we need that kind of configuration is:
Create a VPC. You need to do this in VPC because you need to make subnets in them.
Create one subnet in each Availability Zone that you will have instances that are registered with the ELB. You should subnet so that you have a small number of IP addresses in each subnet, as every IP address will become overhead. (We currently use subnets that are a /26)
Begin creating a DNS Director backend in your varnish VCL. Add in the 3 subnets you created above. (https://www.varnish-cache.org/docs/3.0/reference/vcl.html#the-dns-director)
Set the host setting in the DNS Director backend to be the host name that varnish should expect to see. (for instance, if your front end service is called, say, front-end-service.subdomain.example.com, put front-end-service.example.com as the Host setting in the VCL.)
Set the suffix setting in the DNS Director to something that you can resolve. Continuing the example above, you could easily use '-varnish.example.com' for your suffix. When a request hits varnish, varnish will look at the HTTP Host header, and if the name matches what varnish has in the VCL's DNS Director Backend for the Host header setting, varnish will append the suffix and perform a DNS lookup on the hostname that is the result of concatenating the contents of the Host header with the suffix. Thus, in this example, a DNS lookup will be performed by varnish for the host named "front-end-service.subdomain.example.com-varnish.example.com"
Create your loadbalancer backend and make attach it to each subnet that you have created.
Set a DNS record for the result of concatenation to be a CNAME for the DNS name that amazon provides your for your loadbalancer.
Start varnish, optionally look at varnishstat to verify the number of backends.
-
Test your setup by issuing a
curl -H "Host: front-end-service.subdomain.example.com" http://varnish-server-hostname.example.com/whatever-path
Watch the request come in with varnishlog to verify that everything is working.
It may be useful to note that AWS recommends that you should have a subnet with at least 20 unused IP addresses in it if you are going to place a loadbalancer in that subnet. (See http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/UserScenariosForVPC.html)
We have done this for a recent project that needed ELBs for a requirement spec, but we are concerned about how this scales with respect to ease of management and are looking into service discovery based approaches along with something like an automated VCL update, plus automated VCL deploy via something like Varnish Agent (https://github.com/varnish/vagent2)
However, if you don't mind managing your VPC subnets, the above description works very well.
Cheers!
Varnish can actually work as a load balancer. You should try Varnish -> AppServers.
Just define each app server as a backend in a director in Varnish config.
You can even add probes to check backend availability, retries to switch to another server when one fails during a request process, etc.
Where is your Varnish instance hosted ? ASW too ? You could try Varnish hash director and host Varnish on the same servers than apps. Each instance will process requests it's supposed to handle and forward others to the right backend. Each unique URL will only be cached on 1 (available) server and your cache memory will be multiplied by the number of Varnish instances while cache misses will be limited.
Part of the purpose of ELB is to survive host outages. Even with auto scaling and CloudWatch if a dead instance needs to be replaced you're looking at possibly several minutes of down time.
I recommend:
[Front End ELB] -> [Varnish] -> [Back End ELB] -> [AppServers]
I know that you want to take advantage of caching as much as possible, but you really should spread load across all availability zones. This means having an equal number of instances in zone A, B and C for the region(s) your stack is in (so 3x Varnish). This will of course cost more, but it gives you the ability to survive entire AZ outages*. Cutting this cost will mean that at some point you'll probably need to incur down time. That's your decision to make, but at least you can make it an informed decision.
Have a two security groups, one for Varnish and one for AppServers. Configure each so that only the associated ELB can access it on the appropriate port.
For the Varnish config, set the DNS director to have a low TTL. Set it to equal (or half) of the TTL of the CNAME that Amazon provides for the Back End ELB. That should be enough for Varnish to stay up to date.
* And if you want to go for ultimate availability. use Route53 with multi-region, multi-az redundancy.