AWS ELB poor performance
We're using ELB for SSL termination. Our applications run on EC2 instances behind ELB using HTTP. We're monitoring the application response times and performance hit when going through the ELB seems to be very high: - Average response time when going through the ELB: 1+ second, - Average response time when ELB is bypassed: ~250ms.
It looks like going through the ELB adds about a second and occasionally goes up to 4-5 seconds. Is this expected? What's the typical overhead added by ELB SSL termination?
Any pointers would be much appreciated!
This may be due to how ELB is configured for you and it's definitely not normal. ELB nodes change size and scale based on load. If your site isn't receiving very much traffic it's possible your ELB has reduced in size down to a smaller node. You should contact AWS via Premium Support or the AWS Forums to have them check it out.
For those with similar issues who are trying to figure out how to diagnose their ELB, I wrote a tool called elbping that aims to make ELB diagnosis a little easier. I wrote this tool specifically to test that all of an ELBs nodes are responding by triggering an HTTP 405 (method not allowed) for ELBs in HTTP(s) mode.
It's available as a ruby gem, so if you have rubygems then you can install it by simply doing:
$ gem install elbping
Now you can run:
$ elbping -c 4 https://elb-123456789.us-east-1.elb.amazonaws.com
Response from 54.243.63.96: code=405 time=210 ms
Response from 23.21.73.53: code=405 time=189 ms
Response from 54.243.63.96: code=405 time=191 ms
Response from 23.21.73.53: code=405 time=188 ms
Response from 54.243.63.96: code=405 time=190 ms
Response from 23.21.73.53: code=405 time=192 ms
Response from 54.243.63.96: code=405 time=187 ms
Response from 23.21.73.53: code=405 time=189 ms
--- 54.243.63.96 statistics ---
4 requests, 4 responses, 0% loss
min/avg/max = 187/163/210 ms
--- 23.21.73.53 statistics ---
4 requests, 4 responses, 0% loss
min/avg/max = 188/189/192 ms
--- total statistics ---
4 requests, 4 responses, 0% loss
min/avg/max = 188/189/192 ms
If you see code=405 then that means that the ELB and its nodes are responding. These "pings" never go to your backend so these response times are only the round-trip time to/from the ELB.
In the case of SSL, this will help you discern how much time is being spent establishing the SSL connection to the ELB versus how long it's taking the ELB to communicate with your backends.
HTH