AWS ElastiCache/SimpleQueue vs DynamoDB

I'm wondering about the rationale for using ElastiCache/SimpleQueue vs just having "Cache" and "Queue" tables inside of DynamoDB respectively.

It seems that the network latency to the Cache/Queue services would trump a lot of the performance gains, and that having EC2 treat Dynamo as it's cache/queue service would offer the same latency and throughput (since Dynamo allows a fixed low latency under any load).

Is it mainly about the price of dynamo vs other services under load?

Does anyone have any rough latency numbers comparing Dynamo with ElastiCache/SQS?

Are there other more important considerations that I'm missing which justify the additional complexity?

Thanks.

Solution 1:

We're using DynamoDB and ElastiCache Redis for different reasons.

DynamoDB:

• Has a query language which is able to do more complex things (greater than, between etc.)
• Is reachable via an external internet-facing API (different regions are reachable without any changes or own infrastructure)
• Permissions based on tables or even rows are possible
• Scales in terms of data size to infinity
• You pay per request -> low request numbers means smaller bill, high request numbers means higher bill
• Reads and Writes are different in costs
• Data is saved redundant by AWS in multiple facilities
• DynamoDB is highly available out-of-the-box
• Autoscaling is available in the service itself

ElastiCache Redis:

• Simple query language - no complex features
• Is (out-of-the-box) not reachable from other regions.
• You're always limited to the amount of memory (or the sum of all primary instances in a cluster)
• Sharding over multiple instances is only possible within your application - Redis doesn't do anything here (Redis cluster helps here but the sharding logic is still inside of the driver/sdk you're using in your application) - scale-in and scale-out is not possible without downtime at the moment
• You pay per instance no matter how the load or the number of requests are.
• If you want redundancy of the data you need to setup replication (not possible between different regions)
• You need to use replicas for high availability
• No autoscaling available (see the part about no scaling at all above)

So our setup most of the time is: Simple caches with high volume of requests in Redis backed by DynamoDB as the permanent and long-durable storage. With this we limit the costs as we get an implicit discount for our reads by the pay-per-instance model of Redis but also get the benefit of the redundancy of DynamoDB and are even able to use the DynamoDB query language for more complex stuff (if we need it).

Hope that helps!

Update: With the announcement of Amazon DynamoDB Accelerator (https://aws.amazon.com/de/dynamodb/dax/) we're switching over to use DAX as it is (in the end) exactly what we were doing with the combination of DynamoDB and Redis. As DAX ist fully-managed by AWS and gives us the chance to always use the DynamoDB language in our application but also get the benefits from a write-through cache like Redis.

Solution 2:

The main reason we use Elasticache rather than DynamoDB is the speed - you get sub 1ms round trip latency for small objects. The box is really close to your EC2 machine, and memory is that much faster than disk, even SSD.

There could also be a cost advantage given the different pricing models, although I haven't gone into that much detail there.

Solution 3:

Redis/memcached are in-memory stores and should generally be faster than DynamoDB for cache/queue-type data. They also have handy additional items like expiring keys, Pub/Sub in Redis, etc. that Dynamo may not have.