AWS SQS + SNS + Lambda

Solution 1:

One thing I did was to create a CloudWatch alarm on ApproximateNumberOfMessagesVisible (>= 1 for 5 minutes) for the SQS queue. The alarm publishes to an SNS topic that triggers the lambda function. The lambda function loops until it clears the queue.

It can take up to 5 minutes to trigger from the alarm, but it works fantastically for batch-scheduled tasks without needing to poll the queue. (Alarm granularity is 5 minutes for active queues.)

Solution 2:

You can't go SQS -> SNS, only SNS -> SQS.

Lambda now supports scheduling so one option is to implement an SQS poller in a Lambda function and run it frequently.

Another option to consider is whether you actually need a queue. Lambda supports asynchronous processing (via the Event invocation mode) and should transparently scale horizontally to handle parallel invocations. If your lambda function doesn't require access to a central state store which might constrain parallel execution then you could probably just run all your invocations in parallel. I believe there's a 100 concurrent execution limit per account though, so you may need to batch your messages to stay under that.

Solution 3:

SQS queue can be subscribed to SNS topic and so to process received SNS messages. Currently, it is not doable in other direction without additional coding (see e.g. Lambda FAQ).

I would say there is a couple of options how to do it but it is not so elegant as using more common event-driven system AWS event->SQS->Lambda. Otherwise you may need to customize/implement the code how SQS queues are processed:

1. you can implement your own event sources
2. you can have some intermediate EC2 instance to listen to SQS queues and then to trigger Lambda on SQS events