Is there a way to use length of a RabbitMQ queue used by Celery to start instance in an autoscale group?
Is there any way for Celery to emit events when the length of a queue exceeds a threshold? I want to use that event to start an EC2-instance.
We have two queues for two different tasks in Celery. One of these queues has tasks which will require an instance to be started in the autoscaling group. In the first autoscaling group which handles one kind of tasks, we always have one instance running. It is then easy to start another instance using Amazon CloudWatch. But for the other autoscaling group we don't want it to have any instances if there are no tasks because many a time there might not be tasks to be performed by this group. So only if the length of the Celery queue for this task goes above a given threshold, we want to start an instance in this autoscaling group. What would be a good way to do this?
I'd suggest a cron job to publish the current queue length as a CloudWatch custom metric, which will allow you to create auto-scaling alarms based on its value.