Why use Celery instead of RabbitMQ?

From my understanding, Celery is a distributed task queue, which means the only thing that it should do is dispatching tasks/jobs to others servers and get the result back. RabbitMQ is a message queue, and nothing more. However, a worker could just listen to the MQ and execute the task when a message is received. This achieves exactly what Celery offers, so why need Celery at all?

Solution 1:

You are right, you don't need Celery at all. When you are designing a distributed system there are a lot of options and there is no right way to do things that fits all situations.

Many people find that it is more flexible to have pools of message consumers waiting for a message to appear on their queue, doing some work, and sending a message when the work is finished.

Celery is a framework that wraps up a whole lot of things in a package but if you don't really need the whole package, then it is better to set up RabbitMQ and implement just what you need without all the complexity. In addition, RabbitMQ can be used in many more scenarios besides the task queue scenario that Celery implements.

But if you do choose Celery, then think twice about RabbitMQ. Celery's message queueing model is simplistic and it is really a better fit for something like Redis than for RabbitMQ. Rabbit has a rich set of options that Celery basically ignores.

Solution 2:

Celery basically provides a nice interface to doing just what you said, and deals with all the configuration for you. Yes you could do it by hand, but you'd just be rewriting celery.