MongoDB as a queue service? [closed]

I am using mongodb as a queue service for email sending. It will work in the following way:

1. When a new message comes I store it in the mongodb.
2. A background job then loads the message from mongodb via the atomic operation findAndModify and sets the flag Processing to true, so it does not process the same message twice (because my background job runs multiple threads in parallel).
3. Once the email has been sent, I remove the document from mongodb.
4. You can also keep a count of the failures for each message and remove it after 3 failed attempts.

In general, I use mongodb as a queue service only for one reason: because I need to send emails by specified schedule (each message contains information on what time it should be sent).

If you do not have any schedule and need to process messages immediately, I suggest that you look into existing queue services, because they probably handle all cases that you may not see without a deeper understanding of message queues.

Update

When a background job crashes during message processing you could do the following:

1. Move this message to another, message queue errors collection, or...

2. Increase processing attempts counter in a message and again assign status "New", to try process it again. Just make sure that the background job is idempotent (can process the same message multiple times and not corrupt data) and transactional (when a job fails you must undo changes that were made. if any). When a job fails after 5 attempts (config value) perform #1.

3. Once the bug with message processing was fixed you could process it again once more by assigning "New" status and moving to the message queue or just deleting the message. It depends on business processes actually.

I know that this question is back from 2012, but during my own research i found this article and just want to inform any other user that the devs from serverdensity replaced rabbitmq in favor of a simple queueing system with mongodb.

A detailed article is given here:

https://blog.serverdensity.com/replacing-rabbitmq-with-mongodb/

Here is a great article explaining how someone used mongoDB's replication oplog as a queue.

You can do the same with a different collection. The main piece of advice seems to be to use a capped collection — mongo drivers have efficient means of waiting on a capped collection so that the client isn't constantly polling.

I've searched a lot and found the JavaScript version https://github.com/chilts/mongodb-queue. But I want a go version, so a simple implementation in Go, including a manager to poll messages was made: https://github.com/justmao945/mongomq