RabbitMQ/AMQP - Best Practice Queue/Topic Design in a MicroService Architecture [closed]
We are thinking of introducing a AMQP based approach for our microservice infrastructure (choreography). We do have several services, let's say the customer-service, user-service, article-service etc. We are planning to introduce RabbitMQ as our central Messaging System.
I'am looking for best practices for the design of the system regarding topics/queues etc. One option would be to create a message queue for every single event which can occur in our system, for example:
user-service.user.deleted
user-service.user.updated
user-service.user.created
...
I think it is not the right approach to create hundreds of message queues, isn't it?
I would like to use Spring and these nice annotations, so for example:
@RabbitListener(queues="user-service.user.deleted")
public void handleEvent(UserDeletedEvent event){...
Isn't it better to just have something like "user-service-notifications" as one queue and then send all notifications to that queue? I would still like to register listeners just to a subset of all events, so how to solve that?
My second question: If I want to listen on a queue which was not created before, I will get an exception in RabbitMQ. I know I can "declare" a queue with the AmqpAdmin, but should I do this for every queue of my hundreds in every single microservice, as it always can happen that the queue was not created so far?
I generally find it is best to have exchanges grouped by object type / exchange type combinations.
in you example of user events, you could do a number of different things depending on what your system needs.
in one scenario, it might make sense to have an exchange per event as you've listed. you could create the following exchanges
| exchange | type | |-----------------------| | user.deleted | fanout | | user.created | fanout | | user.updated | fanout |
this would fit the "pub/sub" pattern of broadcasting events to any listeners, with no concern for what is listening.
with this setup, any queue that you bind to any of these exchanges will receive all messages that are published to the exchange. this is great for pub/sub and some other scenarios, but it might not be what you want all the time since you won't be able to filter messages for specific consumers without creating a new exchange, queue and binding.
in another scenario, you might find that there are too many exchanges being created because there are too many events. you may also want to combine the exchange for user events and user commands. this could be done with a direct or topic exchange:
| exchange | type | |-----------------------| | user | topic |
With a setup like this, you can use routing keys to publish specific messages to specific queues. For example, you could publish user.event.created as a routing key and have it route with a specific queue for a specific consumer.
| exchange | type | routing key | queue | |-----------------------------------------------------------------| | user | topic | user.event.created | user-created-queue | | user | topic | user.event.updated | user-updated-queue | | user | topic | user.event.deleted | user-deleted-queue | | user | topic | user.cmd.create | user-create-queue |
With this scenario, you end up with a single exchange and routing keys are used to distribute the message to the appropriate queue. notice that i also included a "create command" routing key and queue here. this illustrates how you could combine patterns through.
I would still like to register listeners just to a subset of all events, so how to solve that?
by using a fanout exchange, you would create queues and bindings for the specific events you want to listen to. each consumer would create it's own queue and binding.
by using a topic exchange, you could set up routing keys to send specific messages to the queue you want, including all events with a binding like user.events.#.
if you need specific messages to go to specific consumers, you do this through the routing and bindings.
ultimately, there is no right or wrong answer for which exchange type and configuration to use without knowing the specifics of each system's needs. you could use any exchange type for just about any purpose. there are tradeoffs with each one, and that's why each application will need to be examined closely to understand which one is correct.
as for declaring your queues. each message consumer should declare the queues and bindings it needs before trying to attach to it. this can be done when the application instance starts up, or you can wait until the queue is needed. again, this depends on what your application needs.
i know the answer i'm providing is rather vague and full of options, rather than real answers. there are not specific solid answers, though. it's all fuzzy logic, specific scenarios and looking at the system needs.
FWIW, I've written a small eBook that covers these topics from a rather unique perspective of telling stories. it addresses many of the questions you have, though sometimes indirectly.
Derick's advice is fine, except for how he names his queues. Queues should not merely mimic the name of the routing key. Routing keys are elements of the message, and the queues shouldn't care about that. That's what bindings are for.
Queue names should be named after what the consumer attached to the queue will do. What is the intent of the operation of this queue. Say you want to send an email to the user when their account is created (when a message with routing key user.event.created is sent using Derick's answer above). You would create a queue name sendNewUserEmail (or something along those lines, in a style that you find appropriate). This means it's easy to review and know exactly what that queue does.
Why is this important? Well, now you have another routing key, user.cmd.create. Let's say this event is sent when another user creates an account for someone else (for example, members of a team). You still want to send an email to that user as well, so you create the binding to send those messages to the sendNewUserEmail queue.
If the queue was named after the binding, it can cause confusion, especially if routing keys change. Keep queue names decoupled and self descriptive.
Before answering the "one exchange, or many?" question. I actually want to ask another question: do we really even need a custom exchange for this case?
Different types of object events are so natual to match different types of messages to be published, but it is not really necessary sometimes. What if we abstract all the 3 types of events as a “write” event, whose sub-types are “created”, “updated” and “deleted”?
| object | event | sub-type |
|-----------------------------|
| user | write | created |
| user | write | updated |
| user | write | deleted |
Solution 1
The simplest solution to support this is we could only design a “user.write” queue, and publish all user write event messages to this queue directly via the global default exchange. When publishing to a queue directly, the biggest limitation is it assumes that only one app subscribes to this type of messages. Multiple instances of one app subscribing to this queue is also fine.
| queue | app |
|-------------------|
| user.write | app1 |
Solution 2
The simplest solution could not work when there is a second app (having different processing logic) want to subscribe to any messages published to the queue. When there are multiple apps subscribing, we at least need one “fanout” type exchange with bindings to multiple queues. So that messages are published to the excahnge, and the exchange duplicates the messages to each of the queues. Each queue represents the processing job of each different app.
| queue | subscriber |
|-------------------------------|
| user.write.app1 | app1 |
| user.write.app2 | app2 |
| exchange | type | binding_queue |
|---------------------------------------|
| user.write | fanout | user.write.app1 |
| user.write | fanout | user.write.app2 |
This second solution works fine if each subscriber does care about and want to handle all the sub-types of “user.write” events or at least to expose all these sub-type events to each subscribers is not a problem. For instance, if the subscriber app is for simply keeping the transction log; or although the subscriber handles only user.created, it is ok to let it know about when user.updated or user.deleted happens. It becomes less elegant when some subscribers are from external of your organization, and you only want to notify them about some specific sub-type events. For instance, if app2 only wants to handle user.created and it should not have the knowledge of user.updated or user.deleted at all.
Solution 3
To solve the issue above, we have to extract “user.created” concept from “user.write”. The “topic” type of exchange could help. When publishing the messages, let’s use user.created/user.updated/user.deleted as routing keys, so that we could set the binding key of “user.write.app1” queue be “user.*” and the binding key of “user.created.app2” queue be “user.created”.
| queue | subscriber |
|---------------------------------|
| user.write.app1 | app1 |
| user.created.app2 | app2 |
| exchange | type | binding_queue | binding_key |
|-------------------------------------------------------|
| user.write | topic | user.write.app1 | user.* |
| user.write | topic | user.created.app2 | user.created |
Solution 4
The “topic” exchange type is more flexible in case potentially there will be more event sub-types. But if you clearly know the exact number of events, you could also use the “direct” exchange type instead for better performance.
| queue | subscriber |
|---------------------------------|
| user.write.app1 | app1 |
| user.created.app2 | app2 |
| exchange | type | binding_queue | binding_key |
|--------------------------------------------------------|
| user.write | direct | user.write.app1 | user.created |
| user.write | direct | user.write.app1 | user.updated |
| user.write | direct | user.write.app1 | user.deleted |
| user.write | direct | user.created.app2 | user.created |
Come back to the “one exchange, or many?” question. So far, all the solutions use only one exchange. Works fine, nothing wrong. Then, when might we need multiple exchanges? There is a slight performance drop if a "topic" exchange has too many bindings. If performance difference of too many bindings on “topic exchange” really becomes an issue, of course you could use more “direct” exchanges to reduce number of “topic” exchange bindings for better performance. But, here I want to focus more on function limitations of “one exchange” solutions.
Solution 5
One case we might natually consider multiple exchanges is for different groups or dimensions of events. For instance, besides the created, updated and deleted events memtioned above, if we have another group of events: login and logout - a group of events describing “user behaviors” rather than “data write”. Coz different group of events might need completely different routing strategies and routing key & queue naming conventions, it is so that natual to have a separate user.behavior exchange.
| queue | subscriber |
|----------------------------------|
| user.write.app1 | app1 |
| user.created.app2 | app2 |
| user.behavior.app3 | app3 |
| exchange | type | binding_queue | binding_key |
|--------------------------------------------------------------|
| user.write | topic | user.write.app1 | user.* |
| user.write | topic | user.created.app2 | user.created |
| user.behavior | topic | user.behavior.app3 | user.* |
Other Solutions
There are other cases when we might need multiple exchanges for one object type. For instance, if you want to set different permissions on exchanges (e.g. only selected events of one object type are allowed to be published to one exchange from external apps, while the other exchange accepts any the events from internal apps). For another instance, if you want to use different exchanges suffixed with a version number to support different versions of routing strategies of same group of events. For another another instance, you might want to define some “internal exchanges” for exchange-to-exchange bindings, which could manage routing rules in a layered way.
In summary, still, “the final solution depends on your system needs”, but with all the solution examples above, and with the background considerations, I hope it could at least get one thinking in the right directions.
I also created a blog post, putting together this problem background, the solutions and other related considerations.