When to use RabbitMQ shovels and when Federation plugin?

Solution 1:

Shovels and queue provide different means to be forward messages from one RabbitMQ node to another.

Federated Exchange

With a federated exchange, queues can be connected to the queue on the upstream(source) node. In addition, an exchange on the downstream(destination) node will receive a copy of messages that are published to the upstream node.

Federated exchanges are a similar to exchange-to-exchange bindings, in that, they can (optionally) subscribe to a limited set of messages from an upstream exchange.

Federated Queue (NOTE: These are new in RabbitMQ 3.2.x)

With a federated queue, consumers can be connected to the queue on both the upstream(source) and downstream(destination) nodes.

In essence the downstream queue is a consumer on the upstream queue, with the expectation that there will be additional downstream consumers that process the messages in the same manner as a consumer attached to the upstream queue.

Any messages consumed by the downstream (federated) queue will not be available for consumers on the upstream queue.

Use Case:

If consumers are being migrated from one node to another, a federated queue will allow this to happen without messages being missed, or processed twice.

Use Case: from the RabbitMQ docs

The typical use would be to have the same "logical" queue distributed over many brokers. Each broker would declare a federated queue with all the other federated queues upstream. (The links would form a complete bi-directional graph on n queues.)

Shovel

Shovels on the other hand, attach an "upstream" queue to a "downstream" exchange. (I place the terms in quotes because the shovel documentation does not describe the nodes with the same semantics as the federation documentation.)

The shovel consumes the messages from the queue and sends them to the exchange on the destination node. (NOTE: While not normally discussed as part of this the pattern, there is nothing stopping a consumer from connecting to the queue on the origin node.)

To answer the specific questions:

What is the fine grain control in shovels which I am missing when I choose for federation?

A shovel does not have to reside on an "upstream" or "downstream" node. It can be configured and operate from an independent node.

A shovel can create all of the elements of the linkage by itself: the source queue, the bindings of the queue, and the destination exchange. Thus, it is non-invasive to either the source or destination node.

Is it correct that the only way to change the inter-vhost-communication when I use shovels is by changing theconfig file and rebooting the instance?

This has generally been the accepted downside of the shovel.

With the following command (caveat: only tested on RabbitMQ 3.1.x, and with a very specific rabbitmq.config file that only contain ) you can reload a shovel configuration from the specified file. (in this case /etc/rabbitmq/rabbitmq.config)

rabbitmqctl eval 'application:stop(rabbitmq_shovel), {ok, [[{rabbit, _}|[{rabbitmq_shovel, [{shovels, Shovels}] }]]]} = file:consult("/etc/rabbitmq/rabbitmq.config"), application:set_env(rabbitmq_shovel, shovels, Shovels), application:start(rabbitmq_shovel).'

.

Does the setup (vhost per application) make sense or am I missing the point completely?

This decision is going to depend on your use case. vhosts primarily provide logical (and access) separation between queues/exchanges and authorized users.

Solution 2:

Shovel acts like a well-designed built-in consumer. It can consume messages from a source broker and queue, and publish them into a destination broker and exchange. You could write an application to do that, but shovel already got it right - if all you need is to move messages from a queue to an exchange in the same or another broker, shovel can do it for you. Just as a well-behaving app, it can declare exchanges/queues/bindings, reconnect, change the routing key etc. You can set it up on the source or on the destination broker, or even use a third broker. It's basically an AMQP client.

Federation, on the other hand is used to connect your broker to one or multiple upstream brokers, or you can even create chains of brokers, bending the topology any way you like. You can federate exchanges or queues, and e.g. distribute messages to multiple brokers without the need to bind additional queues to a topic exchange or using a fanout exchange, and shoveling messages from each queue to a downstream broker.

To recap, federation operates at a higher level, while shovel is mostly "just" a well-written client.

To reconfigure shovel, you have to restart the broker, unfortunately.

I don't think you really need a per app vhost. You can add a per-app user to the broker without separate vhosts. Not sure what you mean on "share an event between vhosts", though.