How to run Puppet on multiple nodes at once using MCollective?

Aim

At the moment there are more than 100 systems in the office and the expectation is that the number will increase. All systems are managed by Puppet. If Puppet code changes on the Puppet Master and needs to be implemented on multiple systems, Puppet is run manually by executing sudo puppet agent -t on every system. Sometimes a change needs to be implemented on more than 10 systems. In order to safe time the idea is to use MCollective, i.e. run puppet one time on all systems instead of running puppet on every system.

Attempt

ActiveMQ, MCollective Server and MCollective Client have been installed on the Puppetmaster. Once mco ping is run the Puppetmaster can be found. After installing MCollective Puppet Agent and running mco rpc puppet runonce Puppet is run on the Puppetmaster.

In order to connect a remote mcollective-client the idea was that activemq and mcollective-client have to be installed and run in order to find the second host by issuing mco ping.

/var/log/mcollective.log

INFO -- : activemq.rb:114:in `on_connecting' TCP Connection attempt 23 to stomp://mcollective@stomp1:6163

/etc/activemq/activemq.xml

    <transportConnectors>
        <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
            <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
        </transportConnectors>
</broker>

/etc/mcollective/server.cfg

connector = activemq
plugin.activemq.pool.size = 1
plugin.activemq.pool.1.host = localhost
plugin.activemq.pool.1.port = 61613
plugin.activemq.pool.1.user = mcollective
plugin.activemq.pool.1.password = helloworld

One of the assumptions was that multiple hosts should be configured in the server.cfg. While trying to make it work it became unclear how the architecture should look like. One ActiveMQ, one MCollective Server and multiple MCollective Clients (one on every node) are required or not?

Expected result

The expected result is that mco ping finds the Puppetmaster and the remote client. Once this works it would be possible to run Puppet on both systems using MCollective. When it is possible to execute Puppet on these two systems using MCollective, multiple nodes will be added and Puppet will run on these systems once mco rpc puppet runonce has been executed.

Current outcome

As it is not possible to find the remote client by executing mco ping it is not possible to run Puppet on both systems using MCollective.

One ActiveMQ server is required, MCollective needs be installed on every node and the plugin.activemq.pool.1.host snippet needs to contain the IP of the ActiveMQ server.

[vagrant@localhost ~]$ mco ping
test                                  time=225.70 ms
test2                                 time=488.51 ms


---- ping statistics ----
2 replies max: 488.51 min: 225.70 avg: 357.10

/etc/mcollective/server.cfg

connector = activemq
plugin.activemq.pool.size = 1
plugin.activemq.pool.1.host = IP_ACTIVEMQ_SERVER
plugin.activemq.pool.1.port = 61613
plugin.activemq.pool.1.user = mcollective
plugin.activemq.pool.1.password = helloworld

Once the Puppet nodes were found, executing mco rpc puppet runonce runs Puppet on all the systems.