Puppet file transfer slow

I have a puppet master and slaves in different datacenters. The latency between them is ~40ms. When I run "puppet agent --test" on a slave to apply the latest manifest it takes ~360 seconds to finish.

After doing some digging I can see the main cause of the slow down is file transfers. It seems it's taking ~10 seconds to transfer each file. The files are only small (configuration files) so I can't understand why they would take so long.

This is an example of a file in my manifest:

file { "/etc/rsyncd.conf" :
    owner       => "root",
    group       => "root",
    mode        => 644,
    source  => "puppet:///files/rsyncd/rsyncd.conf"
}

Running puppet-profiler I see this:

10.21s - File[/etc/rsyncd.conf]

It also seems I cannot update more than one server at once using puppet. If I run two servers at the same time then puppet takes twice as long.

I have changed the puppet master from using webrick to mongrel, but this doesn't seem to help. This is making deploying changes painful. A simple config change can take an hour to roll out to all servers.

Solution 1:

Disclaimer: I am one of the Puppet developers.

When you source files with puppet:// URLs, Puppet will make two SSL connections back to the file server - one for the metadata, including checksum, and another for the content. The second is made, hopefully as expected, only if the content on disk is out of date.

The transfer itself is standard HTTPS, with nothing fancy going on, so beyond the SSL overhead and the latency cost of two connections, there is no reason that Puppet should be so slow.

Puppet will also checksum the file on both sides, the client and the server, when it comes to checking if it is up to date. That may be slow if your backing store is slow, too.

Finally, you should check if your network link is slow because it is saturated, or something like that. Wouldn't be the first time big buffers led to terrible network performance - and someone put a traffic shaper in the middle that made pings super-fast. ;)

Beyond that, I suggest you file a bug report. Including the details, and network traces, would be a big help in our being able to resolve this.

What you describe certainly isn't normal, and it isn't what we would expect to see for small configuration files on a reasonably fast network such as you describe. (Also, it isn't what others see, so clearly something unusual is happening in your setup.)

Solution 2:

In my experience older versions of puppet would serialize and DE-serialize all FILE data as they transferred it back and forth. And it was particularly bad and transferring large (many megabyte) binaries.

Mr Pittman suggests above that this may no longer be the case, but you may want to dig deeper and see if the version of puppet you're running has that optimization.

It looks like the change went in to puppet 0.25 - see bug 583 (http://projects.puppetlabs.com/issues/583) when they converted from xlrpc to rest.

My solution at the time was to use puppet to push rsync pre-shared keys and use an rsync daemon to serve up files (read-only) to end clients. Rsync can be quite efficient and transferring and updating lots of binary data.

So check your puppet versions, but also see if rsync can work for you as a performance comparison.