Differences between local 'puppet apply' and 'puppet agent' to a puppetmaster

We're in the process of migrating away from a monolithic puppet repository that contains all of the config. This one repo contains things that really shouldn't be on every node ever, so a puppetmaster-based system seems the best way to appropriately segregate things.

The problem I'm having is that the same repo works just fine when copied to the local nodes and puppet apply /etc/puppet/manifests/site.pp run on it. Our installs run very cleanly.

When I put the puppet repo on the puppetmaster and get the agent signed, it doesn't appear to do anything other than report it's local catalog to the puppetmaster.

For a while there, and I haven't been able to reproduce the config that did this, one of our self-built modules was throwing an error that suggested it was attempting to copy a file from the local machine's modules directory, rather than the puppetmaster like it should.

Which suggests that there may be module and manifest syntax differences when building a repo for local use and via puppetmaster.

If there are any differences, what are they, or what are the main tripping points for conversion efforts?

The /etc/puppet/puppet.conf file on the master:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
pluginsync=true

# Set up Environments
## Each environment has a dedicated 'modules' directory under /environments/ from
## which puppet will preferentially pull. Otherwise, it'll use the top level  
## 'modules' directory. That is where common code goes.
[master]
  manifest=$confdir/manifests/site.pp
  modulepath=$confdir/environments/$environment/modules:$confdir/modules
  ssl_client_header= SSL_CLIENT_S_DN
  ssl_client_verify_header = SSL_CLIENT_VERIFY
[production]
  manifest=$confdir/manifests/site_prod.pp
[integration]
  manifest=$confdir/manifests/site_integ.pp
[development]
  manifest=$confdir/manifests/site_dev.pp
[operations]
  manifest=$confdir/manifests/site_ops.pp

At the moment, the site_prod.pp file and ilk are symlinks to site.pp.

The classes defined in the site.pp file work off of the hostname. As I mentioned, when run via puppet apply things work just fine. If worse comes to worst I can do what I need by way of a temporary NFS mount but I'd rather not do that.

site.pp For your enjoyment:

filebucket { "main": server => $server; }
File { backup => "main" }

Exec { path => "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" }

import "stages.pp"
import "int_setup.pp"
import "nodes.pp"

The imported .pp files are in the same directory as site.pp. nodes.pp is where the meat is, but also contains secret-sauce. Some excerpts:

node /clunod\-wk\d+\.sub\.example\.local/ {

  class { "int_setup": stage => "setup"; }
  include base
  include curl
  include custommodule
  [...]
}

Which can and does match nodes named clunod-wk01234.sub.example.local when run via puppet apply.

[int-setup.pp]

class int_setup {
  file { "/var/cluebat":
    ensure => directory,
    mode => "755",
    owner => "root",
    group => "root";
  }

  group{"puppet":
    ensure => present
  }

}

Solution 1:

The only trickiness that I'm aware of is in the file resource type.

Backup for replaced files behaves differently, using the server's filebucket by default instead of the local filebucket.

The more significant thing to be aware of is the source parameter.

source => '/tmp/somepath/sshd_config',

With a raw file path, it'll always try the local path.

source => 'puppet://puppetmaster1/modules/sshd/sshd_config',

With a puppet://server/ path, it'll always try the remote path.

source => 'puppet:///modules/sshd/sshd_config',

With an empty server specification, then it gets interesting.

Applied locally, the local puppet module path is used to find the file.

When reporting to a puppetmaster, the server that gave it the manifest is treated as the server.

Additionally, if you need to get creative about the source of a file, you can give the source parameter a list:

source => [ '/tmp/somepath/sshd_config', 'puppet:///modules/sshd/sshd_config'],

The first location where something's found will be used.

Solution 2:

The problem ended up being a difference in how the nodes were being selected. The code in the original nodes.pp file:

node /clunod\-wk\d+\.sub\.example\.local/ { )

Which correctly matches a node named clunod-wk01234.sub.example.local. This was working just fine when puppet apply was run against a local puppet manifest. But was not remotely.

The issue was how the localhost line was defined in /etc/hosts.

Non-working:

127.0.0.1  localhost clunod-wk01234 clunod-wk01234.sub.example.local

Working:

127.0.0.1  localhost clunod-wk01234.sub.example.local clunod-wk01234

The first form was reporting a node-name "clunod-wk01234" to the puppetmaster, the second form was reporting the FQDN. The second form satisfies the node delcaration, where the first does not. This was cracked by changing the node lines to not include the FQDN, at which point everything worked just fine.

The machines with the remote-puppet were an updated image, so did have some slight differences from the ones running local-puppet. One of these changes ended up being in how that one line in /etc/hosts was declared. Now we know.

I then found two file calls were written using hard paths, the rest were used correct puppet:/// syntax.

Solution 3:

There is no difference between 'local' and 'remote' puppet rules.

Can you post your site.pp so we can check for mistakes?

Is the server and clients use the same DNS server? Do their /etc/resolv.conf files differ in 'search' or 'domain' lines ?

You can also run puppet with --debug --verbose --no-daemonize options and get more output.