How do I prevent puppet interfering with manual server changes

A lot of our server configuration is managed via puppet. Puppet runs every 15 minutes via cron and applies configuration changes.

If a server has problems I sometimes need to change configuration manually on the server (e.g try different settings, enable more logging). The problem is that puppet then overwrites my manual changes whenever the cron job runs.

What would you recomment to handle this problem?

Maybe deactivate puppet while working on the servers? Could be the risk that I forget to reenable it and then have one server out of sync.

Solution 1:

What I normally do is the following

• Manually stop puppet on that machine
• Have a nagios or monitoring tool of your choice alert that will warn me when puppet hasn't been run for more than 2 hours
• If you want to be extra cautious also make a pupet watchdog, one that checks that the /var/run file is in place, if you stop puppet properly it'll erase it, if the file is there but there's no process it died so it'll start up again

With #1 and #2 you'll cover 99% of your case scenarios, and if you want to maybe #3 will help you when puppet misbehaves (sometimes it did on me)

Good luck!

Solution 2:

What I do is simply make it part of the routine to check the Puppet Dashboard for hosts that haven't sent new reports recently, and disable Puppet while working. Dashboard has a section specifically for those hosts, which makes it pretty easy to identify them.

Alternatively, you could comment out the configuration for that node, or add a schedule to it that excludes the time you'll be working on it. In both cases you can still forget it, though the latter case the problem is mitigated.

Solution 3:

lynxman's answer is a good one. I would follow his advice.

In addition, I would look at what you are asking puppet to do.

For example, if you are asking puppet to copy a file over /etc/hosts, could you instead edit the file, adding/setting the definitions you need (via puppet-augeas)? If you needed to make other changes to the file for debugging on a host, so long as your changes weren't related to the ones puppet was making, puppet would then no-longer overwrite your work.