Linux system configuration management: versioning and deployment

I'm looking for the best way (for my purposes and situation) to version system configuration files and manage deployment (ownership and mode) between just two machines (live and backup, both should be regarded as 'production'). I would like to be able to use this system to automatically (but selectively) duplicate and manage relevant and specified configuration on the backup machine (an EC2 instance).

Currently, those machines are primarily web and SMTP servers (we don't store user mailboxes), but our project is growing and who knows what future requirements may be. For example, there is the possibility that we'll implement radio and web-based chat at some point in the future.

I prefer to manage the server conventionally (ie, go in and make changes to the live server as required) and then store those changes (once settled) in the management system. (Okay, if I'm honest, I should say organically: I don't want to lose the flexibility to do stuff as needs and situations demand.) I find this approach, rather than deployment of config that was tweaked and tested elsewhere, more predictable and with the potential for fewer surprises (outside of the possibility of screwing up configuration on a live system, that is!)

I realise that, philosophically speaking, this is back-to-front and I probably would do it the other way around if I had the resources (test infra and, more importantly, manpower). As it is, I've got to make do with what I've got. I like structure well enough, but excessive infrastructure takes on a life of its own and at some point the value in it is lost.

I've done some homework and reviewed a couple of options, but none of them seem to really do what I want them to, so I'm currently tempted to roll my own solution. The purpose of this question is to see what I've not thought of and to sanity-check before I go barking up the wrong tree.

Options

1. Declarative metaconfiguration systems like Puppet, inter alia, are probably good choices where many servers are involved with little or no differentiation between them, and especially when it's clear ahead of time what configuration should look like. I don't think it'd be wise to employ such a tool without staging infrastructure.

   It's also a bit heavy-weight and overly complicated. I'm the only admin, I do this in my spare time, and should something happen to me whatever I leave behind needs to be reasonably sane for the lucky guy/girl who steps in after me.

2. etckeeper might do, but so far as I've seen, it doesn't really lend itself to managing anything other than /etc. ie, it wouldn't be suitable for storing custom scripts which, by tradition, live in /usr/local/bin. Also, etckeeper presumably manages all of /etc, where I think I'd prefer to version only specific stuff (eg postfix, apache, fail2ban, ssh and possibly munin).

3. A simple git or svn repo would certainly not do, for while git tracks permissions, it doesn't track ownership, and svn tracks neither.

   svn keeps a .svn subdirectory in every tracked directory, whereas .git exists only in the working copy's root directory. There are benefits and disadvantages in both cases. The presence of an .svn directory in a certain location might be okay in some places (eg /etc/apache2/sites-enabled) but upset brain-dead scripts/tools elsewhere. (I'm sure I read something a while back: was it that cron was okay with a .svn dir in /etc/cron.d, but depmod was not with /lib/modules?)

   On the other hand, with a .git in /, git considers everything a candidate for tracking, even (presumably) other git repos!

Custom solution

Here's what I propose to do: a subversion repo stored on the primary machine (say /usr/local/sc-repo) and a management script written in Pysvn that implements the following:

• add (but, by default, not recursively), which stores the file's mode and owership as custom properties. Also, I like $Id$ tags in my config files, so this will ensure svn:keywords is properly set.

• commit

• update, which applies ownership and mode after the file is written

• revert, ditto

• perms, like diff, but for ownership and mode, with fix flag to correct if required.

The backup machine will access the repo via ssh and perform an update operation every night. (It will do other things, too, like restore site SQL and application filesystems from the main machine's backup repo (S3), and I will probably have to write some hackery that looks for new packages added on the main machine and adds them on the backup machine — but all that is outside of the scope of this Question.)

Why subversion?

• I know, like and am much more familiar with svn than I am with git. Yes, I'm probably a dinosaur.

• Pysvn is easy, and I've used it before.

• svn supports versioned custom properties, where git doesn't seem to

• this is not distributed development: distributed repos complicate matters without benefit. Remote repo availability is irrelevant, because commits from the remote machine will be rare.

I'd be grateful for your feedback. I have thought as much of this through as I could, but I'm bound to have missed something.

Solution 1:

Don't reinvent the wheel. The tools exist to do what you need. You may want to look at bcfg2, especially since you seem to be particularly focused on configuration files. Think about services, too. Where does the central repository need to be in relation to the production server?

Maybe an intermediate option could be carful modification of the output of the Blueprint tool. I use Blueprint to reverse-engineer existing systems. The output of a Blueprint run can create a shell script, Puppet Module or a Chef Cookbook... Once you have a verified configuration on the production server, you can use this tool to capture the system's state, then tag and version the result. Read through the examples and report back if this seems like it fits your use case.

Solution 2:

Declarative metaconfiguration systems like Puppet, inter alia, are probably good choices where many servers are involved with little or no differentiation between them, and especially when it's clear ahead of time what configuration should look like. I don't think it'd be wise to employ such a tool without staging infrastructure.

I use Puppet (but yes, Chef and Ansible and Salt and such exist) even for single-machine setups. I'm yet to have a situation where anyone's been clear ahead of time what machine configuration should look like (arguably, many haven't been clear after-time what the configuration should look like, but that's a different problem altogether)

As for "staging infrastructure", that's Vagrant and VirtualBox (or some other backing virtualisation tech) - run it on your dev machine for zero-spend. I don't do any Puppet manifest development without it. Also automated tests with Travis CI

It's also a bit heavy-weight and overly complicated. I'm the only admin, I do this in my spare time, and should something happen to me whatever I leave behind needs to be reasonably sane for the lucky guy/girl who steps in after me.

It seems inconsistent to say this and then describe how you're planning a home-grown solution instead of existing ones with plenty of documentation, examples, existing "libraries" and active development.