Automating server deployment

Puppet sounds perfect for what you're trying to do, with the caveat that as of right now, there is no support for Windows.

In your case, you would define a Server node in terms of all of the packages that are identical across the machines. Then, you define the individual hosts as nodes which inherit from Server, and set up the specific unique things for it.

Puppet is declarative - it allows you to describe of your boxes in terms of the resources each box should have. So if you want ssh - you write a class for that resource - and inside the class you can include logic about how ssh is called slightly different on FreeBSD vs Ubuntu. It also knows to use yum inside Redhat and apt-get inside Debian based distros, and ports in the BSDs. Now in your Server node, you'll just have a line like include ssh - and puppet will do the right thing and put SSH on the machine without you having to remember if that's Ubuntu or Redhat or FreeBSD.

What's nice is that all of the Server stuff lives in one place - and if at any point you add to the Server node definition, ALL machines would update their configuration accordingly.

Right now, I'm only managing three boxes using Puppet - but it's already paid off. After spending a week setting up a box we'll be using for stimulus presentation in an experiment, it turned out the graphics card driver was too old in the version of Ubuntu I put on it (8.04). I had to install the latest Ubuntu (9.04), but after that I just had to apt-get and run puppet - and everything I had spent a week setting up was restored.

Puppet does have a little bit of a learning curve, but I've successfully avoided learning Ruby - I know I'm using it, since that's what puppet is written in - but so far I've been successful in just modifying the examples in the documentation and the recipes on the wiki . Another downside is that puppet does take a bit longer to do things the first time. The upside is that everything you change across all of your machines is stored in one place - it's standard practice to keep your puppet configuration in a version control system - so you can always look back and see how you've set up servers in the past - or roll-back some unsuccessful changes.

Finally, here's a quick video that does a simple puppet demo that got me started quickly.

We use Cobbler and Puppet for build and configuration automation of both real and virtual machines.

Cobbler ties together DHCP, PXE boot and Kickstart to make deployment nothing more than adding a machine profile and hitting the power button. For VMs, the koan command does the (in our case) Xen magic to get the installation started -- on the dom0 I just type:

koan --system vps.fqdn --server cobbler --no-gfx

then virsh console to watch a VPS building without any interaction.

We use RHEL and have a bunch of profile set up to partition disks, configure networking and install base packages for different server classes. Cobbler supports Debian and Ubuntu breeds but I've never tried it. An aside: other interesting uses for Cobbler include running memtest ISOs and HP firmware updates.

Once our systems are built with Cobbler Puppet takes over to configure the applications, system daemons, register the box with RHN, etc. Puppet runs as a daemon which periodically checks that the systems's configuration matches the defined manifests -- you know your updates have gone to all servers. It's also a great way to be certain that a box that's been down for maintenance has the correct configuration before you return it to live service.

Puppet really is awesome. You don't need to get every aspect of your configuration under its control -- start off by having it manage something simple that you need to configure on every box (sudoers is the canonical example) and take it from there. Make sure your Puppet manifests are versioned, too; nothing's better than being easily able to roll back to a known-good configuration without having to remember what to adjust.

Where I am working at the moment, we have to manage the Linux portion of our server farm which is just over 300 Linux servers. This includes mostly HP Proliants, followed by IBM 3850s, some IBM blades, VMware ESX, and some KVM for our internal management servers.

cobbler

We looked at cobbler but the problem there was that cobbler is very RHEL/Red Hat specific. We need to support RHEL and SLES at the least, and Ubuntu is next.

puppet

We did consider puppet, however later decided against it as it depends on Ruby, which means that an upgrade of Ruby, could potentially break our management system.

hotwire

Hotwire is what we use (developed internally, but is open-source), and have done so for the last few years. It firstly inventories the systems that are going to be built, which means inventorying the data-center, rack, hardware, operating system, network, etc, and secondly perform the rapid build & deploy. Once the system is built, hotwire's auto-inventory keeps the inventory in-sync, while cfengine maintains them. Hotwire knows about the server hardware by talking to the SMBIOS/DMI data in the Bios via python-dmidecode.

The bonus points are that it is combines the inventory and build process into one, so there's less to manage, and the live-inventory feature is great as we know if something is not quite right.

The disadvantages are that the user-interface still needs polishing, and there are bugs here and there, but development is still hot, and bugs reported are fixed relatively fast.

cfengine

We use cfengine because other than it, and puppet, there is nothing else. It actually is a good tool, but "good" only as a function of how good your policies are - if you set dangerous policies, then a small mistake can cause alot of damage. For example, by policy, we don't "modify" files, we either replace them, or we don't. Also all files replaced have a header that makes any person editing it know that it will be replaced the next time it runs (it is run via cron hourly).

The configuration and all files pushed out by cfengine to the servers are also kept in a SCM, and using post-commit hooks, where possible, we check the syntax and if that fails, then the commit is rejected. This is easy for nice applications such as Apache, but not so easy for most enterprise applications.