Manage chef cookbooks in a team environment

Solution 1:

I work with multiple projects, so cjc's solution won't work for me. There's also an issue of common vs custom configuration (addresses etc are common to the company, there's also a bit of magic in the configs). The scheme I finally settled on is a bit of a hack, but it's a convenient-to-use one.

Instead of global ~/.chef, I use '.chef' subdirectory within chef-repo, which is not stored in git (it's added to .gitignore). I have also file config/knife.rb file which is checked into Git and contains shared configuration. It starts with this snippet:

root_dir = File.join(File.dirname(__FILE__), '..')
%w(knife-secrets.rb knife-local.rb).each do |conf_name|
  conf = File.join(root_dir, ".chef", conf_name)
  Kernel::load(conf) if File.exists? conf
end

This loads files .chef/knife-local.rb containing custom configuration (in basic version it's just OPSCODE_USER='username' constant that is used later on, but it can contain any knife config), and .chef/knife-secrets.rb which contains shared secrets (AWS keys etc).

Below that, there is regular knife config that uses constants defined in these files, e.g.:

client_key               "#{root_dir}/.chef/#{OPSCODE_USER}.pem"

This way, I achieve standarization of knife config across the company, which in turn means that any code snippet or knife invocation shared in a wiki will work for everyone. There is enough confusion and magic in knife itself - different configurations would only make it worse. Also, everyone gets benefit of small magic snippets, like this one to make knife ssh use login configured in user's ~/.ssh/config

There's also issue of shared secrets: chef server's validation key, AWS keys stored in knife-secrets.rb, EC2's SSH private key, encrypted data bag keys, and so on. We definitely don't want them to be stored in the repository - or, actually, anywhere where they aren't safely encrypted. So we distribute those files as a .tar.gz file, which is GPG-encrypted to everyone in the company, and shared over Dropbox.

Configuring all this is getting complicated, and I want people on team to actually use the thing, so there's the final element: rake init task which creates .chef directory, symlinks config/knife.rb there, decrypts and untars chef-secrets.tgz file, makes sure user's private Opscode Platform key is there and .chef/knife-local.rb is properly configured, symlinks knife plugins, and sets proper permissions on the directory and files inside. This task is set up so that it's safe to run it many times on already initialized repository (e.g. to update secrets or knife plugins).

There's also a helper task that repacks all the secrets, encrypts the tarball to everyone, and copies it into dropbox, to make it easier to add new employees or change secrets.

Regarding multiple environments: Chef has a feature called environments. I haven't used it yet, but it should do what you need. You can also strictly separate production environment (to avoid developers having any keys that relate in any way to production env) by having two separate Hosted Chef organizations or Chef servers. This knife.rb snippet show how to configure knife in a different way based on currently checked out branch - you can use this to set environment as well as chef server's url. There's also knife plugin called knife-flow , providing more complete, two-organization workflow.

Solution 2:

You need to set up two Chef servers, one for production and one for development. The reason is that no single Chef server can support branched development; even with environments.

Or you can abandon the Chef server concept and use chef-solo. You can maintain your cookbooks in Git. You can branch & merge. You can ignore the problems with knife credentials because you won't use them any more.

You won't be able to use knife search or data bags**. But some people don't need those features anyway.

** Well you sort-of can: http://wiki.opscode.com/display/chef/Data+Bags#DataBags-UsingDataBagswithChefSolo

Solution 3:

I have two directories in my home, .chef and chef-repo. The chef-repo is in git. The .chef is some private directory that's the default for knife. You don't have to put your secrets from .chef into git; knife will look for ~/.chef.

Solution 4:

Your ~/.chef directory should not be in the git repo.

I have a ~/projects/ directory under which I keep my chef repo. In this goes my servers' configs.

My last job was as a Systems Engineer at at Ruby-on-Rails shop. Our nginx, varnish, and rails configs (among others) go in the chef repo, but the Rails apps themselves were kept in separate git repos, and were deployed separately.

Our Staging environment was a single server that ran the entire staging environment. This was not ideal because it didn't resemble Production where Rails thins and the DB were on separate boxes. What I would recommend is using Chef's Environments to separate Staging and Production. (That's how it was when I got there, and I just disn't have the time to fix that before I left.)