Best strategy to keep chef cookbooks versions under control

I'm looking for ideas about chef cookbooks versions management. I know that you pin specific versions in the environment, but I'm not sure how to go about it.

We use librarian-chef which installs 3rd party community books into the cookbooks folder. We never touch those books and just update to more recent versions from time to time.

We also have our custom site-specific cookbooks from where we include community ones (include_recipe).

Theoretically we could specify specific versions of community books our custom books depend on and then set our cookbook versions in the environment config, but the problem is that those community books could rely on some other books without specified versions. And that deep nested dependency could go on an on.

So there is no gurantee that when you upload cookbook to the chef server, it won't break prod, since dependent cookbooks might change as well.

The only solution I can see at the moment is to specify every single cookbook verion we use in the environment config, including community and custom ones. But then I have to go through each cookbook and figure out those versions.

We also do librarian-chef update for time to time and I imagine it might become difficult to track down versions that changed and not to forget to update version in the environmet when time comes.

Please, share your experience and best practices. I'm sure it will be super useful for other people.

Solution 1:

It wasn't long after I started using Chef in earnest that I was coming up against these same issues. I only came to some sense of sanity when I started doing four things operationally. Note that these may not be considered "best practices" by some in the Chef community. Nonetheless, this is how I brought sanity, repeatability, and order to my world.

1. Create your own recipes. I stopped using community cookbooks altogether and simply created my own recipes, to my specifications. This way I manage and control my own dependencies. Many would argue against this, but honestly - if I would have read some of the Opscode and community recipes first, I probably wouldn't have chosen Chef as my solution to begin with. I keep my recipes simple, and in line with my way of working. I have precisely zero community cookbooks in my repository.
2. Be disciplined about upgrades. If I update a recipe, I make sure it works everywhere and I go through the extra hassle of deploying it everywhere, even if it disrupts my workflow and adds friction. In the long run, this is the key to Chef sanity. In extreme cases, if I need variations for some hosts, such as a test vs production environment, then I code them into the cookbook. But my philosophy is that the most recent version of every cookbook should be able to be safely applied everywhere they are needed.
3. Use Chef Solo for everything. Every few months I somehow get it in my head that I should try using Chef Server again. The community edition is improving, but the whole paradigm never seems to fit my world. And every time I try, I facepalm and kick myself. The Chef Server paradigm is suited for a world with long-living servers that need frequent systems changes. I make systems changes so rarely that having my servers constantly check in to a chef server for updates is just silly. And I have far better tools to ensure that my hosts are healthy. My work is in a world of disposable virtual machines, where they may only ever survive one or two configuration changes. I now use Chef Solo exclusively, and push changes to my hosts, while also pushing the exact same cookbooks to all the hosts that need them.
4. Avoid compiling software during Chef runs. The most extreme (ie stupid) case for me involved compiling ruby-1.9.3 from source every time I bootstrapped a new box. But creating custom packages can often be a pain in the butt. Once I discovered the excellent tool fpm, it became trivial to package up my own rpms, debs, and gems and made my life much more efficient and easy.

Hope this helps someone!

- UPDATE -

Nearly three years later, these principles have remained helpful to me. But I'll add one more piece of advice, and it's really for the same reasons I preferred chef-solo over chef.

5. USE ANSIBLE INSTEAD

Solution 2:

There're 2 problems:

1. manage cookbook versions in different environment objects
2. manage recipe version in node run_list.

The article Essentials of cookbook versions is the best reference for cookbook versions. According to #1, you're right because it's a tough job to manage different version of cookbooks to serve different configuration set especially it's mixed with cookbook dependencies where most of the cookbook from cookbook site doesn't do this job well. So configuration may break. and if you didn't manage the versions by testing the runtime behavior of any component, it just breaks. So it's a bad idea to upload cookbook without specifying the version number in you environment object. So manage the cookbook versions in environment object, and carefully test when promoting version of any new cookbook. I usually manage the environment object in SCM and didn't upload to chef server via automated job until the changed cookbook can work well with rest of other existing component.

According to #2, it's a tricky topic because this is where the actual recipe dependency works on each node. In a nutshell, for critical nodes, you'd better control the dependency of recipes by specifying recipe version in node/role run list. I hardly do this because it's fine grit control and cost more on testing/promoting. However, for critical role/node, this is not a bad idea but provide insurance on configuration changes.