Chef: encrypted data bags, protecting the encryption key

When you are using the encrypted data bag feature for Chef how do you go about deploying the key to many servers? If you put it into a recipe anyone who has access to any of the chef servers or clients can pull the key and potentially decrypt any of the databags.

How do you go about ensuring that the key is on the machines that need it, but also safe from anyone snooping around?

Solution 1:

Unfortunately there isn't really a lot you can do as the key needs to be somewhere on the Chef node in plain-text. If someone has shell or local access to the box then it might be possible for them to read the key(s).

To mitigate things a bit I add the following to my basenode (i.e some recipe or role that is common to all nodes):

directory "/etc/chef/keys" do
  mode 0700
  owner "root"
  group "root"
end

and whatever key distribution mechanism you have puts keys in that location. The permissions and ownership prevents reading of the keys if someone forgets to put the correct permissions on a key file.

As I see it encrypted data bags are more to protect key material from being readable in a source control system, and less as a security feature on Chef nodes themselves.

Solution 2:

My EDBs are separated by:

• environment
• role

We upload all keys with a special suffix to every new EC2 instance as we bootstrap it and then remove all keys that have not been used by any of the recipes in the run_list at the end of the first time chef-client runs (which will be right as the instance starts.)

All files are uploaded as owner and group "root" and with only read permissions.

Every recipe that uses an EDB, generates the EDB name and the key file name at recipe run time by concatenating 'edb_' + the nodes environment + recipe/item-specific name + '.key' and then looks for the key by this name. (If it doesn't exist this throws an exception by default.)

Thus, for our couchdb server, running a role called, 'couch', to get the credentials that we're using for the admins user(s) in the dev environment, the recipe looks for a key named 'edb_dev_couch.key'

It then search a data bag named 'edb_dev' for an item named 'couch_credentials'.

For managing keys, I'm currently using the simple approach of:

• upload all EDB keys via bootstrap script and append '_x' to the key names
• Have each recipe that uses an EDB look in the keys directory for the key that it needs.
• if the key exists with a '_x' suffix, rename the key to remove the '_x' suffix.
• add a recipe at the end of every run_list that deletes all keys with a '_x' suffix

Hopefully, this limits the time that the keys outside the scope of a single node are susceptible until the machine has been bootstrapped and had the first run of chef_client.

This is our first round of testing how to secure the keys, but so far it meets our current needs as it prevents one rooted dev server from being able to immediately access any other servers credentials that are stored in an EDB.

To maintain having one recipe at the end of every run list, I use a knife exec job that makes sure this delete_keys recipe is exactly the last recipe on every node.