How to keep secrets out of version control with kustomize?

Solution 1:

I've been checking on topic and I think that you might want to check overlays. An overlay is just another kustomization, referring to the base, and referring to patches to apply to that base.

This arrangement makes it easy to manage your configuration.

If we are speaking about some VCS, the base could have files from an upstream repository managed by someone else. The overlays could be in a repository you own. That is explained here.

As a result you won't need storing mariadb.env under version control. You can if you like, but that could be a local set of files as well.

Additionally, you might find interesting ideas in an overview of "Branch Structure Based Layout" (but that concep focuses on VCS and you told that you'd like to have only a part of your code in VCS).

Edit. Vault and auto restart pod after secret update

Solution 2:

The answer tends to involve encryption of the secrets. This is an overly simplistic so let's explore that further, because different solutions approach the problem quite differently.

There are (as my current research is telling me) largely two or three common tools for this:

• Sealed Secrets
• Helm-Secrets (which is Helm-specific, but uses Mozilla SOPS, which is just about secrets and YAML etc., and thus not Kubernetes specific)
• KSOPS (which is a Kustomize plugin to use SOPS)

I don't have any practical experience with these at this time, but I can tell you that one of the biggest differences is that with Sealed Secrets, the decryption is done inside the cluster at the time when the SealedSecret object enters the system. The keying material lives inside the cluster. The encrypted SealedSecret (a type of CRD) can live happily in Git with whatever confidence you have in the encryption. Developers don't require access to the private-key, but can't view the contents of the secret [without it].

Compare this to SOPS, whereby the decryption occurs on the client-side, and the secret will generally be maintained in some external vault, such as HashiCorp Vault, Amazon KMS, or similar.

Comparing the two, one significant aspect would appear to be the use of public CI/CD infrastructure, and which parties would need to handle your secrets; this is one area in which Sealed Secrets has an advantage, but there is pros and cons either way, particularly in terms of what access all/some of your developers might require.

Unsurprisingly, both SOPS (in one of its guises) and Sealed Secrets are both widely used, which explains why the likes of Kuberenetes (or even opinionated distributions such as OpenShift) don't come with any particular opinions with regard to how to do it; which is frustrating!

If you're familiar with Ansible Vault, you could use SOPS with a PGP key in a similar way, potentially alongside the use of other KMS solutions.