How can I manage multiple administrators with juju?

I manage some deployments with juju. However I am not an island, I have coworkers who also want to manage shared environments.

I know I can use the following stanza in ~/.juju/environments.yaml to give people access to my juju environment:

authorized-keys: [and then put their ssh IDs in here]

What other best practices are available to manage multiple environments with multiple system administrators?

Solution 1:

Juju is really not at this time optimized for multiple admins. In particular, some of the security issues that are currently seen in Juju become more pronounced when working with multiple admins. However, with some caveats, this can still remain a useful option for a trusted - and thus likely small - group of admins.

With that in mind, let's look at the relevant configuration items in the ~/.juju/environment.yaml file.

The authorized-keys item is used in the environment bootstrap to define the public SSH keys for the ubuntu user on the bootstrap node (machine 0, on which ZooKeeper runs) and all subsequently provisioned nodes. Simply list authorized one public key per line. It will look like the following:

authorized-keys: |
    ssh-rsa AAAblahblahZZZZ user@domain
    ecdsa-sha2-nistp256 AAAAfoobarZZZZ= user2@domain

This approach is preferrable to using authorized-keys-path - or its default (~/.ssh/) - which is only suitable for one admin, given that you would then have to share SSH keys (don't do that!). Simply put, this is just a limitation of how authorized-keys-path is implemented.

Next, cloud providers define their specific security credentials, which Juju then uses. By way of example, let's take a look at EC2, specifically AWS. For AWS two keys are used to determine access: access-key and secret-key, corresponding to environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (these are the defaults if not specified in the config file). The challenge here for a multiple admin environment - or even the single admin environment - is that this information is copied to ZooKeeper and is readily visible in the ZK node /environment. See Juju bug #907094.

In the AWS example, some degree of control can be provided on your access keys through the use of AWS Identity and Access Management FAQ (search for this), but currently there is no mechanism to provide more fine-grained control to a given admin in a Juju environment.

One additional pattern seen in our own usage, especially in testing, is worth noting: a Juju environment ("control") set up on a given node to control other Juju environments; simply deploy the juju charm to have this set up. It takes as a config option the environment.yaml to be used. Additional admins can then be later authorized as the ubuntu user by manually adding their keys to ~ubuntu/.ssh/authorized-keys.

This allows for a single point to manage some of these concerns, while minimizing the headache. It does nothing to address some of the security concerns mentioned earlier - you still must have a reasonably deep trust of your other admins.

Juju will resync environments.yaml to the ZK node /environment upon the use of certain commands (juju add-unit, juju constraints-get, juju constraints-set, juju deploy, juju terminate-machine). In practice, this will not be terribly useful - except for its real intended usage, of supporting constraints. However, it can help minimize the need to update authorized-keys files as any newly provisioned machines after the sync will get that; you would then just be responsible for updating earlier provisioned machines.

While I'm on this point, it is worth noting how and where in ~ubuntu/.ssh/authorized-keys are actually used:

• All Juju control commands, with two exceptions, use a SSH tunnel to the ZooKeepeer instance to manage or lookup info from the Juju environment. (juju bootstrap and juju destroy-environment directly work with the underlying cloud provider API.) So you definitely need to keep the authorized-keys on machine 0 current.

• juju ssh and juju scp enable working with a machine directly, and they also need to have current authorized-keys that you will have to consider updating in the multiple admin case. These commands by default use the ubuntu user on the target machine.