Encrypted storage for clustered MySQL

I have a requirement to provide a highly available MySQL database with encryption-at-rest on a 2.6.32 linux kernel. The 'highly available' part isn't that hard, but the 'with encryption-at-rest' is proving to be a challenge when used in combination with HA.

The key problem comes with mounting the encrypted storage. On all of our other encrypted-at-rest systems there is a command that needs to be run by a human, who is then prompted for the encryption key. This model has a fairly obvious flaw when it comes to a cluster arrangement where services will have to start automatically.

I am currently at a loss over how to provide encryption-at-rest in an HA environment and not also store key passphrases on the same system.

I can see two possible scenarios, either of which will work with my environment but I'm not sure of the details to make them work. Or even if it is possible.

Scenario 1: CLVM & Cluster

• A volume is shared between my cluster members.
• This volume is set up roughly:
  • cryptsetup stuff on the physical device
  • LVM stuff on the new crypt-device
• Cluster Services are set to not join the cluster automatically, relying on a manual intervention.
• Cluster Services are started via command run by a human, which supplies the decryption key, which in turn activates the CLVM stuff.

This way running nodes have access to the CLVM volume so they can start services when told to by the cluster manager. Reboots of nodes will still require a human, and the crypt passphrase is never kept on disk anywhere.

Scenario 2: DRBD & Cluster

• A volume is created on each cluster member
• cryptsetup stuff is run on the physical device
• drbd is configured on top of the crypted device, to replicate it between each node
• LVM or a filesystem is placed on top of the drbd volume
• Cluster Services are set to not join the cluster automatically, relying on a manual intervention.
• Cluster services are started by a human who provides the decryption key, which in turn makes the LVM (or filesystem) visible-but-not-mounted.

As with the CLVM setup, nodes don't join the cluster until they have visibility into the presumably-shared storage.

The thing is, I'm not sure if either of the above work that way. Both assume it's possible to layer an LVM PV on top of an encrypted volume (e.g. pvcreate /dev/mapper/cryptmysql). This may not be possible.

Solution 1:

The main challenge seems to be the human intervention for key entry. There is some help to that: dm-crypt has support for the TPM which might be available with your platform. See this IBM blueprint for configuration details. Also LUKS/cryptsetup supports reading a slot key from file / from stdin. If you can store your key safely somewhere (like on a smartcard), this might be a viable option.

As for your question whether you can have an LVM PV on a dm-crypt volume: you can, you just would need a run of pvscan / vgchange -a -y after slot unlocking. We've run this kind of setup with much older Kernels a couple of years ago. In the end, we've abandoned it in favor of the use of SEDs for applications with data-at-rest encryption requirements due to performance reasons (dm-crypt used to employ a single thread per encrypted device at that time, which led to a CPU bottleneck in our setup).

Solution 2:

It turns out this is entirely doable with LVM as syneticon-dj suggested. Since then I've verified that it works on cluster configs. However, doing so is not as easy as you'd think.

Before the cLVM volume groups are visible, it has to be decrypted via cryptsetup luksOpen on the device needing encryption. This by necessity happens after the cluster services have started, so any resources based on it shouldn't be a member of anything critical such as a stonith device.

Setting up the cluster is the same as it always is, but a few differences:

• It's a good idea to have a second storage device in the cluster that is NOT encrypted, which could be used by stonith (external/sbd type).
• Create the clvm resource when all nodes have the crypted volume opened via the same cryptsetup command and mapped to the same device-name (e.g. cryptsetup luksOpen /dev/sdd cryptmysql)
• Create the clustered volume group on the device created by cryptsetup (e.g. vgcreate ClusterMySQLVG /dev/mapper/cryptmysql)
• Create a simple bash-script to run the decryption the same way every time, this will be used by an operator post-reboot to get the volume accessible.

Because a node requires manual intervention to be eligible for fail-over, it's a good idea to have more than two nodes in the failover cluster.

Once the volume-group and logical volumes are created as normal, install MySQL. At this point the installation runs as per normal clustering installations.

When a node restarts:

1. Cluster services will start on boot, and the node will join the cluster. However, as it doesn't have visibility to the MySQL volume, it will not be eligible for fail-over.
2. An operator connects to the node via ssh, and runs the decrypt script created above, which prompts for the decryption key.
3. The script runs and creates the needed mapping in /dev/mapper, which LVM then picks up.
4. The LVM service should automatically refresh and see the new volume group metadata.
5. At this point cluster services will see that the MySQL volume group is available, and the node will be eligible for fail-over.