Using LVM snapshots for KVM virtual machine clones

So, I've got a pretty awesome KVM setup running now, CentOS5.5 on the host and guest domains, libvirt managing all the configuration, etc. The guest domain filesystems are stored in LVM on top of a hardware RAID5 volume, so I have flexibility for backups and low-level data redundancy.

I tested out virt-clone today, and it worked amazingly well, except that it took about 30 minutes to replicate 24G of data from the suspended domain's LVM disks to the new LVM volumes for the new virtual machine.

My question is: couldn't I just use an LVM snapshot to create the new VM's root disk? For example: lvcreate -s guest1_root -n guest2_root -L 8G raid_vg

Now, my understanding of LVM snapshots is that the snapshot stores an inverse delta of the changes made to the original blocks so that the snapshot takes up little actual space and the original blocks can be read out even after the original volume has been written to. LVM2 adds read-write snapshots, which opens up this interesting possibility.

Indeed, the LVM HOWTO even suggests using this feature in conjunction with Xen:

This opens up many new possibilities that were not possible with LVM1's read-only snapshots. (...) It is also useful for creating volumes for use with Xen. You can create a disk image, then snapshot it and modify the snapshot for a particular domU instance. You can then create another snapshot of the original volume, and modify that one for a different domU instance. Since the only storage used by a snapshot is blocks that were changed on the origin or the snapshot, the majority of the volume is shared by the domU's.

This seems like a really powerful tool, and I'm wondering if anyone has tried it in their production virtualization environment, can think of any vendors (Citrix, VMWare) that take this approach, or can think of any serious problems with this idea. I can think of the following potential problems:

• Potential filesystem problems if the "original" guest is running at snapshot time.
• Host domain LVM performance as the "inverse delta" and "written block" logs are appended to. It may be very fast, but I don't know, and this seems like a weird idea, so it bears testing and benchmarking.
• More eggs in one basket: if the "original" LVM volume gets corrupted, then the snapshotted volumes are hosed as well. Of course, RAID5 mitigates disk corruption, but it's worth noting.
• Can you snapshot a snapshot? I assume so.

As people who likely have more virtualization experience than I, is there anything that makes want to run screaming from such a setup?

Solution 1:

It's perfectly fine to do this. What you do not want is to have the parent of the snapshot (the original, or the source, or whatever you want to call it) to be in use at the same time, because it will cause IO multiplication (Hubert was right about this, it is just easy to prevent by not using the source volume all the time).

If you have one master OS install on an LVM, and you snapshot that four times, you will not have much of an IO penalty, since you are only writing to the individual snapshot volumes. Of course, it is not free, but neither are other forms of snapshotting on other filesystems or virtual disks. There is always a cost somewhere.

One other thing Hubert is right about, is that you have to think about the sizing of your snapshots. You will want to make sure the snapshot volumes are able to keep writing. A full snapshot volume will break stuff badly. A failsafe way to prevent this, is to make the snapshot volume the same size (or bigger) than the source volume. You loose the benefit of using less diskspace this way, though.

You know that qemu images are snapshot capable too?

Solution 2:

While I didn't try LVM for KVM storage, I did use it for shadow volume feature of samba and I can tell you one thing: the performance was abysmal.

Each and every snapshot requires an additional write to happen. If you have one base snapshoted volume and 4 snapshots, the amount of writes that goes to the drives is multiplied by 5 when you write to base volume.

As for your specific questions:

• LVM freezes a filesystem while the snapshot takes place (stops writes, flushes cache, does the snapshot, resumes writes)
• as I said, it's very slow
• Yes, corrupted base volume renders all snapshots unusable, what's more, if you run out of space allocated for the snapshot deltas, the snapshot is hosed as well
• yes, you can snapshot a snapshot

Unfortunately, there are only 3 systems I know of that work well with snapshots: NetApp WAFL, ZFS and btrfs. If the system is not critical btrfs could be worth a shot.