How to manage Xen virtual machines on shared SAN storage?

We have a small cluster of Xen servers in the office. Each virtual machine system image is on its own block device on our iSCSI SAN. The servers all connect their iSCSI initiator to every VM LUN simultaneously.

To prevent the same VM from being started on multiple hosts we just ensure that each virtual machine is only added to one server through the use of xm new. Judicious monitoring of all the servers through virt-manager helps as well. However, there really there are no safeguards that would prevent someone from accidentally launching the same VM on multiple hosts.

My question is, what would be a good way to implement a better management system for a setup like this? Are there any existing tools that will manage a Xen setup like this with a shared storage back-end?

We're using openSUSE 11.1 as the host OS.

Xen has a built-in locking mechanism that should be used, especially if using hot or live migration. We have a NFS share mounted on all of our Dom0 hosts, and Xen checks there first to see if someone else is running an instance of a DomU before attempting to start it.

Look in your xend-config.sxp file -- the config settings your looking for start at about line 261 on my system, and are called xend-domain-lock, xend-domain-lock-path, and if you want to set up an external utility to manage it in another way, you can specify an external command to run with xend-domain-lock-utility. I'm using Xen 3.3.1 on SLES11; your mileage may vary.

We manage xend, xendomains, and nfs with pacemaker/openais, and store the lock files on an NFS mount that is made highly available by means of pacemaker and a partition on the san.

You have quite a few management tools for Xen clusters :