The state of XEN vs KVM for inhouse virtual server farm like live farm

so we have a life server Farm, conventional stuff not virtualized. This is to stay that way for the time beeing. The servers we are talking about are: Loadbalancer, DB master, DB Slaves (2x), Webservers (2x), CMS server. All in all 7 servers per farm.

We want to have one to many rather exact "copies" of the life structure for inhouse purposes.

• Testing/Staging: to find bugs before live that might slip through if we do not use the same software-configuration
• Debugging/Benchmarking: Same like testing but with additional tools. debug flags and isolated from Tetsing/Staging to not interfeer with QA. Intended to get a deeper insight into how things are on our live-systems
• Development/Experimental: Changing System Components, Software-Versions, Libraries and Configuration to improve performance, developer effectivness, future-proofing our systems etc.

All in all 3 virtual "farms" a 7 virtual servers each, neatly isolated.

Now I know this can be achieved with many different flavors of virtualization. The Questions is, what is the best? VMware not beeing open source is nothing we would like to use on a strategic point of view. Looking at XEN and KVM, the two biggest players in open source virtualization I would love to hear some advise on what/how to select. The Web seems undecided.

Additional Info:

• "life time" of the server hardware that we purchase will be 2-3 years
• There is the possibility that we might migrate systems into cloud-environments later, if that is a factor to consider in the selection of the virtualization-technology
• Isolation of Server-Farms/Servers in High-Load scenarios is important. QA should not suffer when someone wrecks havoc in the experimental farm
• Efficent ressource usage is of course appreciated (memory overcommitment/shared pages? Automaticly Shared objects on hard-drive like the linux-vserver hashify?)
• Regular maintenance requierements and easy of management

I know things are in the flow but I would appreciate if you would tell me your opinion on what to choose right now seeing we want to live with the technique for at least 3 years, build up and reuse know-how. Also maybe in the end we can all realize that neither one, XEN or KVM is deciding the battle but other factors. Enlightenment in that regards is even more valuable to us right now.

p.s.: and lets not flame :D

Solution 1:

We use Xen currently, but I think that in 2011 we will be migrating to KVM. There are some reasons why:

• KVM development is more integrated with the Linux kernel than Xen's development.
• KVM vms run as processes to the linux kernel. That has many implications, like scheduling, memory management and etc. That also permits KVM to do overcommit of memory (it's simply swapping the vm/process memory) and all that is using tested and proven code from the kernel. Xen uses it's own code to do all that, and while it's not bad, it's not as tested and proven as the linux kernel.
• Full virt seems to work better on KVM.

The main point against KVM is performance, but last reports using VirtIO driver on Linux and Windows VMs seems to make that point less and less proehminent.

As for managing the machines, I use ganeti. Ganeti is a cluster virtual server management system where you add your nodes and can do all the operations with the VMs on those nodes like creating, starting, rebooting, migrating and etc. It also supports the creation of DRBD instances that have mirrored disk images on two nodes providing automatic failover/migration if one of the nodes fail. It supports KVM or Xen, but not mixed clusters. Ganeti is text-based but there's a web interface project that is doing pretty well. Using ganeti + debootstrap we can deploy/clone various types of vms very fast, do LVM snapshots from disks for tests and so on, so I think it will get you covered on that.

Remember only that no matter what one you choose, you should never virtualise high IO machines like file servers or DBs and expect they will perform the same. In some cases, high IO VMs can even degrade all other VMs on a host. Not everything can be safely virtualised.

EDIT: Since you mentioned testing and development, read this article. I am thinking in assembling something very close to that.