Replace VMware vSphere infrastructure with open source alternatives?

Having recently gone through the same song and dance with my own management ("VMWare is really expensive! Check out what OSS options there are out there.") I have some observations to share.

• I/O performance does vary by hypervisor, though CPU performance is much less variant.
  • In general, 'thin' provisioning is a great way to dock I/O performance. Some (KVM until very recently) are pretty bad about this.
• The big three non-VMware alternatives (Xen, KVM, Hyper-V) all have some kind of vMotion-like technology, though there are limits.
• Some are very sensitive to CPU architectures, and don't allow live-migrations to systems that aren't identical. VSphere gets around this with their "Enhanced vMotion Compatibility" technology that dumbs a cluster down to the lowest CPU architecture in the cluster. Not everything else has something like that. This can be a big barrier to expanding your VM plant.
• A Hypervisor is only as good as its management console.

That last point is the big one. It's all well and good to have 150 KVM instances, but without some kind of automation to move machines around it doesn't do you much good. There are many, many OSS and non-OSS orchestration frameworks out there, a lot of them built on Libvirt. Once you've found a hypervisor that works the way you'd like it to, you'll probably spent just as much time if not longer evaluating management frameworks for something that works the way you need it to.

I've been impressed with CloudStack. It was recently purchased by Citrix, but it's an OSS management framework that (as of a couple months ago at least) has a few features only found in paid frameworks. That said, you do tend to get a much more polished framework when you pay for it; CloudStack is under active development so is rapidly changing.

I haven't used it myself but there is Proxmox-VE. According to Hak5, it supports a variety of OS's including Windows. It also supports clustering