KVM on Illumos (OpenIndiana vs. SmartOS)
I'm considering virtualizing a number of guests onto a single server running a recent port of KVM to Illumos. It sounds like my two primary options will be OpenIndiana and SmartOS. The distribution I will end up using needs to meet the following requirements:
- Need to be able to manage and customize via CLI (e.g. change ZFS filesystem/zvol options, attach an external drive and copy data to it, or automatically replicate data to an offsite server using zfs send/receive).
- Need to implement automated ZFS snapshots (e.g. using zfs-auto-snapshot).
- Need to be able to setup automatic email notifications if the health of the server degrades. Essentially, setup periodic ZFS scrubbing, monitor zpool, fault manager, and/or SMART issues, and email when problems are detected, etc. Manually setting this up is OK, as long as the OS lets me.
- Should handle Debian, Ubuntu, and Windows 2008 Server R2 guests with good stability and reasonable performance. These guests will be used in production.
- There should be a reasonable expectation that future releases will continue to be delivered - I don't want to get stuck using a dead-end product.
- Would be nice if it was easy to setup and has some sort of GUI, but this is optional.
Based on these requirements, which distribution would you recommend?
You can assume that this environment won't be deployed until the upcoming OpenIndiana stable release is released. Also, you can assume the server will use a Sandy Bridge Xeon E3-1xxx CPU, so that should take care of KVM compatibility.
Also, how robust/stable is the KVM port to Illumos on either of these distributions? Should I even consider KVM/Illumos for a production environment for now?
Solution 1:
I've been using SmartOS and KVM in production for a few months now and am very happy with it. It sounds like it should suit your needs just fine. All the ZFS stuff you require is supported. For the monitoring stuff you'll need setup up some third party stuff though.
I'm working a couple of projects related to monitoring and specifically for doing the things you mentioned. Check them out and feel free to drop me a line.
https://github.com/gflarity/nervous https://github.com/gflarity/response
Solution 2:
I'll ask, how important is it that you specifically use KVM?
My preference for the type of solution you're inquiring about is to build around VMWare ESXi. You can build an all-in-one server running VMWare ESXi booting from flash media (SDHC, USB, CF) and leverage the DirectPath I/O (PCI-passthrough) available on current servers to present a SAS/SATA HBA to a virtualized ZFS-based OS (let's assume OpenIndiana, but I usually use NexentaStor Community Edition). From there, you can create a loopback vSwitch and present your ZFS storage to ESXi as 10GbE NFS or iSCSI to house the guest virtual machines (Windows, Linux, etc.).
Using this, you have full access to ZFS features like compression, deduplication and snapshots. You can augment this setup with a ZIL and L2ARC quite easily.
If you choose NexentaStor for your ZFS solution, you'll also have a full GUI to manage autosnapshots/tiering. The monitoring tools for the disks are also built-in.
VMWare handles a number of guest types very well, so you're covered well.
Nexenta, OpenIndiana and VMWare are here to stay, so this isn't a poor technology decision.
Provided you have the budget for hardware, your ESXi, ZFS OS and Linux are free...
Also see:
http://blog.laspina.ca/ubiquitous/encapsulating-vt-d-accelerated-zfs-storage-within-esxi
http://www.napp-it.org/napp-it/all-in-one/index_en.html