Software RAID underneath ESXi datastore

Solution 1:

ESXi is not a general-purpose OS and shouldn't be considered so - stick absolutely to the Hardware Compatibility List and therefore use a tested and approved hardware RAID adapter. Choose to do anything else and you'll join the swathes of others who cut corners and end up back here complaining that their systems don't work or they've lost data - we get LOTS of those.

Solution 2:

This does seem kind of round-about, do I have any better options?

Your general idea is spot-on. I would personally suggest using ZFS with Solaris or FreeBSD, but mdadm might also work. Maybe though you don't get all the advantages I write about in this post, so take that as a disclaimer. This post will be quite long, I apologize in advance for the wall of text.

From my research, passthrough seems to come with quite a few drawbacks, such as no suspend/resume etc.

There are some, notably:

1. Only works with vt-d (or AMD IOMMU) support on CPU (Intel) or CPU+board (AMD): no problem, it is hard to get a server without it today, as every Intel CPU except the Atoms have it (even in the basic systems from HP, Dell and others)
2. VMware-snapshots of all VMs with passed-through elements cannot be created: in theory a problem, but this affects only your storage VM, which has minimal configuration setup on it and is used for nothing else. You can do your snapshots at the storage level, where they are faster, cheaper and do not slow down the system. Additionally you can snapshot all other VMs on the host without problems (and even combine it with filesystem snapshots for powerful restore options and longtime archival of states).
3. Your internal complexity is increased in some areas: At first sight, this is true - you add an additional layer and you need to manage more stuff, like your new internal SAN (network/VLAN setup in VMware) or your storage VM itself (updates etc). But on the other hand you also have simplicity and flexibility:
   • Consistent backups of running virtual machines can be created automatically with a few simple scripts and are completely free. They can also be stored on another machine, disk or cloud (encrypted), all without any additional expensive software solutions.
   • In case your server dies, just buy another off-the-shelf replacement, install ESXi, enable passthrough, configure your network, add your disks and boot your storage VM. After it is up, rescan your storage and it is as if there was just a power failure, all your data is safe and you know it (instead with HW RAID, you hope it is).
   • Special requirements can be met with minimal change as need arises. The business has a legacy application that requires local disks for backups? Just configure iSCSI and present your storage transparently. They experience growth and need more storage? Just grow the pool with more disks and present them either directly over iSCSI or via VMware (NFS or iSCSI with vmdk on top). They want to use a database on a beefy separate server? Just open up your NFS on another LAN/VLAN and supply it to the new server as a "real" SAN.
4. GPU passthrough works only for expensive Nvidia cards and all AMD cards: This is currently true, but your storage VM does not need a dedicated GPU in any case.

There are also general annoyances unrelated to passthrough:

1. To reboot the storage VM, all other VMs that depend on it need to be shutdown first: This obvious problem is seldom talked about, but in my eyes the most annoying. Of course, updates to ESXi itself also require a full reboot, but as you now have two systems, the times may not sync perfectly. I recommend a stable operating system and lining up non-critical updates between both systems. Additionally you should limit the storage VM to its own internal virtual LAN, further reducing the need for applying fixes as soon as they are released. Note that this also applies to accidental reboots of the storage VM from the GUI.
2. Errors in the underlying stack render your whole machine inoperable: This risk is increased in comparison to ESXi only, because now you have two systems and two network stacks between them. On the bright side, both your storage VM and ESXi should normally be stable and errors should be few. Nonetheless, I advise to schedule updates some days/weeks after release so that you can see if others have experienced problems. Not changing the configuration on the other hand means that it is very stable, which is a plus for SME (less support needed).
3. Solution is not known to 3rd party support personnel: It is a quite rare setup, so the chance that your random replacement can figure it out without your documentation from the start is slim - this may be a problem or an advantage, depending on your business goals. It can be mitigated by some basic documentation explaining the structure of the setup (use pictures/diagrams!), comparison to a traditional RAID setup, and what to do in common problem cases (backup, restore, disk replacement, updates, network changes, hardware expansion).

Technical issues aside, you have to think about your goals and the ways to achieve them. This determines the practicability of your chosen solution along with its upsides and downsides, and your overall outcome (success or failure). This is something that largely depends on the needs of the business itself. Some considerations for or against your proposed solution from a business perspective:

• Budget: There are businesses that can justify to pay several thousand dollars a year for a support contract that is used almost never, because the one time it is used, it is worth much more to them. There are businesses that can only pay for immediate value, and that can cope with unexpected downtime quite cheaply/flexible, so the money would be wasted.
• Safety requirements: There are businesses where a single destroyed or damaged file from 10 years ago is not an option, and others where even a complete loss of all current data is only an inconvenience. The needs for backups, snapshots, etc. change dependent on this.
• Support structure: There are businesses that want to buy a machine, set it up and then run it (virtually) forever without any support needs at all, and others that want and need continuous change, upgrades and direct support from people they trust.
• Flexibility requirements: There are businesses that change so rapidly that you could not see their needs in advance, which may or may not be an argument for a more flexible setup. On the other hand there are those that never change and value stability and predictability over anything else.

You should keep those points in mind in any case. Your solution can only be successful if you meet the goals, it does not matter what is the choice of the majority, only what is a) technically possible to implement, b) within the budget, c) meeting the goals of the business. If all those points are met and you still have to decide, choose the easier/less complicated solution (KISS). If they are equally easy, decide for the one that brings you more money and/or happiness.

Solution 3:

I would fire you if I were a small business and you deployed something like this... This is a common theme, though. VMware has a well-defined hardware compatibility list. However, when used as a standalone server, you NEED hardware RAID. Non-RAIDed disks will work as well, but that's not what you want. So my questions:

• Not enough funds for storage? What type of server hardware is this? You can afford disks but no RAID controller? A compatible RAID controller is not expensive.
• Isn't this a case of managing customer expectations? Separate storage would clearly be more expensive than hardware RAID.
• While all-in-one storage solutions are possible, they are best for specific technical requirements, not cost-cutting.

It's a case of VMware-abuse. Software RAID is not supported. I would go back to the customer and revise the build/requirements.

"How much is your data worth?"

Solution 4:

Check out this blog for an example of using ZFS as a backing store http://blog.laspina.ca/ubiquitous/encapsulating-vt-d-accelerated-zfs-storage-within-esxi

That being said the warnings about using standard hardware are valid. ESXi is quite picky.