Should a KVM host be doing things other than hosting VMs? (such as ZFS)?

Solution 1:

Hypervisors, regardless of the flavor, should be 100% dedicated to serving its VMs. This is for a few reasons.

First, one of the key benefits of virtualization is hardware independence - if a host dies, it's very simple to restore those VMs to a different host and be up and running in a very short period. As soon as you start deploying services directly on the host OS, that process becomes much more complicated.

Second, hypervisor kernels are tuned to deal with virtualized load, not userland processes running directly on the host OS.

Third, process configuration and deployment becomes overly complicated if you have to consider processes running on VMs and processes running on bare metal. This isn't insurmountable, but it isn't ideal and should be avoided if possible.

So to answer your question directly: don't run any unnecessary processes directly on the host OS. Let the hypervisor do its thing, and use VMs for your services.