ZFS: best practices for adding cache to pools

Solution 1:

First of all, for general ZFS best practices the Solaris Internals wiki is quite useful: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

If you want, you can create a separate ZFS pool composed entirely of SSDs. However, this will not improve the performance of any other pools. A more cost-effective approach for most workloads is to use the hybrid storage pool feature; that is, adding SSDs to normal spinning disk in order to improve read and/or write performance.

Key points:

• SSDs can be added in to an existing pool to improve the performance of that pool only. They don't have different names - they are just devices within the pool. You can add multiple cache and log devices to the pool depending on your performance requirements.

• To improve the read performance of the pool, assign SSDs as cache devices ("L2ARC").

• To improve the write performance of the pool, assign SSDs as log devices ("ZIL").

The L2ARC is a read cache that fills up over time and stores data based on a combination of which blocks are most frequently used and most recently used. This provides fast access to "hot" data. It gives almost the speed benefit of RAM at a lower cost per gigabyte.

The ZIL is an intent-log - adding SSDs allows synchronous writes to happen at high IOPS and very low latency, which improves application performance in some cases (e.g. databases and NFS shares).

Solution 2:

If you are going to be deploying NFS, a ZIL device or devices may be more important than adding cache devices. Instead of cache devices you can increase the amount of RAM in the system, and instead spend more on write-optimized small capacity SSDs for ZIL. At Nexenta, we always recommend mirroring your ZIL. Not doing so does expose you to a very real possibility of data corruption under certain conditions. With cache devices you would actually look at read optimized SSDs, and no real need for mirroring, because all data and metadata could be discarded at any moment without any risk to the rest of the pool.