ZFS L2ARC for Mirror Pool

Solution 1:

Your pool black currently consists of two mirrored WD10JFCX drives, which according to Western Digital's spec sheet are:

• 5400 rpm (rotational latency about 5 ms on average, 11 ms worst case)
• 16 MB cache
• 1 TB
• 2.5" form factor
• SATA 6 Gb/s (SATA 3)

Neither the 5400 rpm rotation speed nor the puny 16 MB on-drive cache is conducive to highest performance, and the 1 TB is pretty small by today's standards as SATA drives go. 5400 rpm gives you, in theory, about 90 IOPS to the platters, and assuming that on average the platter will be half a revolution away from the data, its 11 ms rotation time gives you an average about 5 ms rotational latency.

Compare to this that you can get the Intel 540S series for about $308 apiece from Amazon (in the US), for which Intel gives the specs as:

• Up to 78,000 IOPS doing random reads, 85,000 IOPS doing random writes (latency 50 µs)
• Up to 560 MB/s sequential read, 480 MB/s sequential write
• 1 TB
• 2.5" form factor
• SATA 6 Gb/s (SATA 3)

Even if we play nice, you are looking at a 1,000-fold reduction in latency and a 1,000-fold increase in IOPS for about $600 and change. On a performance per dollar ratio, I'm pretty sure there's no beating that by adding L2ARC to your pool, and it's a drop in replacement. If you are cash-strapped, you can even replace one drive at a time with a corresponding SSD and still start reaping the benefits immediately.

Replacing those 5400 rpm disks with SSDs will do more to your black pool performance than any L2ARC you can realistically throw at it, almost regardless of your workload. What's more, while the Intel 540S series may or may not be a good fit for your requirements (see also here), even such a cheap SSD is almost certain to provide a better return on investment as compared to simply adding a L2ARC. If you want something higher spec'd, looking just at Intel, a DC S3500 1.2 TB SSD will set you back about $900 apiece, and a DC S3710 1.2 TB SSD will set you back about $1,000 apiece, both of which have similar performance but are likely to hold up better to use over time.

No endorsement of any specific products or resellers mentioned is either expressed or implied.

Solution 2:

The general advice for this is to maximize your system RAM (ARC), and then add L2ARC if the need is there.

This can be measured using the arcstat command in your server.

Something like:

arcstat.py -f "time,read,hit%,hits,miss%,miss,arcsz,c" 1

Where the output shows the time number of reads/second, the hit ratio, the number of hits/misses, and the present and target ARC size:

    time  read  hit%  hits  miss%  miss  arcsz     c
15:15:21  1.5K    99  1.5K      0     2    55G   55G
15:15:22  221K    99  220K      0   584    55G   55G
15:15:23  245K    99  245K      0   544    55G   55G
15:15:24  262K    99  262K      0   551    55G   55G
15:15:25  321K    99  321K      0   489    55G   55G
15:15:26  262K    99  261K      0   479    55G   55G
15:15:27  203K    99  203K      0   585    55G   55G
15:15:28  207K    97  202K      2  5.3K    55G   55G