Can enabling a RAID controller's writeback cache harm overall performance?

Solution 1:

Well, a basic dd is probably not the best way to measure drive throughput. It's not a realistic load. However, if you do run dd, please pass the oflag=direct flag in the command line to eliminate the effect of filesystem cache. Also see: How to measure disk throughput? for suggestions on how to measure workloads.

I think your scheduler choice is having a larger effect on your results than anything else. For RAID controllers with battery or flash-backed cache (write cache), I used to run with the deadline scheduler, but now use the noop scheduler if the cache is 512MB or 1GB. You can swap the scheduler on the fly, so try the tests with the noop algorithm and the oflag=direct and see how the results look.

Have you run bonnie++ or iozone?

Solution 2:

If you do plan on using iozone here are some ways to check your performance. These are better than dd as they allow the kind of test you're looking for.

iozone -s 4G -a -i 0 -i 1 -i 2

That will run tests with a 4GB dataset (-s 4G), using a variable record size and run the write-test (-i 0), the read-test (-i 1), and the random read/write test (-i 2). Selecting the file-size is critical. If you pick one that fits in RAM your results will be based more in file-cache than actual storage performance. So if you have server with 4GB of RAM, test with a file size larger then that.

However, if you have obscene amounts of RAM (I have one server with 12GB) and want your tests to finish in under many hours, you can supply the -I option, which tells iozone to set O_DIRECT and bypass the filesystem cache. You'll get your true storage sub-system performance there.

You can also do tests that check for concurrent access.

iozone -s 128M -r 4k -t 32 -i 0 -i 1 -i 2

That will run 32 concurrent 128MB threads running the same tests as the previous command but with a 4K record size (-r 4k). The working set is 4GB, but some of the files will fit in file-cache. Depending on what you're doing with this storage, this may be a more accurate test of your probable performance. As with before, the -I parameter will set O_DIRECT.

iozone -s 128M -r 4k -l 16 -u 32 -i 0 -i 1 -i 2

This does the same as the above command, but runs a series of tests starting with 16 threads and increasing to 32 threads.