How can I benchmark Linux file systems on a SAN?

Being an iozone fan, I've been using it for benchmarking on Linux and Windows systems for years. Sean has the key point, test with a dataset that won't fix in RAM + Cache. IOZone makes this pretty easy.

iozone -s 64G -r 16k

Test with a 64G dataset and 16K read sizes. You can specify -r multiple times to give it a range of I/O operation sizes. You can even specify individual tests:

iozone -s 32G -s 64G -r 8k -r 16k -r 32k -r 64k -i 1 -i 2

-i 1 is required since that creates the dataset, but -i 2 tells it to also run the random-read and random-write tests. There are a couple of other tests it can run. An interesting test is the 'stride read' test, which skips a number of slices between reads; correctly configuring this can test the limits of read-ahead as well as RAID stripe alignments.

It can also use Direct IO as part of tests, if that's important for you. Some DBMSs use DirectIO, which bypasses the Linux caching system:

iozone -s 8G -r 1k -I

It even has a mode that'll test multiple files simultaneously. This is handy for testing cases where some files may fit in cache, but not all.

iozone -t 32 -s 2G -r 8k -r 16k

This tells it to use 32 threads, each with its own 2GB file, and test various record sizes.

One thing that I've seen a few times is when I test a record size the same size as my RAID stripe width. Frequently that'll be a slower access than the record sizes on either side of it. That's a sign of a misaligned partition.

When benchmarking it is typical to try to "blow out the cache" by using a data-set at least twice the size of the RAM+cache. This can help get more worst-case performance numbers, but really help you with realistic numbers.

Sadly, for realistic information about performance, you really don't have much option but to create something that will simulate your specific use-case with your specific data-set. Ideally you'd also want to age the file-system before running this benchmark, by loading it with data simulating normal use over time. A freshly "mkfs"ed file-system may respond quite differently from one that has a bunch of other data and has files created and deleted in directories.

In other words, if this system is going to be a web server, load up your pages and data and applications and get a reasonably representative set of URLs to run through siege or ab. If it's a database server, load up a production database and run your typical queries against it, etc...

In my experience, that's really the only way to get realistic numbers about the performance.

However, as far as a quick but not very realistic comparison, tools like bonnie++ can provide good numbers. I usually have problems with Bonnie++ giving me good numbers for the random I/O section, because it tends to run with way, way too small a data-set, so pay attention to the options for controlling that.