Linux utility to record IO statistics (random/sequential, block sizes, read/write ratio)

As part of provisioning our new server (see other SF) I'd like to find out the following:

• ratio of random to sequential reads & writes
• amount of data read & written at a time (pref in histogram form)

I can already figure out our reads/writes on a per-operation and overall data level using iostat & dstat, but I'd like to know more. For example, I'd like to know that we're mostly random 16kb reads, or a lot of sequential 64kb reads with random writes.

We're (currently) on an Ubuntu 10.04 VM.

Is there a utility that I can run that will record and present this information for me?

I like using the collectl utility for this purpose. I mentioned this in another answer for someone looking for specific I/O statistics output for replay. You should be able to tailor the output to your specific needs. With Ubuntu, this should be available via the normal repositories. The caveat is that you won't see the percentages you are you looking for.

nmon is also a nice tool that can provide most of the info you're seeking in an easy interface.

If you're already familiar with iostat, what flags are you currently using?

If you're hardcore and are profiling a specific application, SystemTap could be an option, but I'm not sure if it's the right match. It may be more work than needed.

It's pretty easy to obtain this info from Solaris Dtrace (on ZFS systems - NexentaStor screenshot below), but have to dig a little on the Linux side... So you could consider Dtrace for Linux.

I followed the way from this post, using SystemTap tool.

Fisrt, install systemtap. For Debian/Ubuntu:

apt-get install systemtap linux-image-$(uname -r)-dbg linux-headers-$(uname -r)

Create systemtap script, save as blockio.stp:

global writes
global reads

probe ioblock.request {
    if(bio_rw_num(rw) == BIO_WRITE)
        writes[devname] <<< size
    if(bio_rw_num(rw) == BIO_READ)
        reads[devname] <<< size
}

probe end {
    printf("\n")
    # foreach([devname] in writes-) {
    #     printf("Device: %s\n", devname)
    #     println(@hist_log(writes[devname]))
    # }
    printf("WRITE\n")
    println(@hist_log(writes["sda1"]))
    printf("READ\n")
    println(@hist_log(reads["sda1"]))
}

Replace sda1 with your drive or use foreach method like in original post to show statistics for all drives.

Start monitoring. Run as root

stap -v blockio.stp

Stop it with Ctrl+C when you want to finish monitoring. The output will be like

Pass 1: parsed user script and 95 library script(s) using 84064virt/25528res/2408shr/23916data kb, in 160usr/0sys/164real ms.
Pass 2: analyzed script: 3 probe(s), 21 function(s), 2 embed(s), 4 global(s) using 297648virt/205248res/86464shr/119268data kb, in 1750usr/60sys/1815real ms.
Pass 3: using cached /root/.systemtap/cache/ea/stap_ea33aaf95086fa562bb720d36fca7504_12821.c
Pass 4: using cached /root/.systemtap/cache/ea/stap_ea33aaf95086fa562bb720d36fca7504_12821.ko
Pass 5: starting run.
^C
WRITE
  value |-------------------------------------------------- count
      0 |@@@@@@@@@@@@@@@@@@@@@@@@                           191952
      1 |                                                        0
      2 |                                                        0
        ~
   1024 |                                                        0
   2048 |                                                        0
   4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  395272
   8192 |@@                                                  16273
  16384 |@                                                   12799
  32768 |                                                     4908
  65536 |                                                     4170
 131072 |                                                     2159
 262144 |                                                     6546
 524288 |                                                     4587
1048576 |                                                        0
2097152 |                                                        0

READ
  value |-------------------------------------------------- count
    128 |                                                       0
    256 |                                                       0
    512 |                                                       2
   1024 |                                                       0
   2048 |                                                       0
   4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  38229
   8192 |@@                                                  1550
  16384 |@@@                                                 2525
  32768 |@@                                                  1692
  65536 |@@                                                  1693
 131072 |@@@@@@@@@@@@@@@@@@                                 14455
 262144 |                                                     217
 524288 |                                                       0
1048576 |                                                       0

Pass 5: run completed in 140usr/720sys/3662349real ms.