Is iostat unreliable?

iostat

I have noticed something worrying with iostat that I can't quite explain - it appears iostat may not be quite reliable(?)

I'm trying to troubleshoot poor MySQL performance, and one ovious place to look is the disks, so:

ip-10-14-4-14 root : iostat -x nvme2n1
Linux 5.4.0-1029-aws (ip-10-14-4-14)    10/27/21        _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.19    0.00    0.47    0.84    0.00   97.49

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz  aqu-sz  %util
nvme2n1          4.93    160.70     0.26   4.95    0.53    32.60  216.03   6863.39    57.07  20.90    0.78    31.77    0.00      0.00     0.00   0.00    0.00     0.00    0.00  16.50

That doesn't look too bad - %util is only 16.5; however:

ip-10-14-4-14 root : iostat -y 1 -x nvme2n1
Linux 5.4.0-1029-aws (ip-10-14-4-14)    10/27/21        _x86_64_        (8 CPU)


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.65    0.00    1.26    1.13    0.00   92.96

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz  aqu-sz  %util
nvme2n1         10.00    160.00     0.00   0.00    0.40    16.00 2058.00  40260.00   326.00  13.67    0.65    19.56    0.00      0.00     0.00   0.00    0.00     0.00    0.00 100.00

Same system, almost at the same time, but now %util is 100. I've even tried running the first version in a loop, and then %util stays low:

for i in $(seq 1 50); do iostat -x nvme2n1; done
Linux 5.4.0-1029-aws (ip-10-14-4-14)    10/27/21        _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.19    0.00    0.48    0.84    0.00   97.49

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz  aqu-sz  %util
nvme2n1          4.93    160.70     0.26   4.95    0.53    32.59  216.32   6868.15    57.10  20.88    0.78    31.75    0.00      0.00     0.00   0.00    0.00     0.00    0.00  16.51
...

What's going on - am I just not using the tool correctly?

I think I've figured it out: the option -y throws out the first set of results, which isn't realiable, and waits for iostat to do a full sampling.

IOW, always use iostat -y - or try sar -d -p 1 1

Is iostat unreliable?

Related

Recent Posts