Disk latency is growing since a few days ago, with no special change in disk throughput
Solution 1:
There may be a number of things at work here.
1) Your IO load is increasing
What is not shown in your graphs or information given is what is your IO load like. Is your system spending more time in IO Wait? Are the depths for your Read/Write queues changing? If this is the case then I would suggest that 5400 RPM drives may be way too slow for your needs. The slower the spindle speed (RPMs) the longer it takes for the read head to locate a piece of data.
1a) Fragmentation of data
This is highly dependent on your IO load characteristics, but it's possible your file system has become highly fragmented meaning it takes longer to read larger chunks of data.
2) Swap space thrashing
What is going on with your system's memory? Is the system running out of memory and is now needing to thrash the swap file? If that is the case then your system overall would likely exhibit slowness. Either way check your process table.
3) Possible impending disk failure
It's possible you have a disk that is starting to fail. I would suggest you install the smartmon tools and have a look at the SMART data using the smartctl utility.
Solution 2:
@Red Tux answer makes sense. cannot accurately answer without a few more details:
- what's the server doing? it is DB or file server or web/app server?
- DB then possibly look for 1a)
- web/app then look for 2)
- file server then look for 3)
- vmstat and iostat report will help. run with 5~10s delay during peek for 1min and submit.