Strange high CPU I/O wait on postgresql server

Solution 1:

In short, you need faster disks.

When processes are in iowait, this means that they have issued an IO request, and are waiting for the results of that request. With adequate RAM, most of your working set will be cached in RAM, and as such, reads will not contribute to much of this IO contention, so writes are typically the culprit.

Regarding your disk IO graph, I suspect that New Relic is, for some reason, not collecting IO metrics correctly. I'd recommend possibly looking into a secondary data collection system like Munin. Short of that, you can watch disk IO in realtime by issuing the following command:

$ iostat -k 1

This will output realtime disk reads and writes, in kbps. I suspect you'll see quite a bit of activity there that New Relic is not seeing.