How to identify I/O bottlenecks on a Linux server?

How do you determine that the performance of your Linux server is I/O bound and, perhaps more importantly, what process or processes are casuing the problem?

Solution 1:

I wrote a comprehensive guide to tracking down performance bottlenecks on Linux systems for work: http://web.archive.org/web/20101028025942/https://anchor.com.au/hosting/development/HuntingThePerformanceWumpus . Covers more than you asked for, but it'll (hopefully) help you track down the problem you're seeing regardless of the actual source.

Solution 2:

Top has a field called "iowait". If your system is seeing a lot of that, you know something's up. There's also iotop!

Package: iotop:
Description: simple top-like I/O monitor
 iotop does for I/O usage what top(1) does for CPU usage. It watches I/O
 usage information output by the Linux kernel (requires 2.6.20 or later)
 and displays a table of current I/O usage by processes on the system.
 Handy for answering the question "Why is my disk churning so much?".
Homepage: http://guichaz.free.fr/iotop/

Solution 3:

IoTop is probably what you're looking for.