What creates CPU I/O wait but no disk operations?

NFS can do this, and it wouldn't surprise me if other network filesystems (and even FUSE-based devices) had similar effects.


Is there any chance other VMs on the server are thrashing the disk?

I know with virtualisation that you can get some strange results if the host node is overloaded.


If this is the Amazon EC2 Xen environment using instance-based storage, ask Amazon to check the health of the host containing this image.

If this is a Xen environment that you can gain access to the hypervisor, then check the IOwait from without for the disk image (file, network, LVM-slice, whatever) being used for the xvda and xvdb devices. You'll also want to check the I/O system, in general, for the hypervisor since other disk devices might be monopolizing the system's resources.

iostat -txk 5

is usually a good starting diagnostic tool. It takes 5-second summaries of I/O for ALL devices available to it, and thus is useful both with-in and wither-out the VM image.


Check your available file descriptors / inodes. When you hit the limit, they swap and mimic iowait

Edit

I saw you are using xen, have a look at your current interrupts, you might find blkif is higher than normal.

Bit late now, but get munin installed and it will really help future debugging.