Is EBS naturally CPU-hungry?
We have an m1.medium EC2 instance with a small (15G) EBS drive, running both Rails 3 and a PostgreSQL server. We've noticed CPU spikes at odd times, and finally realized that even simple, sustained EBS access seems to pin the CPU. For instance, just grepping about 3G of logs resulted in 100% CPU usage - which means both cores, which should be impossible for grep! Deleting a bunch of log files also used about 25% of the CPU, which is more than I'd expect. We're not swapping.
Is this normal? It's hard to Google for this, since "high CPU" is also the name of a type of EC2 instance. I'll happily provide more details and benchmarks, but first I wanted to check if this was a thing.
Performance of EBS volumes can be affected by things like:
New EBS volumes have a first-use penalty, even if they are created from an EBS snapshot. The first time you read or write to each block on the volume will take much longer than subsequent hits.
When an EBS snapshot has been initiated, the EBS volume may experience high iowait when you try to write to a block that has not yet been copied to the S3 snapshot storage.
EBS volumes use network bandwidth on the instance. You may get better IO performance and reduced CPU iowait if you upgrade to a larger instance type.
Here's an article I wrote about the lazy-load of EBS volumes from snapshots:
Identifying When a New EBS Volume Has Completed Initialization From an EBS Snapshot
http://alestic.com/2010/03/ebs-volume-initialization-from-snapshot
Here's an article I wrote describing why we had to move our EBS snapshots to a slave database instead of running them on the master:
EBS Snapshots of a MySQL Slave Database on EC2
http://alestic.com/2009/08/ec2-mysql-slave-snapshot
It's likely that nearly all of the CPU time spent during the grep you mention was due to iowait. Run top in another terminal during the grep and watch the %wa value. That value is the amount of time being spent waiting for IO to complete.
It's a well-known and well-discussed fact that EBS volumes perform quite poorly when it comes to IO. This is why many organizations will bond several EBS vols together via RAID (usually RAID 0, but possibly other levels may be useful as well) to increase performance.