How do I test the effect of ionice (against a device using the cfq scheduler)?
I'd try measuring the effect by using a load generator like stress -i n or stress -d n, where "n" is the number of processes. Run that in one window. Try nmon or iostat in another and try a representative application process on the same block device. See how service time changes in iostat with various ionice settings (or test response from within your app).
As for cfq, there seemed to be changes throughout the RHEL5 lifecycle (on 2.6.18). It was noticeable enough on my application servers that I had to move to the noop and deadline elevators because of contention issues.