what percentage of iowait is considered to be high?
Solution 1:
The best answer I can give you is "iowait
is too high when it's affecting performance."
Your "50% of the CPU's time is spent in iowait
" situation may be fine if you have lots of I/O and very little other work to do as long as the data is getting written out to disk "fast enough". Conversely it could be catastrophic if the server is doing a high amount of disk I/O and is noticeably slow to the point where users are wasting "wall-clock time" waiting for operations to complete.
To determine what affects performance you need to throw some benchmarks (stress tests) at the system and see at what kind of I/O throughput the system can handle before it starts to noticeably bog down.
(The number you get from those benchmarks is largely an academic one - it's nice to know where the choke will happen, but when it happens the "solution" is to reduce the amount of time I/O operations take, or the number of them waiting in the queue. You can do that by switching to faster disks, installing better controllers with more cache, putting in SSDs, splitting the workload to multiple servers, etc.)