top says IOWAIT is high, how to I find the culprit processes

Solution 1:

Many USB "pen" drives are very slow, particularly for writing, so there might be very little you can do other than to try a faster drive.

To answer your direct question: You can get some picture of what is going on disk wise with iotop -o - if left running during busy periods that will show you which processes are actively performing disk reads/writes and how much. This might highlight some process or service that you can do without and turn off to help things along. There are some crons jobs in many Linux setups that scan a chunk of the filesytsem (if not all of it/them) occasionally to update search caches (such as the updatedb task that updates the data used by commands like locate) - if you find one of those is causing your slowness then they can be safely removed from cron's list of things to regularly run.

How much RAM do you have in the target machine? If it is a small machine by that measure then this will not help as there will be little room to cache reads (so things read again a short time later need to come from the USB stick again rather than a cached copy in RAM).

You could try a cut-down version of Ubuntu too, particularly if you are in a low-RAM situation but it could also help if you have lots of RAM but slow drive(s) by reducing boot times if nothing else. lubuntu is a popular choice for this - all the usual packages are available should you need them but it installs less by default, starts less services on boot by default, and uses a lower weight desktop environment (everything other than these changes is the same arrangement, so you should find it relatively familiar even if Ubuntu is the only Linux variant you have much experience with).