du command takes WAY too long to run

I am running du -sh in a variety of directories to find disk hogs. I got two identical servers (Dell PE2850s), both with RHEL5 and it will take significantly longer to run du on one server over the other.

For example, doing du -sh /opt/foobar will take 5 minutes on server A (which has about 25 GB in it), and on server B, the same command with the same amount of data will report back to me almost instantaneous. I don't see anything glaringly obvious when running top, etc.

Any advice is greatly appreciated.

Solution 1:

If you have huge number of files in that directory and the contents of the directory constantly change, the directory entry itself gets fragmented over time. Then when the OS is reading the directory contents, there will be lots and lots of unnecessary disk seeks. This happens especially with ext* filesystems (ext4 might be better though) and the old ReiserFS v3.x filesystems (if that got past 85% full or so).

The solution is quite easy:

cp -pr origdir newdir
mv origdir origdir.bak
mv newdir origdir

Of course if everything is cached in RAM, this does not matter that much; usually Linux caches frequently accessed files and dirs quite aggressively. If you truly want to keep the contents of those directories in RAM, you can put something like ls -lah /your/dir 2>&1 >/dev/null to your cron.

EDIT: Oh, one thing popped on to my mind. If your server has a battery-backed up RAID controller with some cache in it, please check that the battery is OK. I've seen situations where the battery is dead and the controller disables the cache completely, ruining the performance very bad. For example HP servers might tell in the iLO logs something about the controller battery; in the actual server health dashboard everything seems to be fine and green, but only the log entry will tell you about this.

Solution 2:

I suggest to try the simple du command without any switches. You will eventually see which directory is slowing down the process. Might be a faulty disk, or some other reason, ...