Why do file copy operations in Linux get slower over time?

Solution 1:

This is a known issue with the Linux kernel itself that has been known for ages, no one was tackled the issue yet.

there is currently no fix or known work-around, aside from using Windows which increases in speed and then normalizes. I find this bug to be a very high-priority issue, but the programmers do not feel the same way. It is this bug alone that is starting to drive me away from Linux.

Refs:

• Bug 12309 - Large I/O operations result in poor interactive performance and high iowait times
• USB file transfer causes system freezes; ops take hours instead of minutes

Solution 2:

You should try rsync to monitor your transfer speed. For example:

rsync -PSauv sourcedir/* destdir/

If you actually see a drop in transfer speed, as opposed to files that are just big, then you need to look at a few other things.

1. If your disk is I/O-bound, you will have problems. You often see this when writing heavily to a flash drive or an external USB device.

2. Test both your disks with hdparm -Tt <device> to see if one is dramatically slower than it should be, or dramatically slower than the other device which can lead to I/O issues.

3. Check your dmesg output and your SMART status to make sure you don't have any obvious disk errors. You might also consider badblocks -n if you suspect disk errors on a non-SMART drive.

There are lots of reasons why a disk can thrash or perform sub-optimally, but until you can measure the problem, you won't know what the problem is. :)

Solution 3:

Remember that also the file copy process is somehow cached by Linux. If you take a single file (even if a 200MB one) and start copying it to another place, Linux caches, with great probability, the whole file in memory and writes it in its new location. Then, if you choose to copy that file again and with it a bunch of other files, then the first file is already cached and then probably skips the whole "read part", while the others are first read from and then written to their sources/destinations. This feature can even lead to infinite transfer speeds and similar oddities, so don't trust any measurement that could be invalidated by previous chaching mechanisms.