Writing to disk using `pv` seems to be fast at first and slow at the end

I was writing a Kubuntu image to my USB stick using this command

pv /home/manuel/Downloads/torrents/kubuntu-16.10-desktop-amd64.iso > /dev/sdb

However, the output progress bar of pv was half-filled right from the very beginning. Hence, it looked like transmission would have started with an incredible speed and slowed down substantially.

1,49GiB 0:03:03 [8,31MiB/s] [===============// //===============>] 100%

That makes using the pv command rather useless.

How can I disable this caching functionality which seems to have been activated?

Solution 1:

I found your post while looking into a similar issue trying to write an image to an SD card (via USB reader/writer). Using pv, it would show 100% almost immediately, but take several minutes afterwards to actually complete.

I found a solution to my issue using dd, pv and pipes, and setting the write dd to direct mode. This also has had the effect of improving the write speed quite a lot. I don't know if there are any downsides of using this method (I read the image back and performed a checksum - and all appears ok).

My example is using an image of raspbian (2017-09-07-raspbian-stretch-lite.img), which is 1854590976 bytes (1.8GB) in size. I've shown a couple of other methods and you can see the differences in time taken.

So, using dd | pv | dd with direct mode, it took only 2 minutes 57 seconds:

dd if=2017-09-07-raspbian-stretch-lite.img ibs=1M status=none | pv -s 1854590976 | dd of=/dev/sdX obs=1M oflag=direct status=none

(setting status to none stops dd messing up the pv display).

Using pv by itself shows 100% immediately, but then took 8 minutes 28 seconds to actually complete:

pv 2017-09-07-raspbian-stretch-lite.img > /dev/sdX

Using dd by itself without any progress view, took 8 minutes 15 seconds:

dd if=2017-09-07-raspbian-stretch-lite.img of=/dev/sdX bs=1M

Using the same dd above but adding oflag=direct makes it 2 minutes 54:

dd if=2017-09-07-raspbian-stretch-lite.img of=/dev/sdX bs=1M oflag=direct

I've tried loads of different combinations, which are either far slower (some take up to 12 minutes!), or again show 100% immediately.

One caveat of using pv this way is that you need to find out the size of the image first, and specify it with the -s option. If you're scripting things like me, it's easy enough to determine the file size at the same time.

Hope it helps someone.