Why does dd show slow disk speed first time?

I'm checking the disk speed of a hdd on CentOS 6 using dd command. The setup is a DRBD setup - with this server being primary. To ensure that I get correct values, I execute the dd command 3 times with different output files and then take average of the read/write time. However the first reading of the dd command is order of magnitude slower than the next 2. Eg.

time -p dd if=/dev/zero of=/mailstore/testfile bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 88.5175 s,
3.0 MB/s
real 90.12
user 0.00
sys 0.66

time -p dd if=/dev/zero of=/mailstore/testfile1 bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.226015 s, 1.2 GB/s
real 0.30
user 0.00
sys 0.22


time -p dd if=/dev/zero of=/mailstore/testfile2 bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.22094 s, 1.2 GB/s
real 0.22
user 0.00
sys 0.21

Is this normal ? Should I ignore the first reading and take 3 more after that ?

As suggested by poige, the dd command gives consistent output using the opflag=direct option. Eg.

time -p dd if=/dev/zero of=/mailstore/filetest33 bs=16k count=16384 oflag=direct
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 296.587 s, 905 kB/s
real 296.61
user 0.03
sys 1.07

time -p dd if=/dev/zero of=/mailstore/filetest44 bs=16k count=16384 oflag=direct
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 260.308 s, 1.0 MB/s
real 260.42
user 0.04
sys 1.13

time -p dd if=/dev/zero of=/mailstore/filetest56 bs=16k count=16384 oflag=direct
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 253.681 s, 1.1 MB/s
real 253.68
user 0.03
sys 1.06

Solution 1:

dd has direct option which requires kernel to bypass any caching and send data directly to block device. If you need benchmarking your device, not RAM, you have to use it, for e. g.:

dd if=/dev/zero of=direct_output bs=1M count=100 oflag=direct