linux: how to simulate hard disk latency? I want to increase iowait value without using a cpu power

device-mapper "delay" devices

Look at the "delay" target for device-mapper devices. This is exactly why it exists.

Example

Here's an example of how to get that going:

Create a place to read/write from

[root@centos6 ~]# dd if=/dev/zero of=/tmp/100M-of-zeroes bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.141834 s, 739 MB/s

Make it into a block device

Dev-mapper only maps from one block device to another, not between files and block devices. That's the job of the loopback device.

[root@centos6 ~]# losetup --show --find /tmp/100M-of-zeroes
/dev/loop0

Get the device size in blocks

Since this is what dev-mapper will need here in a moment...

[root@centos6 ~]# blockdev --getsize /dev/loop0
204800

Set up a "slow" device

# echo "0 204800 delay /dev/loop0 0 200" | dmsetup create dm-slow
(about a 30 second pause here with no output)

The fields in the device mapper setup table in the "echo" command above are:

starting sector of this section of the device mapper device (0)
number of sectors of this section of the device mapper device (204800)
the type of device mapper device for this section (delay)
the first argument to "delay", which is the device to use for real reads/writes after the delay (/dev/loop/0)
the second argument to "delay" which is the offset in the source device to use (0)
the third argument to "delay" which is the ms of time to delay reads (or reads and writes if no further parameters are specified.) (200)

We only have one line since we're treating the entire device mapper device the same, but this lets you have different sectors with different backing devices, only have some of them slow, only having some of them give errors, etc.

See https://linux.die.net/man/8/dmsetup for more info, including the possibly-also-useful "flakey" mapper type. Authoritative documentation on device-mapper's delay feature is at https://www.kernel.org/doc/Documentation/device-mapper/delay.txt

Is it slow?

[root@centos6 ~]# dd if=/dev/mapper/dm-slow of=/dev/null count=25000
25000+0 records in
25000+0 records out
12800000 bytes (13 MB) copied, 10.2028 s, 1.3 MB/s

Yeah, that's pretty slow, especially compared to the original:

[root@centos6 ~]# dd if=/dev/loop0 of=/dev/null count=25000
25000+0 records in
25000+0 records out
12800000 bytes (13 MB) copied, 0.0361308 s, 354 MB/s

So the mapped device is definitely introducing a delay.

Combine the above

I intentionally broke things apart so the process was easy to follow. However, you could easily combine steps above into fewer commands.

With fio and blktrace, you can replay an I/O trace. What type of disruption are you trying to simulate?

If the delays you wish to induce are from an existing production system, you can use that as the basis for your trace.