QEMU + KVM + LVM - performance of block device drive vs file image

I'm creating new setup for my virtual machines and testing which method for storage is most fastest. My test environment consists of HDD drive with LVM on LUKS. I created single LV for virtual machine drive and reused it for both tests to maintain same place on HDD drive to maintain consistent performance (HDD read/write speed depends on physical position).

• Host: Arch Linux, kernel 4.12.8
• Guest: Ubuntu Desktop 17.04

Performance tested with command:

    dd if=/dev/zero of=test bs=16M count=100 conv=sync

First test: using LV directly as virtual machine's drive

Command:

qemu-system-x86_64 \
  -drive format=raw,file=/dev/mapper/vg_vm-lv_vm_test,if=virtio,aio=native,cache.direct=on \
  -net nic,model=virtio \
  -net user \
  -vga virtio \
  -display gtk,gl=on \
  -smp 3 \
  -cpu host \
  -machine type=pc,accel=kvm \
  -m 3G

Results (each value represents single run):

• Creating new file: 98.4 MB/s ; 112 MB/s
• Writing to existing file: 62.5 MB/s ; 68.7 MB/s ; 64.8 MB /s

Second test: creating ext4 on LV and putting raw image file on that

Command:

qemu-system-x86_64 \
  -drive format=raw,file=./ubuntu_17,if=virtio,aio=native,cache.direct=on \
  -net nic,model=virtio \
  -net user \
  -vga virtio \
  -display gtk,gl=on \
  -smp 3 \
  -cpu host \
  -machine type=pc,accel=kvm \
  -m 3G

Results (each value represents single run):

• Creating new file: 254 MB/s ; 242 MB/s
• Writing to existing file: 187 MB/s ; 189 MB/s ; 190 MB/s

Third test: using LV directly as virtual machine's drive, different settings

Command:

qemu-system-x86_64 \
  -drive format=raw,file=/dev/mapper/vg_vm-lv_vm_test,if=virtio,cache=none \
  -net nic,model=virtio \
  -net user \
  -vga virtio \
  -display gtk,gl=on \
  -smp 3 \
  -cpu host \
  -machine type=pc,accel=kvm \
  -m 3G

Results (each value represents single run):

• Creating new file: 129 MB/s ; 125 MB/s
• Writing to existing file: 103 MB/s ; 97 MB/s ; 81.9 MB /s

Question

Clearly there is difference between these two solutions however I expected raw block device to be at least as fast as image file because there should be no overhead of host's filesystem. I suppose that some caching occurs in between for file image or options for raw block device are not optimal. Why is raw LV slower in this case? What can I do to improve its performance? Or if it just should be slower, then why?

EDIT: I added third test case using settings from: http://www.linux-kvm.org/page/Tuning_KVM. Turns out to be a bit faster but still slower than file image. I observed also that with each run for existing file it is getting slower - however fragmentation shouldn't occur for file overwrite so I am not sure why it happens.

Block size mismatch might be the problem.

You ideally want to match your File System with the underlying block size of the media. While you didn't share what you picked for block sizes I think you have 4 kilobytes for Ext4 attempt and 512 bytes for the LV. If your underlying media's natural block size is 4 kilobytes then I think this explains both your speed issue and your diminishing speed. Since you are possibly only writing 512 in to a 4k block you wasted 75% of the block and subsequent writes will use more blocks, incurring more overhead and waste.

Retry your test with a nicely matched File System block size matched with the underlying media block size. In this case try your LV with a 4k block size.

LVM does add a bit more overhead than Ext4. Stack Overflow Answer So I think this would explain why it is just a bit slower. :)