QEMU - incorrect real disk size of a virtual drive

df and ls report different sizes on my host machine because of the difference between the allocated size and the amount of space that's actually used in the EXT4 filesystem. The problem is that both report the wrong size. qemu-img also doesn't report the actual space that's used inside the guest's filesystem (also EXT4).

On the host:

# qemu-img info sdb.raw
image: sdb.raw
file format: raw
virtual size: 2.0T (2173253451776 bytes)
disk size: 1.9T

# ls -larth sdb.raw 
-rw-r--r-- 1 hypervisor hypervisor 2.0T Mar  6 13:47 sdb.raw

# du -sh sdb.raw 
1.9T   sdb.raw

# fdisk -l sdb.raw 
Disk sdb.raw: 2 TiB, 2173253451776 bytes, 4244635648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E6242A7E-1253-E74A-9389-68654A21E4F4

Device     Start        End    Sectors Size Type
sdb.raw1    2048 4244635614 4244633567   2T Linux filesystem

On the guest (/dev/vdb1):

# df -h
Filesystem      Size  Used Avail Use% Mounted on
dev              32G     0   32G   0% /dev
run              32G  496K   32G   1% /run
/dev/vda2       220G  100G  111G  48% /
tmpfs            32G     0   32G   0% /dev/shm
tmpfs            32G     0   32G   0% /sys/fs/cgroup
tmpfs            32G   26M   32G   1% /tmp
/dev/vdb1       2.0T  761G  1.2T  41% /root
tmpfs           6.3G     0  6.3G   0% /run/user/1002

As you can see, I'm only using 761G on the guest system and there's still 1.2T available. Despite that, qemu-img says that I'm using 1.9T. What's the reason? Is there a way to fix this? I'd like to see the actual used space on the host machine. I know it can be achieved because it worked correctly before I filled sdb.raw with data - qemu reported the correct disk size based on the space used inside the disk. Unfortunately, it worked only one way - when the used space was gradually increasing. After I deleted some files and reduced the used space from 1.9T to 761G, the size reported by qemu-img didn't change to a smaller value, it remained at 1.9T.

Solution 1:

qemu-img and du both report the actual allocated space from host point of view. They can not know/understand if the guest OS is really using that space or if, as in you case, it was freed by the user.

To inform the host that your guest has a ton of free space you need to fstrim your guest filesystem and you must be sure that your qemu/guest block device stack correctly passes down TRIM/discard requests. In order to do that, you need to:

• use a qemu block device driver with TRIM/discard support. Example of such drives are virtio-scsi, scsi and sata/ahci. All these drivers expose block devices using the classical /dev/sd[abcd...] nomenclature. On the other hand it appears that your guest is using a plain virtio driver (with /dev/vd[abcd...] names), which does not support TRIM/discard. You can check if your driver support discards by issuing lsblk -D;

• enable the relative libvirt domain discard=unmap option;

Finally, your host must be using a filesystem with hole_punching support as ext4 (which you are using) and xfs.

You can read here for some more information.

If all prerequisites are true, issuing fstrim <mount_point> inside your guest will automatically de-allocate unused space on host also. Be sure to understand that the logical/apparent file size will remain unchanged (ie: it will show 2.0 TB ever after fstrim), but using qemu-img or du will reveal the true (smaller) allocated file size.

If the prerequisites are not met, you need to offline compact your image file via virt-sparsify or qemu-img convert after cleaning your guest free space with zeroes (ie: dd if=/dev/zero of=bigfile bs=1M count=<almost_all_free_space>; rm bigfile).

Be sure to understand what the above steps means: doing any error can irremediably corrupt your image file. And be sure to have a know-good backup before attempting any operation on your disk image file!*

Solution 2:

It’s expected behavior. As you wrote, “after I deleted some files and reduced the used space from 1.9T to 761G” - raw virtual image cannot “collect garbage” automatically and the real space will remain written and allocated even for removed files. Workaround is to perform (double)conversion raw>qcow2(>raw).

More detailed explained here - https://balau82.wordpress.com/2011/05/08/qemu-raw-images-real-size/ and here - https://techpiezo.com/tech-insights/raw-vs-qcow2-disk-images-in-qemu-kvm/