Hot swapping physical disks passed through to a qemu VM

I have a virtual server running in qemu on proxmox. It has two physical disks attached to the host using a virtio: directive in 101.cfg as folllows

virtio2: /dev/disk/by-id/ata-vol1,backup=0 
virtio3: /dev/disk/by-id/ata-vol2,backup=0

These contain a mirrored ZFS volume from which my files are served using nextcloud. This all works fine because these disks are never removed from the guest.

I also have a rotation of three other physical disks to which I back up (the physical controller is an LSI raid controller in JBOD configuration, the physical hardware is an HP Proiliant DL380e Gen8). I do this by zfs sending an updated snapshot from the storage array to whatever backup disk is present on the system. I had added these to the VM as follows:

virtio4: /dev/disk/by-id/ata-backup1-volname,backup=0
virtio5: /dev/disk/by-id/ata-backup2-volname,backup=0
virtio6: /dev/disk/by-id/ata-backup3-volname,backup=0

This sort of works but the problem is that none of those disks are hot swappable, so if I remove and insert disks in rotation, the guest stops being able to see them. I have disk hotplug enabled in my qemu config.

My question is what is the correct way to dynamically attach and detach these disks to my VM as they are inserted and removed?

I have been able to work out a partial solution using "qm monitor" to attach and detach the disk I am taking out by attaching and detaching virtio-scsi "pci cards" as follows:

To attach

qm> drive_add 0 file=/dev/disk/by-id/ata-<disk-id>,if=none,id=backup_vol,cache=none,detect-zeroes=on
qm> device_add virtio-blk-pci,drive=backup_vol,id=backup_scsi_controller

To detach

qm> device_del backup_scsi_controller

(Source: https://blog.chrishowie.com/2019/09/19/hot-swapping-virtio-disks-on-qemu/ via Google cache)

I was thinking I could potentially add these to my backup scripts, but it doesn't seem to be possible however to run qm monitor commands from a shell script, and qm monitor will not take arguments or input from stdin. From reading aroud, it might be possible to create a socket into qemu-agent on the guest and inject this using JSON, but I know nothing about JSON and am really hoping there is an easier way.

Solution 1:

I have been somewhat around the houses on this one, and indeed did get to the point where I was able to send QMP commands to a guest via a UNIX socket. This is supported by the default Proxmox config using a socket at /var/run/qemu-server/<server-id>.qmp. I can then interactively send QMP JSON messages using

rlwrap -C qmp socat - UNIX:/var/run/qemu-server/101.qmp

(rlwrap is not installed by default, but can be installed using "apt install rlwrap")

However, the QMP message format was as opaque as I thought it was going to be and has no direct equivalent to "drive_add" for reasons of ideological purity (which are perhaps understandable but nonetheless hugely unhelpful in the context of my present difficulty). There is a "blockdev_add" command that may well have done what I wanted assuming I could work out the correct syntax.

I therefore took a step back and came to the conclusion that "qm monitor" probably wasn't accepting input from stdin because of readline support. This lead me to think of "expect". A quick "apt install expect" later, and I was able to create scripts to attach and detach drives as follows:

qemu-drive-attach <vm-id> <device-name> <path-to-block-device>

with qemu-drive-attach script (device-name is arbitrary - I use "virtio8" for consistency with the Proxmox naming scheme)

!/usr/bin/expect
set vm_id               [lindex $argv 0];
set device_name         [lindex $argv 1];
set device_file         [lindex $argv 2];

spawn qm monitor $vm_id
expect "qm> "
send "drive_add 0 file=$device_file,if=none,id=drive-$device_name,cache=none,detect-zeroes=on\r"

expect "OK" {
    expect "qm> "
    send "device_add virtio-blk-pci,drive=drive-$device_name,id=$device_name\r"
}

expect "qm> "
send "quit\r"

send_user "\n"

And for detaching

qemu-drive-detach <vm-id> <device-name>

with qemu-drive-detach script

#!/usr/bin/expect
set vm_id               [lindex $argv 0];
set device_name         [lindex $argv 1];

spawn qm monitor $vm_id
expect "qm> "
send "device_del $device_name\r"

expect "qm> "
send "quit\r"

send_user "\n"

Error checking with these scripts leaves something to be desired, particularly as device_add and device_del are silent on success, but this will enable external block device drives to be hotswapped into the VM from a script.