[pve-devel] [PATCH qemu/storage/qemu-server v3 00/51] let's switch to blockdev, blockdev, blockdev, part four (final)

[pve-devel] [PATCH qemu/storage/qemu-server v3 00/51] let's switch to blockdev, blockdev, blockdev, part four (final)

[Fiona Ebner]

v2 https://lore.proxmox.com/pve-devel/20250701154117.434512-1-f.ebner@proxmox.com/

The preliminary final part in the series. I'm sure there will be some
follow-ups, and the decisions about edge cases like cache mode for EFI
disk and querying file child are not yet set in stone. But this should
essentially be it.

The switch from '-drive' to '-blockdev' is in preparation for future
features like external snapshots, FUSE exports via qemu-storage-daemon
and also generally the more modern interface in QEMU. It also allows
to address some limitations drive-mirror had, in particular this
series makes it possible to mirror between storages having a different
aio default as well as mirror when the size of the allocated image
doesn't exactly match for EFI disks, see patch "partially fix #3227:
ensure that target image for mirror has the same size for EFI disks".

The switch is guarded by machine version 10.0 to avoid any potential
incompatibilities between -drive and -blockdev options/defaults.

What is still missing is support for the rather obscure 'snapshot'
drive option where writes will go to a temporary image (currently in
'/var/tmp', which is far from ideal to begin with). That requires
inserting an overlay node.

When using -drive, storage plugins currently give us a path that QEMU
understands, some using special protocols such as 'iscsi://'. We'd
like to switch to using the more modern -blockdev for PVE 9. The
plan is to have the storage plugins return the very basic information
required to access the image, and qemu-server can then add other
settings like cache, aio, etc. on top. In fact, pretty similar to what
we have now for -drive, just with a structured hash rather than a
string.

This is also a prerequisite for qemu-storage-daemon, that would be
useful for TPM-as-qcow2 exported via NBD or FUSE or external backup
provider restore providing an NBD export for the provider to write to.


Changes from the previous version:
* Use latest iteration of QEMU patch for child node name info.
* Restrict allowed drivers and options returned by storage plugins
  (Thanks Thomas, restricting drivers makes a ton of sense, as there
  is way more than just protocol paths).
* Implement protocol path parsing for common protocols in default
  storage implementation.

qemu:

Fiona Ebner (4):
  PVE backup: prepare for the switch to using blockdev rather than drive
  block/zeroinit: support using as blockdev driver
  block/alloc-track: support using as blockdev driver
  block/qapi: include child references in block device info

 block/qapi.c               | 10 ++++++++
 block/zeroinit.c           | 12 ++++++---
 pve-backup.c               | 51 +++++++++++++++++++++++++++++---------
 qapi/block-core.json       | 40 +++++++++++++++++++++++++++---
 tests/qemu-iotests/184.out |  8 ++++++
 5 files changed, 103 insertions(+), 18 deletions(-)


storage:

Fiona Ebner (12):
  plugin: add method to get qemu blockdevice options for volume
  iscsi direct plugin: implement method to get qemu blockdevice options
  zfs iscsi plugin: implement new method to get qemu blockdevice options
  zfs pool plugin: implement method to get qemu blockdevice options
  ceph/rbd: set 'keyring' in ceph configuration for externally managed
    RBD storages
  rbd plugin: implement new method to get qemu blockdevice options
  plugin: qemu block device: add hints option and EFI disk hint
  plugin: qemu block device: add support for snapshot option
  plugin: add machine version to qemu_blockdev_options() interface
  qemu blockdev options: restrict allowed drivers and options
  plugin: qemu blockdev options: parse protocol paths in default
    implementation
  plugin api: bump api version and age

 ApiChangeLog                         |  15 +++
 src/PVE/CephConfig.pm                |  50 ++++++++
 src/PVE/Storage.pm                   | 135 ++++++++++++++++++-
 src/PVE/Storage/ISCSIDirectPlugin.pm |  17 +++
 src/PVE/Storage/Plugin.pm            | 185 +++++++++++++++++++++++++++
 src/PVE/Storage/RBDPlugin.pm         |  60 +++++++++
 src/PVE/Storage/ZFSPlugin.pm         |  19 +++
 src/PVE/Storage/ZFSPoolPlugin.pm     |  16 +++
 8 files changed, 495 insertions(+), 2 deletions(-)


qemu-server:

Fiona Ebner (35):
  mirror: code style: avoid masking earlier declaration of $op
  test: collect mocked functions for QemuServer module
  drive: add helper to parse drive interface
  drive: drop invalid export of get_scsi_devicetype
  blockdev: add and use throttle_group_id() helper
  blockdev: introduce top_node_name() and parse_top_node_name() helpers
  blockdev: add helpers for attaching and detaching block devices
  blockdev: add missing include for JSON module
  backup: use blockdev for fleecing images
  backup: use blockdev for TPM state file
  blockdev: introduce qdev_id_to_drive_id() helper
  blockdev: introduce and use get_block_info() helper
  blockdev: move helper for resize into module
  blockdev: add helper to get node below throttle node
  blockdev: resize: query and use node name for resize operation
  blockdev: support using zeroinit filter
  blockdev: make some functions private
  blockdev: add 'no-throttle' option to skip generationg throttle top
    node
  block job: allow specifying a block node that should be detached upon
    completion
  block job: add blockdev mirror
  blockdev: add change_medium() helper
  blockdev: add blockdev_change_medium() helper
  blockdev: move helper for configuring throttle limits to module
  clone disk: skip check for aio=default (io_uring) compatibility
    starting with machine version 10.0
  print drive device: don't reference any drive for 'none' starting with
    machine version 10.0
  blockdev: add support for NBD paths
  blockdev: add helper to generate PBS block device for live restore
  blockdev: support alloc-track driver for live-{import,restore}
  live import: also record volid information
  live import/restore: query which node to use for operation
  live import/restore: use Blockdev::detach helper
  command line: switch to blockdev starting with machine version 10.0
  test: migration: update running machine to 10.0
  partially fix #3227: ensure that target image for mirror has the same
    size for EFI disks
  blockdev: pass along machine version to storage layer

 src/PVE/API2/Qemu.pm                          |   5 +-
 src/PVE/QemuConfig.pm                         |  12 +-
 src/PVE/QemuServer.pm                         | 304 +++++-----
 src/PVE/QemuServer/BlockJob.pm                | 216 ++++++-
 src/PVE/QemuServer/Blockdev.pm                | 551 +++++++++++++++++-
 src/PVE/QemuServer/Drive.pm                   |  21 +-
 src/PVE/QemuServer/OVMF.pm                    |  26 +-
 src/PVE/VZDump/QemuServer.pm                  |  50 +-
 src/test/MigrationTest/QemuMigrateMock.pm     |  13 +
 src/test/MigrationTest/QmMock.pm              |  56 +-
 src/test/cfg2cmd/aio.conf.cmd                 |  42 +-
 src/test/cfg2cmd/bootorder-empty.conf.cmd     |  12 +-
 src/test/cfg2cmd/bootorder-legacy.conf.cmd    |  12 +-
 src/test/cfg2cmd/bootorder.conf.cmd           |  12 +-
 ...putype-icelake-client-deprecation.conf.cmd |   6 +-
 src/test/cfg2cmd/efi-raw-template.conf.cmd    |   7 +-
 src/test/cfg2cmd/efi-raw.conf.cmd             |   7 +-
 .../cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd  |   7 +-
 src/test/cfg2cmd/efi-secboot-and-tpm.conf.cmd |   7 +-
 src/test/cfg2cmd/efidisk-on-rbd.conf.cmd      |   7 +-
 src/test/cfg2cmd/ide.conf.cmd                 |  15 +-
 src/test/cfg2cmd/q35-ide.conf.cmd             |  15 +-
 .../q35-linux-hostpci-mapping.conf.cmd        |   7 +-
 .../q35-linux-hostpci-multifunction.conf.cmd  |   7 +-
 .../q35-linux-hostpci-template.conf.cmd       |  10 +-
 ...q35-linux-hostpci-x-pci-overrides.conf.cmd |   7 +-
 src/test/cfg2cmd/q35-linux-hostpci.conf.cmd   |   7 +-
 src/test/cfg2cmd/q35-simple.conf.cmd          |   7 +-
 src/test/cfg2cmd/seabios_serial.conf.cmd      |   6 +-
 src/test/cfg2cmd/sev-es.conf.cmd              |   7 +-
 src/test/cfg2cmd/sev-std.conf.cmd             |   7 +-
 src/test/cfg2cmd/simple-btrfs.conf.cmd        |  15 +-
 src/test/cfg2cmd/simple-cifs.conf.cmd         |  15 +-
 .../cfg2cmd/simple-disk-passthrough.conf.cmd  |   9 +-
 src/test/cfg2cmd/simple-lvm.conf.cmd          |  12 +-
 src/test/cfg2cmd/simple-lvmthin.conf.cmd      |  12 +-
 src/test/cfg2cmd/simple-rbd.conf.cmd          |  27 +-
 src/test/cfg2cmd/simple-virtio-blk.conf.cmd   |   6 +-
 .../cfg2cmd/simple-zfs-over-iscsi.conf.cmd    |  15 +-
 src/test/cfg2cmd/simple1-template.conf.cmd    |   9 +-
 src/test/cfg2cmd/simple1.conf.cmd             |   6 +-
 src/test/run_config2command_tests.pl          |  19 +
 src/test/run_qemu_migrate_tests.pl            |  16 +-
 43 files changed, 1211 insertions(+), 418 deletions(-)


Summary over all repositories:
  56 files changed, 1809 insertions(+), 438 deletions(-)

-- 
Generated by git-murpp 0.5.0


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu v3 01/51] PVE backup: prepare for the switch to using blockdev rather than drive

[Fiona Ebner]

Also allow finding block nodes by their node name rather than just via
an associated block backend, which might not exist for block nodes.

For regular drives, it is essential to not use the throttle group,
because otherwise the limits intended only for the guest would also
apply to the backup job.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 pve-backup.c | 51 +++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 39 insertions(+), 12 deletions(-)

diff --git a/pve-backup.c b/pve-backup.c
index 0450303017..457fcb7e5c 100644
--- a/pve-backup.c
+++ b/pve-backup.c
@@ -847,29 +847,56 @@ static PVEBackupDevInfo coroutine_fn GRAPH_RDLOCK *get_single_device_info(
     Error **errp)
 {
     BlockBackend *blk = blk_by_name(device);
-    if (!blk) {
-        error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
-                  "Device '%s' not found", device);
-        return NULL;
+    BlockDriverState *root_bs, *bs;
+
+    if (blk) {
+        root_bs = bs = blk_bs(blk);
+    } else {
+        /* TODO PVE 10 - fleecing will always be attached without blk */
+        root_bs = bs = bdrv_find_node(device);
+        if (!bs) {
+            error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
+                      "Device '%s' not found", device);
+            return NULL;
+        }
+        /* For TPM, bs is already correct, otherwise need the file child. */
+        if (!strncmp(bs->drv->format_name, "throttle", 8)) {
+            if (!bs->file || !bs->file->bs) {
+                error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
+                          "Device '%s' not found (no file child)", device);
+                return NULL;
+            }
+            bs = bs->file->bs;
+        }
     }
-    BlockDriverState *bs = blk_bs(blk);
+
     if (!bdrv_co_is_inserted(bs)) {
         error_setg(errp, "Device '%s' has no medium", device);
         return NULL;
     }
+
     PVEBackupDevInfo *di = g_new0(PVEBackupDevInfo, 1);
     di->bs = bs;
-    di->device_name = g_strdup(bdrv_get_device_name(bs));
+    /* Need the name of the root node, e.g. drive-scsi0 */
+    di->device_name = g_strdup(bdrv_get_device_or_node_name(root_bs));
 
     if (device_uses_fleecing && device_uses_fleecing(device)) {
         g_autofree gchar *fleecing_devid = g_strconcat(device, "-fleecing", NULL);
+        BlockDriverState *fleecing_bs;
+
         BlockBackend *fleecing_blk = blk_by_name(fleecing_devid);
-        if (!fleecing_blk) {
-            error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
-                      "Device '%s' not found", fleecing_devid);
-            goto fail;
+        if (fleecing_blk) {
+            fleecing_bs = blk_bs(fleecing_blk);
+        } else {
+            /* TODO PVE 10 - fleecing will always be attached without blk */
+            fleecing_bs = bdrv_find_node(fleecing_devid);
+            if (!fleecing_bs) {
+                error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
+                          "Device '%s' not found", fleecing_devid);
+                goto fail;
+            }
         }
-        BlockDriverState *fleecing_bs = blk_bs(fleecing_blk);
+
         if (!bdrv_co_is_inserted(fleecing_bs)) {
             error_setg(errp, "Device '%s' has no medium", fleecing_devid);
             goto fail;
@@ -927,7 +954,7 @@ static GList coroutine_fn GRAPH_RDLOCK *get_device_info(
 
             PVEBackupDevInfo *di = g_new0(PVEBackupDevInfo, 1);
             di->bs = bs;
-            di->device_name = g_strdup(bdrv_get_device_name(bs));
+            di->device_name = g_strdup(bdrv_get_device_or_node_name(bs));
             di_list = g_list_append(di_list, di);
         }
     }
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu v3 02/51] block/zeroinit: support using as blockdev driver

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 block/zeroinit.c     | 12 +++++++++---
 qapi/block-core.json |  5 +++--
 2 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/block/zeroinit.c b/block/zeroinit.c
index f9d513db15..036edb17f5 100644
--- a/block/zeroinit.c
+++ b/block/zeroinit.c
@@ -66,6 +66,7 @@ static int zeroinit_open(BlockDriverState *bs, QDict *options, int flags,
     QemuOpts *opts;
     Error *local_err = NULL;
     int ret;
+    const char *next = NULL;
 
     s->extents = 0;
 
@@ -77,9 +78,14 @@ static int zeroinit_open(BlockDriverState *bs, QDict *options, int flags,
         goto fail;
     }
 
-    /* Open the raw file */
-    ret = bdrv_open_file_child(qemu_opt_get(opts, "x-next"), options, "next",
-                               bs, &local_err);
+
+    next = qemu_opt_get(opts, "x-next");
+
+    if (next) {
+        ret = bdrv_open_file_child(next, options, "next", bs, &local_err);
+    } else { /* when opened as a blockdev, there is no 'next' option */
+        ret = bdrv_open_file_child(NULL, options, "file", bs, &local_err);
+    }
     if (ret < 0) {
         error_propagate(errp, local_err);
         goto fail;
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 2fb51215f2..f8ed564cf0 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -3586,7 +3586,7 @@
             { 'name': 'virtio-blk-vfio-pci', 'if': 'CONFIG_BLKIO' },
             { 'name': 'virtio-blk-vhost-user', 'if': 'CONFIG_BLKIO' },
             { 'name': 'virtio-blk-vhost-vdpa', 'if': 'CONFIG_BLKIO' },
-            'vmdk', 'vpc', 'vvfat' ] }
+            'vmdk', 'vpc', 'vvfat', 'zeroinit' ] }
 
 ##
 # @BlockdevOptionsFile:
@@ -5172,7 +5172,8 @@
                       'if': 'CONFIG_BLKIO' },
       'vmdk':       'BlockdevOptionsGenericCOWFormat',
       'vpc':        'BlockdevOptionsGenericFormat',
-      'vvfat':      'BlockdevOptionsVVFAT'
+      'vvfat':      'BlockdevOptionsVVFAT',
+      'zeroinit':   'BlockdevOptionsGenericFormat'
   } }
 
 ##
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu v3 03/51] block/alloc-track: support using as blockdev driver

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 qapi/block-core.json | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index f8ed564cf0..07c5773717 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -3567,7 +3567,8 @@
 # Since: 2.9
 ##
 { 'enum': 'BlockdevDriver',
-  'data': [ 'blkdebug', 'blklogwrites', 'blkreplay', 'blkverify', 'bochs',
+  'data': [ 'alloc-track',
+            'blkdebug', 'blklogwrites', 'blkreplay', 'blkverify', 'bochs',
             'cloop', 'compress', 'copy-before-write', 'copy-on-read', 'dmg',
             'file', 'snapshot-access', 'ftp', 'ftps',
             {'name': 'gluster', 'features': [ 'deprecated' ] },
@@ -3668,6 +3669,21 @@
 { 'struct': 'BlockdevOptionsNull',
   'data': { '*size': 'int', '*latency-ns': 'uint64', '*read-zeroes': 'bool' } }
 
+##
+# @BlockdevOptionsAllocTrack:
+#
+# Driver specific block device options for the alloc-track backend.
+#
+# @backing: backing file with the data.
+#
+# @auto-remove: whether the alloc-track driver should drop itself
+#     after completing the stream.
+#
+##
+{ 'struct': 'BlockdevOptionsAllocTrack',
+  'base': 'BlockdevOptionsGenericFormat',
+  'data': { 'auto-remove': 'bool', 'backing': 'BlockdevRefOrNull' } }
+
 ##
 # @BlockdevOptionsPbs:
 #
@@ -5114,6 +5130,7 @@
             '*detect-zeroes': 'BlockdevDetectZeroesOptions' },
   'discriminator': 'driver',
   'data': {
+      'alloc-track':'BlockdevOptionsAllocTrack',
       'blkdebug':   'BlockdevOptionsBlkdebug',
       'blklogwrites':'BlockdevOptionsBlklogwrites',
       'blkverify':  'BlockdevOptionsBlkverify',
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu v3 04/51] block/qapi: include child references in block device info

[Fiona Ebner]

In combination with using a throttle filter to enforce IO limits for
a guest device, knowing the 'file' child of a block device can be
useful. If the throttle filter is only intended for guest IO, block
jobs should not also be limited by the throttle filter, so the
block operations need to be done with the 'file' child of the top
throttle node as the target. In combination with mirroring, the name
of that child is not fixed.

Another scenario is when unplugging a guest device after mirroring
below a top throttle node, where the mirror target is added explicitly
via blockdev-add. After mirroring, the target becomes the new 'file'
child of the throttle node. For unplugging, both the top throttle node
and the mirror target need to be deleted, because only implicitly
added child nodes are deleted automatically, and the current 'file'
child of the throttle node was explicitly added (as the mirror
target).

In other scenarios, it could be useful to follow the backing chain.

Note that iotests 191 and 273 use _filter_img_info, so the 'children'
information is filtered out there.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 block/qapi.c               | 10 ++++++++++
 qapi/block-core.json       | 16 ++++++++++++++++
 tests/qemu-iotests/184.out |  8 ++++++++
 3 files changed, 34 insertions(+)

diff --git a/block/qapi.c b/block/qapi.c
index 2c50a6bf3b..e08a1e970f 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -51,6 +51,8 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
     ImageInfo *backing_info;
     BlockDriverState *backing;
     BlockDeviceInfo *info;
+    BlockdevChildList **children_list_tail;
+    BdrvChild *child;
 
     if (!bs->drv) {
         error_setg(errp, "Block device %s is ejected", bs->node_name);
@@ -77,6 +79,14 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
         info->node_name = g_strdup(bs->node_name);
     }
 
+    children_list_tail = &info->children;
+    QLIST_FOREACH(child, &bs->children, next) {
+        BlockdevChild *child_ref = g_new0(BlockdevChild, 1);
+        child_ref->child = g_strdup(child->name);
+        child_ref->node_name = g_strdup(child->bs->node_name);
+        QAPI_LIST_APPEND(children_list_tail, child_ref);
+    }
+
     backing = bdrv_cow_bs(bs);
     if (backing) {
         info->backing_file = g_strdup(backing->filename);
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 07c5773717..4db27f5819 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -461,6 +461,19 @@
             'direct': 'bool',
             'no-flush': 'bool' } }
 
+##
+# @BlockdevChild:
+#
+# @child: The name of the child, for example 'file' or 'backing'.
+#
+# @node-name: The name of the child's block driver node.
+#
+# Since: 10.1
+##
+{ 'struct': 'BlockdevChild',
+  'data': { 'child': 'str',
+            'node-name': 'str' } }
+
 ##
 # @BlockDeviceInfo:
 #
@@ -486,6 +499,8 @@
 # @backing_file_depth: number of files in the backing file chain
 #     (since: 1.2)
 #
+# @children: Information about child block nodes. (since: 10.1)
+#
 # @active: true if the backend is active; typical cases for inactive backends
 #     are on the migration source instance after migration completes and on the
 #     destination before it completes. (since: 10.0)
@@ -560,6 +575,7 @@
 { 'struct': 'BlockDeviceInfo',
   'data': { 'file': 'str', '*node-name': 'str', 'ro': 'bool', 'drv': 'str',
             '*backing_file': 'str', 'backing_file_depth': 'int',
+            'children': ['BlockdevChild'],
             'active': 'bool', 'encrypted': 'bool',
             'detect_zeroes': 'BlockdevDetectZeroesOptions',
             'bps': 'int', 'bps_rd': 'int', 'bps_wr': 'int',
diff --git a/tests/qemu-iotests/184.out b/tests/qemu-iotests/184.out
index 52692b6b3b..ef99bb2e9a 100644
--- a/tests/qemu-iotests/184.out
+++ b/tests/qemu-iotests/184.out
@@ -41,6 +41,12 @@ Testing:
             },
             "iops_wr": 0,
             "ro": false,
+            "children": [
+                {
+                    "node-name": "disk0",
+                    "child": "file"
+                }
+            ],
             "node-name": "throttle0",
             "backing_file_depth": 1,
             "drv": "throttle",
@@ -69,6 +75,8 @@ Testing:
             },
             "iops_wr": 0,
             "ro": false,
+            "children": [
+            ],
             "node-name": "disk0",
             "backing_file_depth": 0,
             "drv": "null-co",
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 05/51] plugin: add method to get qemu blockdevice options for volume

[Fiona Ebner]

This is in preparation to switch qemu-server from using '-drive' to
the modern '-blockdev' in the QEMU commandline options as well as for
the qemu-storage-daemon, which only supports '-blockdev'. The plugins
know best what driver and options are needed to access an image, so
a dedicated plugin method returning the necessary parameters for
'-blockdev' is the most straight-forward.

There intentionally is only handling for absolute paths in the default
plugin implementation. Any plugin requiring more needs to implement
the method itself. With PVE 9 being a major release and most popular
plugins not using special protocols like 'rbd://', this seems
acceptable.

For NBD, etc. qemu-server should construct the blockdev object.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage.pm        | 17 ++++++++++++
 src/PVE/Storage/Plugin.pm | 56 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 73 insertions(+)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 69eb435..ec8b753 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -719,6 +719,23 @@ sub abs_filesystem_path {
     return $path;
 }
 
+# see the documentation for the plugin method
+sub qemu_blockdev_options {
+    my ($cfg, $volid) = @_;
+
+    my ($storeid, $volname) = parse_volume_id($volid);
+
+    my $scfg = storage_config($cfg, $storeid);
+
+    my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+    my ($vtype) = $plugin->parse_volname($volname);
+    die "cannot use volume of type '$vtype' as a QEMU blockdevice\n"
+        if $vtype ne 'images' && $vtype ne 'iso' && $vtype ne 'import';
+
+    return $plugin->qemu_blockdev_options($scfg, $storeid, $volname);
+}
+
 # used as last resort to adapt volnames when migrating
 my $volname_for_storage = sub {
     my ($cfg, $storeid, $name, $vmid, $format) = @_;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 53b9848..680bb6b 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1961,6 +1961,62 @@ sub rename_volume {
     return "${storeid}:${base}${target_vmid}/${target_volname}";
 }
 
+=pod
+
+=head3 qemu_blockdev_options
+
+    $blockdev = $plugin->qemu_blockdev_options($scfg, $storeid, $volname)
+
+Returns a hash reference with the basic options needed to open the volume via QEMU's C<-blockdev>
+API. This at least requires a C<< $blockdev->{driver} >> and a reference to the image, e.g.
+C<< $blockdev->{filename} >> for the C<file> driver. For files, the C<file> driver can be used. For
+host block devices, the C<host_device> driver can be used. The plugin must not set options like
+C<cache> or C<aio>. Those are managed by qemu-server and will be overwritten. For other available
+drivers and the exact specification of the options, see
+L<https://qemu.readthedocs.io/en/master/interop/qemu-qmp-ref.html#object-QMP-block-core.BlockdevOptions>
+
+While Perl does not have explicit types, the result will need to be converted to JSON later and
+match the QMP specification (see link above), so implicit types are important. In the return value,
+use C<JSON::true> and C<JSON::false> for booleans, C<"$value"> for strings, and C<int($value)> for
+integers.
+
+The volume is activated before the function is called.
+
+Arguments:
+
+=over
+
+=item C<$scfg>: The hash reference with the storage configuration.
+
+=item C<$storeid>: The storage ID.
+
+=item C<$volume>: The volume name.
+
+=back
+
+=cut
+
+sub qemu_blockdev_options {
+    my ($class, $scfg, $storeid, $volname) = @_;
+
+    my $blockdev = {};
+
+    my ($path) = $class->filesystem_path($scfg, $volname);
+
+    if ($path =~ m|^/|) {
+        # The 'file' driver only works for regular files. The check below is taken from
+        # block/file-posix.c:hdev_probe_device() in QEMU. Do not bother with detecting 'host_cdrom'
+        # devices here, those are not managed by the storage layer.
+        my $st = File::stat::stat($path) or die "stat for '$path' failed - $!\n";
+        my $driver = (S_ISCHR($st->mode) || S_ISBLK($st->mode)) ? 'host_device' : 'file';
+        $blockdev = { driver => $driver, filename => $path };
+    } else {
+        die "storage plugin doesn't implement qemu_blockdev_options() method\n";
+    }
+
+    return $blockdev;
+}
+
 # Used by storage plugins for external backup providers. See PVE::BackupProvider::Plugin for the API
 # the provider needs to implement.
 #
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 06/51] iscsi direct plugin: implement method to get qemu blockdevice options

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage/ISCSIDirectPlugin.pm | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/src/PVE/Storage/ISCSIDirectPlugin.pm b/src/PVE/Storage/ISCSIDirectPlugin.pm
index 9b7f77c..8c6b4ab 100644
--- a/src/PVE/Storage/ISCSIDirectPlugin.pm
+++ b/src/PVE/Storage/ISCSIDirectPlugin.pm
@@ -110,6 +110,20 @@ sub path {
     return ($path, $vmid, $vtype);
 }
 
+sub qemu_blockdev_options {
+    my ($class, $scfg, $storeid, $volname) = @_;
+
+    my $lun = ($class->parse_volname($volname))[1];
+
+    return {
+        driver => 'iscsi',
+        transport => 'tcp',
+        portal => "$scfg->{portal}",
+        target => "$scfg->{target}",
+        lun => int($lun),
+    };
+}
+
 sub create_base {
     my ($class, $storeid, $scfg, $volname) = @_;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 07/51] zfs iscsi plugin: implement new method to get qemu blockdevice options

[Fiona Ebner]

Reported-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage/ZFSPlugin.pm | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
index f0fa522..c03fcca 100644
--- a/src/PVE/Storage/ZFSPlugin.pm
+++ b/src/PVE/Storage/ZFSPlugin.pm
@@ -247,6 +247,22 @@ sub path {
     return ($path, $vmid, $vtype);
 }
 
+sub qemu_blockdev_options {
+    my ($class, $scfg, $storeid, $volname) = @_;
+
+    my $name = ($class->parse_volname($volname))[1];
+    my $guid = $class->zfs_get_lu_name($scfg, $name);
+    my $lun = $class->zfs_get_lun_number($scfg, $guid);
+
+    return {
+        driver => 'iscsi',
+        transport => 'tcp',
+        portal => "$scfg->{portal}",
+        target => "$scfg->{target}",
+        lun => int($lun),
+    };
+}
+
 sub create_base {
     my ($class, $storeid, $scfg, $volname) = @_;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 08/51] zfs pool plugin: implement method to get qemu blockdevice options

[Fiona Ebner]

ZFS does not have a filesystem_path() method, so the default
implementation for qemu_blockdev_options() cannot be re-used. This is
most likely, because snapshots are currently not directly accessible
via a filesystem path in the Proxmox VE storage layer.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage/ZFSPoolPlugin.pm | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/src/PVE/Storage/ZFSPoolPlugin.pm b/src/PVE/Storage/ZFSPoolPlugin.pm
index 713d26f..86c52ea 100644
--- a/src/PVE/Storage/ZFSPoolPlugin.pm
+++ b/src/PVE/Storage/ZFSPoolPlugin.pm
@@ -162,6 +162,20 @@ sub path {
     return ($path, $vmid, $vtype);
 }
 
+sub qemu_blockdev_options {
+    my ($class, $scfg, $storeid, $volname) = @_;
+
+    my $format = ($class->parse_volname($volname))[6];
+
+    die "volume '$volname' not usable as VM image\n" if $format ne 'raw';
+
+    my ($path) = $class->path($scfg, $volname, $storeid);
+
+    my $blockdev = { driver => 'host_device', filename => $path };
+
+    return $blockdev;
+}
+
 sub zfs_request {
     my ($class, $scfg, $timeout, $method, @params) = @_;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 09/51] ceph/rbd: set 'keyring' in ceph configuration for externally managed RBD storages

[Fiona Ebner]

For QEMU, when using '-blockdev', there is no way to specify the
keyring file like was possible with '-drive', so it has to be set in
the corresponding Ceph configuration file. As it applies to all images
on the storage, it also is the most natural place for the setting.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

NOTE: This should also be mentioned in the upgrade guide for PVE 9 and
the pve8to9 script should tell the user and/or automatically set it
for existing externally managed RBD storages, that already do have a
custom configuration.

 src/PVE/CephConfig.pm        | 50 ++++++++++++++++++++++++++++++++++++
 src/PVE/Storage/RBDPlugin.pm |  3 +++
 2 files changed, 53 insertions(+)

diff --git a/src/PVE/CephConfig.pm b/src/PVE/CephConfig.pm
index 5347781..e5815c4 100644
--- a/src/PVE/CephConfig.pm
+++ b/src/PVE/CephConfig.pm
@@ -3,6 +3,8 @@ package PVE::CephConfig;
 use strict;
 use warnings;
 use Net::IP;
+
+use PVE::RESTEnvironment qw(log_warn);
 use PVE::Tools qw(run_command);
 use PVE::Cluster qw(cfs_register_file);
 
@@ -420,6 +422,10 @@ sub ceph_connect_option {
         } else {
             $cmd_option->{ceph_conf} = "/etc/pve/priv/ceph/${storeid}.conf";
         }
+    } elsif (!$pveceph_managed) {
+        # No dedicated config for non-PVE-managed cluster, create new
+        # TODO PVE 10 - remove. All such storages already got a configuration upon creation or here.
+        ceph_create_configuration($scfg->{type}, $storeid);
     }
 
     $cmd_option->{keyring} = $keyfile if (-e $keyfile);
@@ -487,6 +493,50 @@ sub ceph_remove_keyfile {
     }
 }
 
+sub ceph_create_configuration {
+    my ($type, $storeid) = @_;
+
+    return if $type eq 'cephfs'; # no configuration file needed currently
+
+    my $extension = 'keyring';
+    $extension = 'secret' if $type eq 'cephfs';
+    my $ceph_storage_keyring = "/etc/pve/priv/ceph/${storeid}.$extension";
+
+    return if !-e $ceph_storage_keyring;
+
+    my $ceph_storage_config = "/etc/pve/priv/ceph/${storeid}.conf";
+
+    if (-e $ceph_storage_config) {
+        log_warn(
+            "file $ceph_storage_config already exists, check manually and ensure 'keyring'"
+                . " option is set to '$ceph_storage_keyring'!\n",
+        );
+        return;
+    }
+
+    my $ceph_config = {
+        global => {
+            keyring => $ceph_storage_keyring,
+        },
+    };
+
+    my $contents = PVE::CephConfig::write_ceph_config($ceph_storage_config, $ceph_config);
+    PVE::Tools::file_set_contents($ceph_storage_config, $contents, 0600);
+
+    return;
+}
+
+sub ceph_remove_configuration {
+    my ($storeid) = @_;
+
+    my $ceph_storage_config = "/etc/pve/priv/ceph/${storeid}.conf";
+    if (-f $ceph_storage_config) {
+        unlink $ceph_storage_config or log_warn("removing $ceph_storage_config failed - $!\n");
+    }
+
+    return;
+}
+
 my $ceph_version_parser = sub {
     my $ceph_version = shift;
     # FIXME this is the same as pve-manager PVE::Ceph::Tools get_local_version
diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index c0bbe2c..3f7ca9f 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -448,6 +448,7 @@ sub on_add_hook {
     my ($class, $storeid, $scfg, %param) = @_;
 
     PVE::CephConfig::ceph_create_keyfile($scfg->{type}, $storeid, $param{keyring});
+    PVE::CephConfig::ceph_create_configuration($scfg->{type}, $storeid);
 
     return;
 }
@@ -469,6 +470,8 @@ sub on_update_hook {
 sub on_delete_hook {
     my ($class, $storeid, $scfg) = @_;
     PVE::CephConfig::ceph_remove_keyfile($scfg->{type}, $storeid);
+    PVE::CephConfig::ceph_remove_configuration($storeid);
+
     return;
 }
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 10/51] rbd plugin: implement new method to get qemu blockdevice options

[Fiona Ebner]

Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage/RBDPlugin.pm | 37 ++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index 3f7ca9f..6e70a19 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -513,6 +513,43 @@ sub path {
     return ($path, $vmid, $vtype);
 }
 
+sub qemu_blockdev_options {
+    my ($class, $scfg, $storeid, $volname) = @_;
+
+    my $cmd_option = PVE::CephConfig::ceph_connect_option($scfg, $storeid);
+    my ($name) = ($class->parse_volname($volname))[1];
+
+    if ($scfg->{krbd}) {
+        my $rbd_dev_path = get_rbd_dev_path($scfg, $storeid, $name);
+        return { driver => 'host_device', filename => $rbd_dev_path };
+    }
+
+    my $blockdev = {
+        driver => 'rbd',
+        pool => $scfg->{pool} ? "$scfg->{pool}" : 'rbd',
+        image => "$name",
+    };
+    $blockdev->{namespace} = "$scfg->{namespace}" if defined($scfg->{namespace});
+
+    $blockdev->{conf} = $cmd_option->{ceph_conf} if $cmd_option->{ceph_conf};
+
+    if (my $monhost = $scfg->{'monhost'}) {
+        my $server = [];
+        my @mons = PVE::Tools::split_list($monhost);
+        for my $mon (@mons) {
+            my ($host, $port) = PVE::Tools::parse_host_and_port($mon);
+            $port = '3300' if !$port;
+            push @$server, { host => $host, port => $port };
+        }
+        $blockdev->{server} = $server;
+        $blockdev->{'auth-client-required'} = ["$cmd_option->{auth_supported}"];
+    }
+
+    $blockdev->{user} = "$cmd_option->{userid}" if $cmd_option->{keyring};
+
+    return $blockdev;
+}
+
 sub find_free_diskname {
     my ($class, $storeid, $scfg, $vmid, $fmt, $add_fmt_suffix) = @_;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 11/51] plugin: qemu block device: add hints option and EFI disk hint

[Fiona Ebner]

For '-drive', qemu-server sets special cache options for EFI disk
using RBD. In preparation to seamlessly switch to the new '-blockdev'
interface, do the same here. Note that the issue from bug #3329, which
is solved by these cache options, still affects current versions.

With -blockdev, the cache options are split up. While cache.direct and
cache.no-flush can be set in the -blockdev options, cache.writeback is
a front-end property and was intentionally removed from the -blockdev
options by QEMU commit aaa436f998 ("block: Remove cache.writeback from
blockdev-add"). It needs to be configured as the 'write-cache'
property for the ide-hd/scsi-hd/virtio-blk device.

The default is already 'writeback' and no cache mode can be set for an
EFI drive configuration in Proxmox VE currently, so there will not be
a clash.

┌─────────────┬─────────────────┬──────────────┬────────────────┐
│             │ cache.writeback │ cache.direct │ cache.no-flush │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│writeback    │ on              │ off          │ off            │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│none         │ on              │ on           │ off            │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│writethrough │ off             │ off          │ off            │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│directsync   │ off             │ on           │ off            │
├─────────────┼─────────────────┼──────────────┼────────────────┤
│unsafe       │ on              │ off          │ on             │
└─────────────┴─────────────────┴──────────────┴────────────────┘

Table from 'man kvm'.

Alternatively, the option could only be set once when allocating the
RBD volume. However, then we would need to detect all cases were a
volume could potentially be used as an EFI disk later. Having a custom
disk type would help a lot there. The approach here was chosen as it
is catch-all and should not be too costly either.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage.pm                   |  4 ++--
 src/PVE/Storage/ISCSIDirectPlugin.pm |  2 +-
 src/PVE/Storage/Plugin.pm            | 23 +++++++++++++++++++++--
 src/PVE/Storage/RBDPlugin.pm         | 20 +++++++++++++++++++-
 src/PVE/Storage/ZFSPlugin.pm         |  2 +-
 src/PVE/Storage/ZFSPoolPlugin.pm     |  2 +-
 6 files changed, 45 insertions(+), 8 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index ec8b753..4920bd6 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -721,7 +721,7 @@ sub abs_filesystem_path {
 
 # see the documentation for the plugin method
 sub qemu_blockdev_options {
-    my ($cfg, $volid) = @_;
+    my ($cfg, $volid, $options) = @_;
 
     my ($storeid, $volname) = parse_volume_id($volid);
 
@@ -733,7 +733,7 @@ sub qemu_blockdev_options {
     die "cannot use volume of type '$vtype' as a QEMU blockdevice\n"
         if $vtype ne 'images' && $vtype ne 'iso' && $vtype ne 'import';
 
-    return $plugin->qemu_blockdev_options($scfg, $storeid, $volname);
+    return $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $options);
 }
 
 # used as last resort to adapt volnames when migrating
diff --git a/src/PVE/Storage/ISCSIDirectPlugin.pm b/src/PVE/Storage/ISCSIDirectPlugin.pm
index 8c6b4ab..12b894d 100644
--- a/src/PVE/Storage/ISCSIDirectPlugin.pm
+++ b/src/PVE/Storage/ISCSIDirectPlugin.pm
@@ -111,7 +111,7 @@ sub path {
 }
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname) = @_;
+    my ($class, $scfg, $storeid, $volname, $options) = @_;
 
     my $lun = ($class->parse_volname($volname))[1];
 
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 680bb6b..4ec4a0d 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1965,7 +1965,7 @@ sub rename_volume {
 
 =head3 qemu_blockdev_options
 
-    $blockdev = $plugin->qemu_blockdev_options($scfg, $storeid, $volname)
+    $blockdev = $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $options)
 
 Returns a hash reference with the basic options needed to open the volume via QEMU's C<-blockdev>
 API. This at least requires a C<< $blockdev->{driver} >> and a reference to the image, e.g.
@@ -1992,12 +1992,31 @@ Arguments:
 
 =item C<$volume>: The volume name.
 
+=item C<$options>: A hash reference with additional options.
+
+=over
+
+=item C<< $options->{hints} >>: A hash reference with hints indicating what the volume will be used
+for. This can be safely ignored if no concrete issues are known with your plugin. For certain use
+cases, setting additional (plugin-specific) options might be very beneficial however. An example is
+setting the correct cache options for an EFI disk on RBD. The list of hints might get expanded in
+the future.
+
+=over
+
+=item C<< $options->{hints}->{'efi-disk'} >>: (optional) If set, the volume will be used as the EFI
+disk of a VM, containing its OMVF variables.
+
+=back
+
+=back
+
 =back
 
 =cut
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname) = @_;
+    my ($class, $scfg, $storeid, $volname, $options) = @_;
 
     my $blockdev = {};
 
diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index 6e70a19..c40c845 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -374,6 +374,16 @@ my sub rbd_volume_exists {
     return 0;
 }
 
+# Needs to be public, so qemu-server can mock it for cfg2cmd.
+sub rbd_volume_config_set {
+    my ($scfg, $storeid, $volname, $key, $value) = @_;
+
+    my $cmd = $rbd_cmd->($scfg, $storeid, 'config', 'image', 'set', $volname, $key, $value);
+    run_rbd_command($cmd, errmsg => "rbd config image set $volname $key $value error");
+
+    return;
+}
+
 # Configuration
 
 sub type {
@@ -514,7 +524,7 @@ sub path {
 }
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname) = @_;
+    my ($class, $scfg, $storeid, $volname, $options) = @_;
 
     my $cmd_option = PVE::CephConfig::ceph_connect_option($scfg, $storeid);
     my ($name) = ($class->parse_volname($volname))[1];
@@ -547,6 +557,14 @@ sub qemu_blockdev_options {
 
     $blockdev->{user} = "$cmd_option->{userid}" if $cmd_option->{keyring};
 
+    # SPI flash does lots of read-modify-write OPs, without writeback this gets really slow #3329
+    if ($options->{hints}->{'efi-disk'}) {
+        # Querying the value would just cost more and the 'rbd image config get' command will just
+        # fail if the config has not been set yet, so it's not even straight-forward to do so.
+        # Simply set the value (possibly again).
+        rbd_volume_config_set($scfg, $storeid, $name, 'rbd_cache_policy', 'writeback');
+    }
+
     return $blockdev;
 }
 
diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
index c03fcca..0f64898 100644
--- a/src/PVE/Storage/ZFSPlugin.pm
+++ b/src/PVE/Storage/ZFSPlugin.pm
@@ -248,7 +248,7 @@ sub path {
 }
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname) = @_;
+    my ($class, $scfg, $storeid, $volname, $options) = @_;
 
     my $name = ($class->parse_volname($volname))[1];
     my $guid = $class->zfs_get_lu_name($scfg, $name);
diff --git a/src/PVE/Storage/ZFSPoolPlugin.pm b/src/PVE/Storage/ZFSPoolPlugin.pm
index 86c52ea..e775dae 100644
--- a/src/PVE/Storage/ZFSPoolPlugin.pm
+++ b/src/PVE/Storage/ZFSPoolPlugin.pm
@@ -163,7 +163,7 @@ sub path {
 }
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname) = @_;
+    my ($class, $scfg, $storeid, $volname, $options) = @_;
 
     my $format = ($class->parse_volname($volname))[6];
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 12/51] plugin: qemu block device: add support for snapshot option

[Fiona Ebner]

This is mostly in preparation for external qcow2 snapshot support.

For internal qcow2 snapshots, which currently are the only supported
variant, it is not possible to attach the snapshot only. If access to
that is required it will need to be handled differently, e.g. via a
FUSE/NBD export.

Such accesses are currently not done for running VMs via '-drive'
either, so there still is feature parity.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage/ISCSIDirectPlugin.pm |  3 +++
 src/PVE/Storage/Plugin.pm            | 11 ++++++++++-
 src/PVE/Storage/RBDPlugin.pm         |  2 ++
 src/PVE/Storage/ZFSPlugin.pm         |  3 +++
 src/PVE/Storage/ZFSPoolPlugin.pm     |  2 ++
 5 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/src/PVE/Storage/ISCSIDirectPlugin.pm b/src/PVE/Storage/ISCSIDirectPlugin.pm
index 12b894d..e0f8a62 100644
--- a/src/PVE/Storage/ISCSIDirectPlugin.pm
+++ b/src/PVE/Storage/ISCSIDirectPlugin.pm
@@ -113,6 +113,9 @@ sub path {
 sub qemu_blockdev_options {
     my ($class, $scfg, $storeid, $volname, $options) = @_;
 
+    die "volume snapshot is not possible on iscsi device\n"
+        if $options->{'snapshot-name'};
+
     my $lun = ($class->parse_volname($volname))[1];
 
     return {
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 4ec4a0d..52652b2 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -2009,6 +2009,9 @@ disk of a VM, containing its OMVF variables.
 
 =back
 
+=item C<< $options->{'snapshot-name'} >>: (optional) The snapshot name. Set when the associated snapshot should be opened rather than the
+volume itself.
+
 =back
 
 =back
@@ -2020,9 +2023,15 @@ sub qemu_blockdev_options {
 
     my $blockdev = {};
 
-    my ($path) = $class->filesystem_path($scfg, $volname);
+    my ($path) = $class->filesystem_path($scfg, $volname, $options->{'snapshot-name'});
 
     if ($path =~ m|^/|) {
+        # For qcow2 and qed the path of a snapshot will be the same, but it's not possible to attach
+        # the snapshot alone.
+        my $format = ($class->parse_volname($volname))[6];
+        die "cannot attach only the snapshot of a '$format' image\n"
+            if $options->{'snapshot-name'} && ($format eq 'qcow2' || $format eq 'qed');
+
         # The 'file' driver only works for regular files. The check below is taken from
         # block/file-posix.c:hdev_probe_device() in QEMU. Do not bother with detecting 'host_cdrom'
         # devices here, those are not managed by the storage layer.
diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index c40c845..800af68 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -530,6 +530,7 @@ sub qemu_blockdev_options {
     my ($name) = ($class->parse_volname($volname))[1];
 
     if ($scfg->{krbd}) {
+        $name .= '@' . $options->{'snapshot-name'} if $options->{'snapshot-name'};
         my $rbd_dev_path = get_rbd_dev_path($scfg, $storeid, $name);
         return { driver => 'host_device', filename => $rbd_dev_path };
     }
@@ -540,6 +541,7 @@ sub qemu_blockdev_options {
         image => "$name",
     };
     $blockdev->{namespace} = "$scfg->{namespace}" if defined($scfg->{namespace});
+    $blockdev->{snapshot} = $options->{'snapshot-name'} if $options->{'snapshot-name'};
 
     $blockdev->{conf} = $cmd_option->{ceph_conf} if $cmd_option->{ceph_conf};
 
diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
index 0f64898..940d4f0 100644
--- a/src/PVE/Storage/ZFSPlugin.pm
+++ b/src/PVE/Storage/ZFSPlugin.pm
@@ -250,6 +250,9 @@ sub path {
 sub qemu_blockdev_options {
     my ($class, $scfg, $storeid, $volname, $options) = @_;
 
+    die "direct access to snapshots not implemented\n"
+        if $options->{'snapshot-name'};
+
     my $name = ($class->parse_volname($volname))[1];
     my $guid = $class->zfs_get_lu_name($scfg, $name);
     my $lun = $class->zfs_get_lun_number($scfg, $guid);
diff --git a/src/PVE/Storage/ZFSPoolPlugin.pm b/src/PVE/Storage/ZFSPoolPlugin.pm
index e775dae..d5552ab 100644
--- a/src/PVE/Storage/ZFSPoolPlugin.pm
+++ b/src/PVE/Storage/ZFSPoolPlugin.pm
@@ -169,6 +169,8 @@ sub qemu_blockdev_options {
 
     die "volume '$volname' not usable as VM image\n" if $format ne 'raw';
 
+    die "cannot attach only the snapshot of a zvol\n" if $options->{'snapshot-name'};
+
     my ($path) = $class->path($scfg, $volname, $storeid);
 
     my $blockdev = { driver => 'host_device', filename => $path };
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 13/51] plugin: add machine version to qemu_blockdev_options() interface

[Fiona Ebner]

Plugins can guard based on the machine version to be able to switch
drivers or options in a safe way without the risk of breaking older
versions.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage.pm                   |  4 ++--
 src/PVE/Storage/ISCSIDirectPlugin.pm |  2 +-
 src/PVE/Storage/Plugin.pm            | 12 ++++++++++--
 src/PVE/Storage/RBDPlugin.pm         |  2 +-
 src/PVE/Storage/ZFSPlugin.pm         |  2 +-
 src/PVE/Storage/ZFSPoolPlugin.pm     |  2 +-
 6 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 4920bd6..5afff26 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -721,7 +721,7 @@ sub abs_filesystem_path {
 
 # see the documentation for the plugin method
 sub qemu_blockdev_options {
-    my ($cfg, $volid, $options) = @_;
+    my ($cfg, $volid, $machine_version, $options) = @_;
 
     my ($storeid, $volname) = parse_volume_id($volid);
 
@@ -733,7 +733,7 @@ sub qemu_blockdev_options {
     die "cannot use volume of type '$vtype' as a QEMU blockdevice\n"
         if $vtype ne 'images' && $vtype ne 'iso' && $vtype ne 'import';
 
-    return $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $options);
+    return $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $machine_version, $options);
 }
 
 # used as last resort to adapt volnames when migrating
diff --git a/src/PVE/Storage/ISCSIDirectPlugin.pm b/src/PVE/Storage/ISCSIDirectPlugin.pm
index e0f8a62..62e9026 100644
--- a/src/PVE/Storage/ISCSIDirectPlugin.pm
+++ b/src/PVE/Storage/ISCSIDirectPlugin.pm
@@ -111,7 +111,7 @@ sub path {
 }
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname, $options) = @_;
+    my ($class, $scfg, $storeid, $volname, $machine_version, $options) = @_;
 
     die "volume snapshot is not possible on iscsi device\n"
         if $options->{'snapshot-name'};
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 52652b2..cfe89dd 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1965,7 +1965,8 @@ sub rename_volume {
 
 =head3 qemu_blockdev_options
 
-    $blockdev = $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $options)
+    $blockdev =
+        $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $machine_version, $options)
 
 Returns a hash reference with the basic options needed to open the volume via QEMU's C<-blockdev>
 API. This at least requires a C<< $blockdev->{driver} >> and a reference to the image, e.g.
@@ -1992,6 +1993,13 @@ Arguments:
 
 =item C<$volume>: The volume name.
 
+=item C<$machine_version>: The QEMU machine version for which the block device will be used. If you
+want to change drivers or change driver options, you should use this as a guard, so that only
+machines with a new enough version will use the new driver or options. Machines with an older
+version should still get the old driver and options. The version is a string
+C<${major}.${minor}+pve${pve_version}>. The pve version is used for certain downstream changes to
+machine models and should be (mostly) irrelevant for third-party plugins.
+
 =item C<$options>: A hash reference with additional options.
 
 =over
@@ -2019,7 +2027,7 @@ volume itself.
 =cut
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname, $options) = @_;
+    my ($class, $scfg, $storeid, $volname, $machine_version, $options) = @_;
 
     my $blockdev = {};
 
diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index 800af68..31cbf8f 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -524,7 +524,7 @@ sub path {
 }
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname, $options) = @_;
+    my ($class, $scfg, $storeid, $volname, $machine_version, $options) = @_;
 
     my $cmd_option = PVE::CephConfig::ceph_connect_option($scfg, $storeid);
     my ($name) = ($class->parse_volname($volname))[1];
diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
index 940d4f0..eed39cd 100644
--- a/src/PVE/Storage/ZFSPlugin.pm
+++ b/src/PVE/Storage/ZFSPlugin.pm
@@ -248,7 +248,7 @@ sub path {
 }
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname, $options) = @_;
+    my ($class, $scfg, $storeid, $volname, $machine_version, $options) = @_;
 
     die "direct access to snapshots not implemented\n"
         if $options->{'snapshot-name'};
diff --git a/src/PVE/Storage/ZFSPoolPlugin.pm b/src/PVE/Storage/ZFSPoolPlugin.pm
index d5552ab..979cf2c 100644
--- a/src/PVE/Storage/ZFSPoolPlugin.pm
+++ b/src/PVE/Storage/ZFSPoolPlugin.pm
@@ -163,7 +163,7 @@ sub path {
 }
 
 sub qemu_blockdev_options {
-    my ($class, $scfg, $storeid, $volname, $options) = @_;
+    my ($class, $scfg, $storeid, $volname, $machine_version, $options) = @_;
 
     my $format = ($class->parse_volname($volname))[6];
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 14/51] qemu blockdev options: restrict allowed drivers and options

[Fiona Ebner]

Everything the default plugin method implementation can return is
allowed, so there is no breakage introduced by this patch.

By far the most common drivers will be 'file' and 'host_device', which
the default implementation of the plugin method currently uses. Other
quite common ones will be 'iscsi' and 'nbd'. There might also be
plugins with 'rbd' and it is planned to support QEMU protocol-paths in
the default plugin method implementation, where the 'rbd:' protocol
will also be supported.

Plugin authors are encouraged to request additional drivers and
options based on their needs on the pve-devel mailing list. The list
just starts out more restrictive, but everything where there is no
good reason to not allow could be allowed in the future upon request.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

New in v5.

 src/PVE/Storage.pm        | 116 +++++++++++++++++++++++++++++++++++++-
 src/PVE/Storage/Plugin.pm |   6 +-
 2 files changed, 118 insertions(+), 4 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 5afff26..25ce5f2 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -131,6 +131,102 @@ our $OVA_CONTENT_RE_1 = qr/${SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+\.(qcow2|raw|vm
 # FIXME remove with PVE 9.0, add versioned breaks for pve-manager
 our $vztmpl_extension_re = $VZTMPL_EXT_RE_1;
 
+# See the QMP reference documentation.
+my $allowed_qemu_blockdev_options_file = {
+    filename => 1,
+    # pr-manager
+    # aio
+    # aio-max-batch
+    # locking
+    # drop-cache
+    # x-check-cache-dropped
+};
+
+# Plugin authors should feel free to request allowing more based on their requirements on the
+# pve-devel mailing list. See the QMP reference documentation:
+# https://qemu.readthedocs.io/en/master/interop/qemu-qmp-ref.html#object-QMP-block-core.BlockdevOptions
+my $allowed_qemu_blockdev_options = {
+    # alloc-track - only works in combination with stream job
+    # blkdebug - for debugging
+    # blklogwrites - for debugging
+    # blkreplay - for debugging
+    # blkverify - for debugging
+    # bochs
+    # cloop
+    # compress
+    # copy-before-write - should not be used directly by storage layer
+    # copy-on-read - should not be used directly by storage layer
+    # dmg
+    file => $allowed_qemu_blockdev_options_file,
+    # snapshot-access - should not be used directly by storage layer
+    # ftp
+    # ftps
+    # gluster - support is expected to be dropped in QEMU 10.1
+    # host_cdrom - storage layer should not access host CD-ROM drive
+    host_device => $allowed_qemu_blockdev_options_file,
+    # http
+    # https
+    # io_uring - disabled by our QEMU build config (would require CONFIG_BLKIO)
+    iscsi => {
+        transport => 1,
+        portal => 1,
+        target => 1,
+        lun => 1,
+        # user - requires 'password-secret'
+        # password-secret - requires adding a 'secret' object on the commandline in qemu-server
+        'initiator-name' => 1,
+        'header-digest' => 1,
+        timeout => 1,
+    },
+    # luks
+    nbd => {
+        server => 1,
+        export => 1,
+        # tls-creds - would require adding a 'secret' object on the commandline in qemu-server
+        # tls-hostname - requires tls-creds
+        # x-dirty-bitmap - would mean allocation information would be reported based on bitmap
+        'reconnect-delay' => 1,
+        'open-timeout' => 1,
+    },
+    # nfs - disabled by our QEMU build config
+    # null-aio - for debugging
+    # null-co - for debugging
+    # nvme
+    # nvme-io_uring - disabled by our QEMU build config (would require CONFIG_BLKIO)
+    # parallels
+    # preallocate
+    # qcow
+    # qcow2 - format node is added by qemu-server
+    # qed
+    # quorum
+    # raw - format node is added by qemu-server
+    rbd => {
+        pool => 1,
+        namespace => 1,
+        image => 1,
+        conf => 1,
+        snapshot => 1,
+        encrypt => 1,
+        user => 1,
+        'auth-client-required' => 1,
+        # key-secret would require adding a 'secret' object on the commandline in qemu-server
+        server => 1,
+    },
+    # replication
+    # pbs
+    # ssh - disabled by our QEMU build config
+    # throttle
+    # vdi
+    # vhdx
+    # virtio-blk-vfio-pci - disabled by our QEMU build config (would require CONFIG_BLKIO)
+    # virtio-blk-vhost-user - disabled by our QEMU build config (would require CONFIG_BLKIO)
+    # virtio-blk-vhost-vdpa - disabled by our QEMU build config (would require CONFIG_BLKIO)
+    # vmdk - format node is added by qemu-server
+    # vpc
+    # vvfat
+    # zeroinit - filter that should not be used directly by storage layer
+};
+
 #  PVE::Storage utility functions
 
 sub config {
@@ -733,7 +829,25 @@ sub qemu_blockdev_options {
     die "cannot use volume of type '$vtype' as a QEMU blockdevice\n"
         if $vtype ne 'images' && $vtype ne 'iso' && $vtype ne 'import';
 
-    return $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $machine_version, $options);
+    my $blockdev =
+        $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $machine_version, $options);
+
+    if (my $driver = $blockdev->{driver}) {
+        my $allowed_opts = $allowed_qemu_blockdev_options->{$driver};
+        for my $opt (keys $blockdev->%*) {
+            next if $opt eq 'driver';
+            if (!$allowed_opts->{$opt}) {
+                delete($blockdev->{$opt});
+                log_warn(
+                    "volume '$volid' - dropping block device option '$opt' set by storage plugin"
+                        . " - not currently part of allowed schema");
+            }
+        }
+    } else {
+        die "storage plugin for '$storeid' did not return a blockdev driver\n";
+    }
+
+    return $blockdev;
 }
 
 # used as last resort to adapt volnames when migrating
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index cfe89dd..3f2c638 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1972,9 +1972,9 @@ Returns a hash reference with the basic options needed to open the volume via QE
 API. This at least requires a C<< $blockdev->{driver} >> and a reference to the image, e.g.
 C<< $blockdev->{filename} >> for the C<file> driver. For files, the C<file> driver can be used. For
 host block devices, the C<host_device> driver can be used. The plugin must not set options like
-C<cache> or C<aio>. Those are managed by qemu-server and will be overwritten. For other available
-drivers and the exact specification of the options, see
-L<https://qemu.readthedocs.io/en/master/interop/qemu-qmp-ref.html#object-QMP-block-core.BlockdevOptions>
+C<cache> or C<aio>. Those are managed by qemu-server. See C<$allowed_qemu_blockdev_options> in the
+C<PVE/Storage.pm> module for allowed drivers and options. Feel free to request more on the pve-devel
+mailing list based on your requirements.
 
 While Perl does not have explicit types, the result will need to be converted to JSON later and
 match the QMP specification (see link above), so implicit types are important. In the return value,
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 15/51] plugin: qemu blockdev options: parse protocol paths in default implementation

[Fiona Ebner]

for better backwards compatibility. This also means using path()
rather than filesystem_path() as the latter does not return protocol
paths.

Some protocol paths are not implemented (considered all that are
listed by grepping for '\.protocol_name' in QEMU):
- ftp(s)/http(s), which would access web servers via curl. This one
  could be added if there is enough interest.
- nvme://XXXX:XX:XX.X/X, which would access a host NVME device.
- null-{aio,co}, which are mainly useful for debugging.
- pbs, because path-based access is not used anymore for PBS,
  live-restore in qemu-server already defines a driver-based device.
- nfs and ssh, because the QEMU build script used by Proxmox VE does
  not enable them.
- blk{debug,verify}, because they are for debugging.
- the ones used by blkio, i.e. io_uring, nvme-io_uring,
  virtio-blk-vfio-pci, virtio-blk-vhost-user and
  virtio-blk-vhost-vdpa, because the QEMU build script used by Proxmox
  VE does not enable blkio.
- backup-dump and zeroinit, because they should not be used by the
  storage layer directly.
- gluster, because support is dropped in Proxmox VE 9.
- host_cdrom, because the storage layer should not access host CD-ROM
  devices.
- fat, because it hopefully isn't used by any third-party plugin here.

Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

New in v5.

 src/PVE/Storage/Plugin.pm | 95 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 94 insertions(+), 1 deletion(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 3f2c638..c2f376b 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1961,6 +1961,34 @@ sub rename_volume {
     return "${storeid}:${base}${target_vmid}/${target_volname}";
 }
 
+my sub blockdev_options_nbd_tcp {
+    my ($host, $port, $export) = @_;
+
+    die "blockdev_options_nbd_tcp: no host" if !defined($host);
+
+    my $blockdev = {};
+
+    my $server = { type => 'inet', host => "$host" };
+    # port is also a string in QAPI, not optional, default for NBD is 10809
+    $server->{port} = defined($port) ? "$port" : "10809";
+    $blockdev = { driver => 'nbd', server => $server };
+    $blockdev->{export} = "$export" if defined($export);
+
+    return $blockdev;
+}
+
+my sub blockdev_options_nbd_unix {
+    my ($socket_path, $export) = @_;
+
+    my $blockdev = {};
+
+    my $server = { type => 'unix', path => "$socket_path" };
+    $blockdev = { driver => 'nbd', server => $server };
+    $blockdev->{export} = "$export" if defined($export);
+
+    return $blockdev;
+}
+
 =pod
 
 =head3 qemu_blockdev_options
@@ -2031,7 +2059,7 @@ sub qemu_blockdev_options {
 
     my $blockdev = {};
 
-    my ($path) = $class->filesystem_path($scfg, $volname, $options->{'snapshot-name'});
+    my ($path) = $class->path($scfg, $volname, $storeid, $options->{'snapshot-name'});
 
     if ($path =~ m|^/|) {
         # For qcow2 and qed the path of a snapshot will be the same, but it's not possible to attach
@@ -2046,6 +2074,71 @@ sub qemu_blockdev_options {
         my $st = File::stat::stat($path) or die "stat for '$path' failed - $!\n";
         my $driver = (S_ISCHR($st->mode) || S_ISBLK($st->mode)) ? 'host_device' : 'file';
         $blockdev = { driver => $driver, filename => $path };
+    } elsif ($path =~ m|^file:(\S+)|) {
+        $blockdev = { driver => 'file', filename => "$1" };
+    } elsif ($path =~ m|^host_device:(\S+)|) {
+        $blockdev = { driver => 'host_device', filename => "$1" };
+    } elsif ($path =~ m|^iscsi://(\S+)/(\S+)/(\d+)$|) {
+        $blockdev =
+            { driver => 'iscsi', portal => "$1", target => "$2", lun => "$3", transport => "tcp" };
+    } elsif ($path =~ m|^iser://(\S+)/(\S+)/(\d+)$|) {
+        $blockdev =
+            { driver => 'iscsi', portal => "$1", target => "$2", lun => "$3", transport => "iser" };
+    } elsif ($path =~ m|^nbd(?:\+tcp)?://(\S+?)(?::(\d+))?/(\S+)?$|) { # new style NBD TCP URI
+        $blockdev = blockdev_options_nbd_tcp($1, $2, $3);
+    } elsif ($path =~ m|^nbd(?:\+tcp)?:(\S+):(\d+)(?::exportname=(\S+))?$|) {
+        # old style NBD TCP URI
+        $blockdev = blockdev_options_nbd_tcp($1, $2, $3);
+    } elsif ($path =~ m|^nbd\+unix:///(\S+)?\?socket=(\S+)$|) { # new style NBD unix URI
+        $blockdev = blockdev_options_nbd_unix($2, $1); # note the order!
+    } elsif ($path =~ m|^nbd:unix:(\S+?)(?::exportname=(\S+))?$|) { # old style NBD unix URI
+        $blockdev = blockdev_options_nbd_unix($1, $2);
+    } elsif ($path =~ m/^rbd:(\S+)$/) {
+        my $rbd_options = $1;
+        $blockdev->{driver} = 'rbd';
+
+        #map options to key=value pair (if not key is provided, this is the image)
+        #options are seprated with : but we need to exclude \: used for ipv6 address
+        my $options = {
+            map {
+                s/\\:/:/g;
+                /^(.*?)=(.*)/ ? ($1 => $2) : (image => $_)
+            } $rbd_options =~ /(?:\\:|\[[^\]]*\]|[^:\\])+/g
+        };
+
+        $blockdev->{'auth-client-required'} = [$options->{'auth_supported'}]
+            if $options->{'auth_supported'};
+        $blockdev->{'conf'} = $options->{'conf'} if $options->{'conf'};
+        $blockdev->{'user'} = $options->{'id'} if $options->{'id'};
+
+        if ($options->{'mon_host'}) {
+            my $server = [];
+            my @mons = split(';', $options->{'mon_host'});
+            for my $mon (@mons) {
+                $mon =~ s/[\[\]]//g;
+                my ($host, $port) = PVE::Tools::parse_host_and_port($mon);
+                $port = '3300' if !$port;
+                push @$server, { host => $host, port => $port };
+            }
+            $blockdev->{server} = $server;
+        }
+
+        if ($options->{'image'} =~ m|^(\S+)/(\S+)$|) {
+            $blockdev->{pool} = $1;
+            $blockdev->{image} = $2;
+            if ($blockdev->{image} =~ m|^(\S+)/(\S+)$|) {
+                $blockdev->{namespace} = $1;
+                $blockdev->{image} = $2;
+            }
+        }
+
+        delete($options->@{qw(auth_supported conf id mon_host image)});
+
+        # Map rest directly. With -drive, it was possible to use arbitrary key-value-pairs. Like
+        # this, there will be warnings for those that are not allowed via blockdev.
+        for my $opt (keys $options->%*) {
+            $blockdev->{$opt} = $options->{$opt};
+        }
     } else {
         die "storage plugin doesn't implement qemu_blockdev_options() method\n";
     }
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v5 16/51] plugin api: bump api version and age

[Fiona Ebner]

Introduce qemu_blockdev_options() plugin method.

In terms of the plugin API only, adding the qemu_blockdev_options()
method is a fully backwards-compatible change. When qemu-server will
switch to '-blockdev' however, plugins where the default implemenation
is not sufficient, will not be usable for virtual machines anymore.
Therefore, this is intended for the next major release, Proxmox VE 9.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

Changes in v5:
* Mention where to find allowed drivers and options and how to request
  if more are needed.

 ApiChangeLog       | 15 +++++++++++++++
 src/PVE/Storage.pm |  4 ++--
 2 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/ApiChangeLog b/ApiChangeLog
index 987da54..9ea7724 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -6,6 +6,21 @@ without breaking anything unaware of it.)
 
 Future changes should be documented in here.
 
+## Version 12:
+
+* Introduce `qemu_blockdev_options()` plugin method
+
+  Proxmox VE will switch to the more modern QEMU command line option `-blockdev` replacing `-drive`.
+  With `-drive`, it was enough to specify a path, where special protocol paths like `iscsi://` were
+  also supported. With `-blockdev`, the data is more structured, a driver needs to be specified
+  alongside the path to an image and each driver supports driver-specific options. Most storage
+  plugins should be fine using driver `host_device` in case of a block device and `file` in case of
+  a file and no special options. See the default implemenation of the base plugin for guidance, also
+  if the plugin uses protocol paths. Implement this this method for Proxmox VE 9. See
+  `$allowed_qemu_blockdev_options` in `PVE/Storage.pm` for currently allowed drivers and option.
+  Feel free to request allowing more drivers or options on the pve-devel mailing list based on your
+  needs.
+
 ##  Version 11:
 
 * Allow declaring storage features via plugin data
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 25ce5f2..6e6bdb2 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -41,11 +41,11 @@ use PVE::Storage::BTRFSPlugin;
 use PVE::Storage::ESXiPlugin;
 
 # Storage API version. Increment it on changes in storage API interface.
-use constant APIVER => 11;
+use constant APIVER => 12;
 # Age is the number of versions we're backward compatible with.
 # This is like having 'current=APIVER' and age='APIAGE' in libtool,
 # see https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
-use constant APIAGE => 2;
+use constant APIAGE => 3;
 
 our $KNOWN_EXPORT_FORMATS = ['raw+size', 'tar+size', 'qcow2+size', 'vmdk+size', 'zfs', 'btrfs'];
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 17/51] mirror: code style: avoid masking earlier declaration of $op

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/BlockJob.pm | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/PVE/QemuServer/BlockJob.pm b/src/PVE/QemuServer/BlockJob.pm
index 0013bde6..d26bcb01 100644
--- a/src/PVE/QemuServer/BlockJob.pm
+++ b/src/PVE/QemuServer/BlockJob.pm
@@ -179,15 +179,15 @@ sub qemu_drive_mirror_monitor {
                         # try to switch the disk if source and destination are on the same guest
                         print "$job_id: Completing block job...\n";
 
-                        my $op;
+                        my $completion_command;
                         if ($completion eq 'complete') {
-                            $op = 'block-job-complete';
+                            $completion_command = 'block-job-complete';
                         } elsif ($completion eq 'cancel') {
-                            $op = 'block-job-cancel';
+                            $completion_command = 'block-job-cancel';
                         } else {
                             die "invalid completion value: $completion\n";
                         }
-                        eval { mon_cmd($vmid, $op, device => $job_id) };
+                        eval { mon_cmd($vmid, $completion_command, device => $job_id) };
                         my $err = $@;
                         if ($err && $err =~ m/cannot be completed/) {
                             print "$job_id: block job cannot be completed, trying again.\n";
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 18/51] test: collect mocked functions for QemuServer module

[Fiona Ebner]

Also order them alphabetically.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/test/MigrationTest/QmMock.pm | 27 ++++++++++++---------------
 1 file changed, 12 insertions(+), 15 deletions(-)

diff --git a/src/test/MigrationTest/QmMock.pm b/src/test/MigrationTest/QmMock.pm
index 3eaa131f..de7f4cd7 100644
--- a/src/test/MigrationTest/QmMock.pm
+++ b/src/test/MigrationTest/QmMock.pm
@@ -50,21 +50,6 @@ $inotify_module->mock(
     },
 );
 
-$MigrationTest::Shared::qemu_server_module->mock(
-    nodename => sub {
-        return $nodename;
-    },
-    config_to_command => sub {
-        return ['mocked_kvm_command'];
-    },
-    vm_start_nolock => sub {
-        my ($storecfg, $vmid, $conf, $params, $migrate_opts) = @_;
-        $forcemachine = $params->{forcemachine}
-            or die "mocked vm_start_nolock - expected 'forcemachine' parameter\n";
-        $MigrationTest::Shared::qemu_server_module->original('vm_start_nolock')->(@_);
-    },
-);
-
 my $qemu_server_helpers_module = Test::MockModule->new("PVE::QemuServer::Helpers");
 $qemu_server_helpers_module->mock(
     vm_running_locally => sub {
@@ -113,6 +98,9 @@ $MigrationTest::Shared::storage_module->mock(
 );
 
 $MigrationTest::Shared::qemu_server_module->mock(
+    config_to_command => sub {
+        return ['mocked_kvm_command'];
+    },
     mon_cmd => sub {
         my ($vmid, $command, %params) = @_;
 
@@ -127,6 +115,9 @@ $MigrationTest::Shared::qemu_server_module->mock(
         }
         die "mon_cmd (mocked) - implement me: $command";
     },
+    nodename => sub {
+        return $nodename;
+    },
     run_command => sub {
         my ($cmd_full, %param) = @_;
 
@@ -149,6 +140,12 @@ $MigrationTest::Shared::qemu_server_module->mock(
         file_set_contents("${RUN_DIR_PATH}/nbd_info", to_json($nbd));
         return $nbd;
     },
+    vm_start_nolock => sub {
+        my ($storecfg, $vmid, $conf, $params, $migrate_opts) = @_;
+        $forcemachine = $params->{forcemachine}
+            or die "mocked vm_start_nolock - expected 'forcemachine' parameter\n";
+        $MigrationTest::Shared::qemu_server_module->original('vm_start_nolock')->(@_);
+    },
 );
 
 our $cmddef = {
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 19/51] drive: add helper to parse drive interface

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Drive.pm | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/src/PVE/QemuServer/Drive.pm b/src/PVE/QemuServer/Drive.pm
index a6f5062f..bc9fbc48 100644
--- a/src/PVE/QemuServer/Drive.pm
+++ b/src/PVE/QemuServer/Drive.pm
@@ -745,6 +745,16 @@ sub drive_is_read_only {
     return $drive->{interface} ne 'sata' && $drive->{interface} ne 'ide';
 }
 
+sub parse_drive_interface {
+    my ($key) = @_;
+
+    if ($key =~ m/^([^\d]+)(\d+)$/) {
+        return ($1, $2);
+    }
+
+    die "unable to parse drive interface $key\n";
+}
+
 # ideX = [volume=]volume-id[,media=d]
 #        [,snapshot=on|off][,cache=on|off][,format=f][,backup=yes|no]
 #        [,rerror=ignore|report|stop][,werror=enospc|ignore|report|stop]
@@ -754,14 +764,8 @@ sub drive_is_read_only {
 sub parse_drive {
     my ($key, $data, $with_alloc) = @_;
 
-    my ($interface, $index);
-
-    if ($key =~ m/^([^\d]+)(\d+)$/) {
-        $interface = $1;
-        $index = $2;
-    } else {
-        return;
-    }
+    my ($interface, $index) = eval { parse_drive_interface($key) };
+    return if $@;
 
     my $desc_hash = $with_alloc ? $drivedesc_hash_with_alloc : $drivedesc_hash;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 20/51] drive: drop invalid export of get_scsi_devicetype

[Fiona Ebner]

The function is called 'get_scsi_device_type' and all callers already
use the full package prefix to call it.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Drive.pm | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/PVE/QemuServer/Drive.pm b/src/PVE/QemuServer/Drive.pm
index bc9fbc48..cd00f37d 100644
--- a/src/PVE/QemuServer/Drive.pm
+++ b/src/PVE/QemuServer/Drive.pm
@@ -21,7 +21,6 @@ our @EXPORT_OK = qw(
     drive_is_cloudinit
     drive_is_cdrom
     drive_is_read_only
-    get_scsi_devicetype
     parse_drive
     print_drive
     storage_allows_io_uring_default
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 21/51] blockdev: add and use throttle_group_id() helper

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 6e6b9245..3caae362 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -39,6 +39,12 @@ my sub read_only_json_option {
     return json_bool($drive->{ro} || drive_is_cdrom($drive) || $options->{'read-only'});
 }
 
+my sub throttle_group_id {
+    my ($drive_id) = @_;
+
+    return "throttle-drive-$drive_id";
+}
+
 sub generate_throttle_group {
     my ($drive) = @_;
 
@@ -69,7 +75,7 @@ sub generate_throttle_group {
     }
 
     return {
-        id => "throttle-drive-$drive_id",
+        id => throttle_group_id($drive_id),
         limits => $limits,
         'qom-type' => 'throttle-group',
     };
@@ -216,7 +222,7 @@ sub generate_drive_blockdev {
     return {
         driver => "throttle",
         'node-name' => "drive-$drive_id",
-        'throttle-group' => "throttle-drive-$drive_id",
+        'throttle-group' => throttle_group_id($drive_id),
         file => $child,
     };
 }
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 22/51] blockdev: introduce top_node_name() and parse_top_node_name() helpers

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 3caae362..0f30f328 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -33,6 +33,23 @@ my sub get_node_name {
     return "${prefix}${hash}";
 }
 
+sub parse_top_node_name {
+    my ($node_name) = @_;
+
+    if ($node_name =~ m/^drive-(.+)$/) {
+        my $drive_id = $1;
+        return $drive_id if PVE::QemuServer::Drive::is_valid_drivename($drive_id);
+    }
+
+    return;
+}
+
+sub top_node_name {
+    my ($drive_id) = @_;
+
+    return "drive-$drive_id";
+}
+
 my sub read_only_json_option {
     my ($drive, $options) = @_;
 
@@ -221,7 +238,7 @@ sub generate_drive_blockdev {
     # this is the top filter entry point, use $drive-drive_id as nodename
     return {
         driver => "throttle",
-        'node-name' => "drive-$drive_id",
+        'node-name' => top_node_name($drive_id),
         'throttle-group' => throttle_group_id($drive_id),
         file => $child,
     };
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 23/51] blockdev: add helpers for attaching and detaching block devices

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

Changes in v3:
* Adapt to latest version of QEMU patch for child node name info.

 src/PVE/QemuServer/Blockdev.pm | 141 +++++++++++++++++++++++++++++++++
 1 file changed, 141 insertions(+)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 0f30f328..50b0b0c6 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -11,6 +11,7 @@ use PVE::JSONSchema qw(json_bool);
 use PVE::Storage;
 
 use PVE::QemuServer::Drive qw(drive_is_cdrom);
+use PVE::QemuServer::Monitor qw(mon_cmd);
 
 my sub get_node_name {
     my ($type, $drive_id, $volid, $snap) = @_;
@@ -244,4 +245,144 @@ sub generate_drive_blockdev {
     };
 }
 
+my sub blockdev_add {
+    my ($vmid, $blockdev) = @_;
+
+    eval { mon_cmd($vmid, 'blockdev-add', $blockdev->%*); };
+    if (my $err = $@) {
+        my $node_name = $blockdev->{'node-name'} // 'undefined';
+        die "adding blockdev '$node_name' failed : $err\n" if $@;
+    }
+
+    return;
+}
+
+=pod
+
+=head3 attach
+
+    my $node_name = attach($storecfg, $vmid, $drive, $options);
+
+Attach the drive C<$drive> to the VM C<$vmid> considering the additional options C<$options>.
+Returns the node name of the (topmost) attached block device node.
+
+Parameters:
+
+=over
+
+=item C<$storecfg>: The storage configuration.
+
+=item C<$vmid>: The ID of the virtual machine.
+
+=item C<$drive>: The drive as parsed from a virtual machine configuration.
+
+=item C<$options>: A hash reference with additional options.
+
+=over
+
+=item C<< $options->{'read-only'} >>: Attach the image as read-only irrespective of the
+configuration in C<$drive>.
+
+=item C<< $options->{size} >>: Attach the image with this virtual size. Must be smaller than the
+actual size of the image. The image format must be C<raw>.
+
+=item C<< $options->{'snapshot-name'} >>: Attach this snapshot of the volume C<< $drive->{file} >>,
+rather than the volume itself.
+
+=back
+
+=back
+
+=cut
+
+sub attach {
+    my ($storecfg, $vmid, $drive, $options) = @_;
+
+    my $blockdev = generate_drive_blockdev($storecfg, $drive, $options);
+
+    my $throttle_group_id;
+    if (parse_top_node_name($blockdev->{'node-name'})) { # device top nodes need a throttle group
+        my $drive_id = PVE::QemuServer::Drive::get_drive_id($drive);
+        $throttle_group_id = throttle_group_id($drive_id);
+    }
+
+    eval {
+        if ($throttle_group_id) {
+            # Try to remove potential left-over.
+            eval { mon_cmd($vmid, 'object-del', id => $throttle_group_id); };
+
+            my $throttle_group = generate_throttle_group($drive);
+            mon_cmd($vmid, 'object-add', $throttle_group->%*);
+        }
+
+        blockdev_add($vmid, $blockdev);
+    };
+    if (my $err = $@) {
+        if ($throttle_group_id) {
+            eval { mon_cmd($vmid, 'object-del', id => $throttle_group_id); };
+        }
+        die $err;
+    }
+
+    return $blockdev->{'node-name'};
+}
+
+=pod
+
+=head3 detach
+
+    detach($vmid, $node_name);
+
+Detach the block device C<$node_name> from the VM C<$vmid>. Also removes associated child block
+nodes.
+
+Parameters:
+
+=over
+
+=item C<$vmid>: The ID of the virtual machine.
+
+=item C<$node_name>: The node name identifying the block node in QEMU.
+
+=back
+
+=cut
+
+sub detach {
+    my ($vmid, $node_name) = @_;
+
+    die "Blockdev::detach - no node name\n" if !$node_name;
+
+    my $block_info = mon_cmd($vmid, "query-named-block-nodes");
+    $block_info = { map { $_->{'node-name'} => $_ } $block_info->@* };
+
+    my $remove_throttle_group_id;
+    if ((my $drive_id = parse_top_node_name($node_name)) && $block_info->{$node_name}) {
+        $remove_throttle_group_id = throttle_group_id($drive_id);
+    }
+
+    while ($node_name) {
+        last if !$block_info->{$node_name}; # already gone
+
+        eval { mon_cmd($vmid, 'blockdev-del', 'node-name' => "$node_name"); };
+        if (my $err = $@) {
+            last if $err =~ m/Failed to find node with node-name/; # already gone
+            die "deleting blockdev '$node_name' failed : $err\n";
+        }
+
+        my $children = { map { $_->{child} => $_ } $block_info->{$node_name}->{children}->@* };
+        # Recursively remove 'file' child nodes. QEMU will auto-remove implicitly added child nodes,
+        # but e.g. the child of the top throttle node might have been explicitly added as a mirror
+        # target, and needs to be removed manually.
+        $node_name = $children->{file}->{'node-name'};
+    }
+
+    if ($remove_throttle_group_id) {
+        eval { mon_cmd($vmid, 'object-del', id => $remove_throttle_group_id); };
+        die "removing throttle group failed - $@\n" if $@;
+    }
+
+    return;
+}
+
 1;
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 24/51] blockdev: add missing include for JSON module

[Fiona Ebner]

Fixes: f2f2edcd ("blockdev: add workaround for issue #3229")
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 50b0b0c6..9d94dc9d 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -6,6 +6,7 @@ use warnings;
 use Digest::SHA;
 use Fcntl qw(S_ISBLK S_ISCHR);
 use File::stat;
+use JSON;
 
 use PVE::JSONSchema qw(json_bool);
 use PVE::Storage;
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 25/51] backup: use blockdev for fleecing images

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuConfig.pm          | 12 ++-------
 src/PVE/QemuServer/Blockdev.pm | 47 +++++++++++++++++++++++++++++++---
 src/PVE/VZDump/QemuServer.pm   | 26 +++++++++++--------
 3 files changed, 61 insertions(+), 24 deletions(-)

diff --git a/src/PVE/QemuConfig.pm b/src/PVE/QemuConfig.pm
index 01104723..82295641 100644
--- a/src/PVE/QemuConfig.pm
+++ b/src/PVE/QemuConfig.pm
@@ -10,6 +10,7 @@ use PVE::INotify;
 use PVE::JSONSchema;
 use PVE::QemuMigrate::Helpers;
 use PVE::QemuServer::Agent;
+use PVE::QemuServer::Blockdev;
 use PVE::QemuServer::CPUConfig;
 use PVE::QemuServer::Drive;
 use PVE::QemuServer::Helpers;
@@ -675,16 +676,7 @@ sub cleanup_fleecing_images {
         };
         $log_func->('warn', "checking/canceling old backup job failed - $@") if $@;
 
-        my $block_info = mon_cmd($vmid, "query-block");
-        for my $info ($block_info->@*) {
-            my $device_id = $info->{device};
-            next if $device_id !~ m/-fleecing$/;
-
-            $log_func->('info', "detaching (old) fleecing image for '$device_id'");
-            $device_id =~ s/^drive-//; # re-added by qemu_drivedel()
-            eval { PVE::QemuServer::qemu_drivedel($vmid, $device_id) };
-            $log_func->('warn', "error detaching (old) fleecing image '$device_id' - $@") if $@;
-        }
+        PVE::QemuServer::Blockdev::detach_fleecing_block_nodes($vmid, $log_func);
     }
 
     PVE::QemuConfig->lock_config(
diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 9d94dc9d..4fc71d45 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -14,8 +14,30 @@ use PVE::Storage;
 use PVE::QemuServer::Drive qw(drive_is_cdrom);
 use PVE::QemuServer::Monitor qw(mon_cmd);
 
+my sub fleecing_node_name {
+    my ($type, $drive_id) = @_;
+
+    if ($type eq 'fmt') {
+        return "drive-$drive_id-fleecing"; # this is the top node for fleecing
+    } elsif ($type eq 'file') {
+        return "$drive_id-fleecing-file"; # drop the "drive-" prefix to be sure, max length is 31
+    }
+
+    die "unknown node type '$type' for fleecing";
+}
+
+my sub is_fleecing_top_node {
+    my ($node_name) = @_;
+
+    return $node_name =~ m/-fleecing$/ ? 1 : 0;
+}
+
 my sub get_node_name {
-    my ($type, $drive_id, $volid, $snap) = @_;
+    my ($type, $drive_id, $volid, $options) = @_;
+
+    return fleecing_node_name($type, $drive_id) if $options->{fleecing};
+
+    my $snap = $options->{'snapshot-name'};
 
     my $info = "drive=$drive_id,";
     $info .= "snap=$snap," if defined($snap);
@@ -174,8 +196,7 @@ sub generate_file_blockdev {
         $blockdev->{'detect-zeroes'} = PVE::QemuServer::Drive::detect_zeroes_cmdline_option($drive);
     }
 
-    $blockdev->{'node-name'} =
-        get_node_name('file', $drive_id, $drive->{file}, $options->{'snapshot-name'});
+    $blockdev->{'node-name'} = get_node_name('file', $drive_id, $drive->{file}, $options);
 
     $blockdev->{'read-only'} = read_only_json_option($drive, $options);
 
@@ -208,7 +229,7 @@ sub generate_format_blockdev {
         $format = $drive->{format} // 'raw';
     }
 
-    my $node_name = get_node_name('fmt', $drive_id, $drive->{file}, $options->{'snapshot-name'});
+    my $node_name = get_node_name('fmt', $drive_id, $drive->{file}, $options);
 
     my $blockdev = {
         'node-name' => "$node_name",
@@ -237,6 +258,8 @@ sub generate_drive_blockdev {
     my $child = generate_file_blockdev($storecfg, $drive, $options);
     $child = generate_format_blockdev($storecfg, $drive, $child, $options);
 
+    return $child if $options->{fleecing}; # for fleecing, this is already the top node
+
     # this is the top filter entry point, use $drive-drive_id as nodename
     return {
         driver => "throttle",
@@ -281,6 +304,8 @@ Parameters:
 
 =over
 
+=item C<< $options->{fleecing} >>: Generate and attach a block device for backup fleecing.
+
 =item C<< $options->{'read-only'} >>: Attach the image as read-only irrespective of the
 configuration in C<$drive>.
 
@@ -386,4 +411,18 @@ sub detach {
     return;
 }
 
+sub detach_fleecing_block_nodes {
+    my ($vmid, $log_func) = @_;
+
+    my $block_info = mon_cmd($vmid, "query-named-block-nodes");
+    for my $info ($block_info->@*) {
+        my $node_name = $info->{'node-name'};
+        next if !is_fleecing_top_node($node_name);
+
+        $log_func->('info', "detaching (old) fleecing image '$node_name'");
+        eval { detach($vmid, $node_name) };
+        $log_func->('warn', "error detaching (old) fleecing image '$node_name' - $@") if $@;
+    }
+}
+
 1;
diff --git a/src/PVE/VZDump/QemuServer.pm b/src/PVE/VZDump/QemuServer.pm
index 243a927e..8b643bc4 100644
--- a/src/PVE/VZDump/QemuServer.pm
+++ b/src/PVE/VZDump/QemuServer.pm
@@ -30,6 +30,7 @@ use PVE::Format qw(render_duration render_bytes);
 use PVE::QemuConfig;
 use PVE::QemuServer;
 use PVE::QemuServer::Agent;
+use PVE::QemuServer::Blockdev;
 use PVE::QemuServer::Drive qw(checked_volume_format);
 use PVE::QemuServer::Helpers;
 use PVE::QemuServer::Machine;
@@ -626,9 +627,8 @@ my sub detach_fleecing_images {
 
     for my $di ($disks->@*) {
         if (my $volid = $di->{'fleece-volid'}) {
-            my $devid = "$di->{qmdevice}-fleecing";
-            $devid =~ s/^drive-//; # re-added by qemu_drivedel()
-            eval { PVE::QemuServer::qemu_drivedel($vmid, $devid) };
+            my $node_name = "$di->{qmdevice}-fleecing";
+            eval { PVE::QemuServer::Blockdev::detach($vmid, $node_name) };
         }
     }
 }
@@ -646,15 +646,21 @@ my sub attach_fleecing_images {
         if (my $volid = $di->{'fleece-volid'}) {
             $self->loginfo("$di->{qmdevice}: attaching fleecing image $volid to QEMU");
 
-            my $path = PVE::Storage::path($self->{storecfg}, $volid);
-            my $devid = "$di->{qmdevice}-fleecing";
-            my $drive = "file=$path,if=none,id=$devid,format=$format,discard=unmap";
+            my ($interface, $index) = PVE::QemuServer::Drive::parse_drive_interface($di->{virtdev});
+            my $drive = {
+                file => $volid,
+                interface => $interface,
+                index => $index,
+                format => $format,
+                discard => 'on',
+            };
+
+            my $options = { 'fleecing' => 1 };
             # Specify size explicitly, to make it work if storage backend rounded up size for
             # fleecing image when allocating.
-            $drive .= ",size=$di->{'block-node-size'}" if $format eq 'raw';
-            $drive =~ s/\\/\\\\/g;
-            my $ret = PVE::QemuServer::Monitor::hmp_cmd($vmid, "drive_add auto \"$drive\"", 60);
-            die "attaching fleecing image $volid failed - $ret\n" if $ret !~ m/OK/s;
+            $options->{size} = $di->{'block-node-size'} if $format eq 'raw';
+
+            PVE::QemuServer::Blockdev::attach($self->{storecfg}, $vmid, $drive, $options);
         }
     }
 }
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 26/51] backup: use blockdev for TPM state file

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 25 ++++++++++++++++++++++++-
 src/PVE/VZDump/QemuServer.pm   | 19 ++++++++++---------
 2 files changed, 34 insertions(+), 10 deletions(-)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 4fc71d45..41780502 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -14,6 +14,18 @@ use PVE::Storage;
 use PVE::QemuServer::Drive qw(drive_is_cdrom);
 use PVE::QemuServer::Monitor qw(mon_cmd);
 
+my sub tpm_backup_node_name {
+    my ($type, $drive_id) = @_;
+
+    if ($type eq 'fmt') {
+        return "drive-$drive_id-backup"; # this is the top node
+    } elsif ($type eq 'file') {
+        return "$drive_id-backup-file"; # drop the "drive-" prefix to be sure, max length is 31
+    }
+
+    die "unknown node type '$type' for TPM backup node";
+}
+
 my sub fleecing_node_name {
     my ($type, $drive_id) = @_;
 
@@ -36,6 +48,7 @@ my sub get_node_name {
     my ($type, $drive_id, $volid, $options) = @_;
 
     return fleecing_node_name($type, $drive_id) if $options->{fleecing};
+    return tpm_backup_node_name($type, $drive_id) if $options->{'tpm-backup'};
 
     my $snap = $options->{'snapshot-name'};
 
@@ -258,7 +271,8 @@ sub generate_drive_blockdev {
     my $child = generate_file_blockdev($storecfg, $drive, $options);
     $child = generate_format_blockdev($storecfg, $drive, $child, $options);
 
-    return $child if $options->{fleecing}; # for fleecing, this is already the top node
+    # for fleecing and TPM backup, this is already the top node
+    return $child if $options->{fleecing} || $options->{'tpm-backup'};
 
     # this is the top filter entry point, use $drive-drive_id as nodename
     return {
@@ -315,6 +329,9 @@ actual size of the image. The image format must be C<raw>.
 =item C<< $options->{'snapshot-name'} >>: Attach this snapshot of the volume C<< $drive->{file} >>,
 rather than the volume itself.
 
+=item C<< $options->{'tpm-backup'} >>: Generate and attach a block device for backing up the TPM
+state image.
+
 =back
 
 =back
@@ -411,6 +428,12 @@ sub detach {
     return;
 }
 
+sub detach_tpm_backup_node {
+    my ($vmid) = @_;
+
+    detach($vmid, "drive-tpmstate0-backup");
+}
+
 sub detach_fleecing_block_nodes {
     my ($vmid, $log_func) = @_;
 
diff --git a/src/PVE/VZDump/QemuServer.pm b/src/PVE/VZDump/QemuServer.pm
index 8b643bc4..f3e292e7 100644
--- a/src/PVE/VZDump/QemuServer.pm
+++ b/src/PVE/VZDump/QemuServer.pm
@@ -158,7 +158,7 @@ sub prepare {
         if ($ds eq 'tpmstate0') {
             # TPM drive only exists for backup, which is reflected in the name
             $diskinfo->{qmdevice} = 'drive-tpmstate0-backup';
-            $task->{tpmpath} = $path;
+            $task->{'tpm-volid'} = $volid;
         }
 
         if (-b $path) {
@@ -474,24 +474,25 @@ my $query_backup_status_loop = sub {
 my $attach_tpmstate_drive = sub {
     my ($self, $task, $vmid) = @_;
 
-    return if !$task->{tpmpath};
+    return if !$task->{'tpm-volid'};
 
     # unconditionally try to remove the tpmstate-named drive - it only exists
     # for backing up, and avoids errors if left over from some previous event
-    eval { PVE::QemuServer::qemu_drivedel($vmid, "tpmstate0-backup"); };
+    eval { PVE::QemuServer::Blockdev::detach_tpm_backup_node($vmid); };
 
     $self->loginfo('attaching TPM drive to QEMU for backup');
 
-    my $drive = "file=$task->{tpmpath},if=none,read-only=on,id=drive-tpmstate0-backup";
-    $drive =~ s/\\/\\\\/g;
-    my $ret = PVE::QemuServer::Monitor::hmp_cmd($vmid, "drive_add auto \"$drive\"", 60);
-    die "attaching TPM drive failed - $ret\n" if $ret !~ m/OK/s;
+    my $drive = { file => $task->{'tpm-volid'}, interface => 'tpmstate', index => 0 };
+    my $extra_options = { 'tpm-backup' => 1, 'read-only' => 1 };
+    PVE::QemuServer::Blockdev::attach($self->{storecfg}, $vmid, $drive, $extra_options);
 };
 
 my $detach_tpmstate_drive = sub {
     my ($task, $vmid) = @_;
-    return if !$task->{tpmpath} || !PVE::QemuServer::check_running($vmid);
-    eval { PVE::QemuServer::qemu_drivedel($vmid, "tpmstate0-backup"); };
+
+    return if !$task->{'tpm-volid'} || !PVE::QemuServer::Helpers::vm_running_locally($vmid);
+
+    eval { PVE::QemuServer::Blockdev::detach_tpm_backup_node($vmid); };
 };
 
 my sub add_backup_performance_options {
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 27/51] blockdev: introduce qdev_id_to_drive_id() helper

[Fiona Ebner]

To be re-used by users of the query-block QMP command. With -blockdev,
the 'device' property is not initialized. See also commit 9af3ef69
("vm devices list: prepare querying block device names for -blockdev")
for context.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm          | 11 +++--------
 src/PVE/QemuServer/Blockdev.pm | 14 ++++++++++++++
 2 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index e7c98520..a15b4557 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -55,6 +55,7 @@ use PVE::QemuConfig;
 use PVE::QemuConfig::NoWrite;
 use PVE::QemuMigrate::Helpers;
 use PVE::QemuServer::Agent qw(qga_check_running);
+use PVE::QemuServer::Blockdev;
 use PVE::QemuServer::BlockJob;
 use PVE::QemuServer::Helpers
     qw(config_aware_timeout get_iscsi_initiator_name min_version kvm_user_version windows_version);
@@ -3813,14 +3814,8 @@ sub vm_devices_list {
     my $resblock = mon_cmd($vmid, 'query-block');
     for my $block ($resblock->@*) {
         my $qdev_id = $block->{qdev} or next;
-        if ($qdev_id =~ m|^/machine/peripheral/(virtio(\d+))/virtio-backend$|) {
-            $qdev_id = $1;
-        } elsif ($qdev_id =~ m|^/machine/system\.flash0$|) {
-            $qdev_id = 'pflash0';
-        } elsif ($qdev_id =~ m|^/machine/system\.flash1$|) {
-            $qdev_id = 'efidisk0';
-        }
-        $devices->{$qdev_id} = 1;
+        my $drive_id = PVE::QemuServer::Blockdev::qdev_id_to_drive_id($qdev_id);
+        $devices->{$drive_id} = 1;
     }
 
     my $resmice = mon_cmd($vmid, 'query-mice');
diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 41780502..d355387c 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -44,6 +44,20 @@ my sub is_fleecing_top_node {
     return $node_name =~ m/-fleecing$/ ? 1 : 0;
 }
 
+sub qdev_id_to_drive_id {
+    my ($qdev_id) = @_;
+
+    if ($qdev_id =~ m|^/machine/peripheral/(virtio(\d+))/virtio-backend$|) {
+        return $1;
+    } elsif ($qdev_id =~ m|^/machine/system\.flash0$|) {
+        return 'pflash0';
+    } elsif ($qdev_id =~ m|^/machine/system\.flash1$|) {
+        return 'efidisk0';
+    }
+
+    return $qdev_id; # for SCSI/SATA/IDE it's the same
+}
+
 my sub get_node_name {
     my ($type, $drive_id, $volid, $options) = @_;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 28/51] blockdev: introduce and use get_block_info() helper

[Fiona Ebner]

When querying the block info, with -blockdev, it is necessary to look
at the 'qdev' property of the QMP result, because the 'device'
property is not initialized. See also commit 9af3ef69 ("vm devices
list: prepare querying block device names for -blockdev").

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm            |  5 ++---
 src/PVE/QemuServer/Blockdev.pm   | 37 ++++++++++++++++++++++++++++++++
 src/PVE/VZDump/QemuServer.pm     |  5 ++---
 src/test/MigrationTest/QmMock.pm | 35 ++++++++++++++++++------------
 4 files changed, 62 insertions(+), 20 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index a15b4557..dedb05f1 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -5732,14 +5732,13 @@ sub vm_start_nolock {
             $migrate_storage_uri = "nbd:${localip}:${storage_migrate_port}";
         }
 
-        my $block_info = mon_cmd($vmid, "query-block");
-        $block_info = { map { $_->{device} => $_ } $block_info->@* };
+        my $block_info = PVE::QemuServer::Blockdev::get_block_info($vmid);
 
         foreach my $opt (sort keys %$nbd) {
             my $drivestr = $nbd->{$opt}->{drivestr};
             my $volid = $nbd->{$opt}->{volid};
 
-            my $block_node = $block_info->{"drive-$opt"}->{inserted}->{'node-name'};
+            my $block_node = $block_info->{$opt}->{inserted}->{'node-name'};
 
             mon_cmd(
                 $vmid,
diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index d355387c..47aa5698 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -58,6 +58,43 @@ sub qdev_id_to_drive_id {
     return $qdev_id; # for SCSI/SATA/IDE it's the same
 }
 
+=pod
+
+=head3 get_block_info
+
+    my $block_info = get_block_info($vmid);
+    my $inserted = $block_info->{$drive_key}->{inserted};
+    my $node_name = $inserted->{'node-name'};
+    my $block_node_size = $inserted->{image}->{'virtual-size'};
+
+Returns a hash reference with the information from the C<query-block> QMP command indexed by
+configuration drive keys like C<scsi2>. See the QMP documentation for details.
+
+Parameters:
+
+=over
+
+=item C<$vmid>: The ID of the virtual machine to query.
+
+=back
+
+=cut
+
+sub get_block_info {
+    my ($vmid) = @_;
+
+    my $block_info = {};
+
+    my $qmp_block_info = mon_cmd($vmid, "query-block");
+    for my $info ($qmp_block_info->@*) {
+        my $qdev_id = $info->{qdev} or next;
+        my $drive_id = qdev_id_to_drive_id($qdev_id);
+        $block_info->{$drive_id} = $info;
+    }
+
+    return $block_info;
+}
+
 my sub get_node_name {
     my ($type, $drive_id, $volid, $options) = @_;
 
diff --git a/src/PVE/VZDump/QemuServer.pm b/src/PVE/VZDump/QemuServer.pm
index f3e292e7..44d3c594 100644
--- a/src/PVE/VZDump/QemuServer.pm
+++ b/src/PVE/VZDump/QemuServer.pm
@@ -1122,14 +1122,13 @@ sub qga_fs_thaw {
 sub query_block_node_sizes {
     my ($self, $vmid, $disks) = @_;
 
-    my $block_info = mon_cmd($vmid, "query-block");
-    $block_info = { map { $_->{device} => $_ } $block_info->@* };
+    my $block_info = PVE::QemuServer::Blockdev::get_block_info($vmid);
 
     for my $diskinfo ($disks->@*) {
         my $drive_key = $diskinfo->{virtdev};
         $drive_key .= "-backup" if $drive_key eq 'tpmstate0';
         my $block_node_size =
-            eval { $block_info->{"drive-$drive_key"}->{inserted}->{image}->{'virtual-size'}; };
+            eval { $block_info->{$drive_key}->{inserted}->{image}->{'virtual-size'}; };
         if (!$block_node_size) {
             $self->loginfo(
                 "could not determine block node size of drive '$drive_key' - using fallback");
diff --git a/src/test/MigrationTest/QmMock.pm b/src/test/MigrationTest/QmMock.pm
index de7f4cd7..78be47d3 100644
--- a/src/test/MigrationTest/QmMock.pm
+++ b/src/test/MigrationTest/QmMock.pm
@@ -43,6 +43,21 @@ sub fork_worker {
 
 # mocked modules
 
+my sub mocked_mon_cmd {
+    my ($vmid, $command, %params) = @_;
+
+    if ($command eq 'nbd-server-start') {
+        return;
+    } elsif ($command eq 'block-export-add') {
+        return;
+    } elsif ($command eq 'query-block') {
+        return [];
+    } elsif ($command eq 'qom-set') {
+        return;
+    }
+    die "mon_cmd (mocked) - implement me: $command";
+}
+
 my $inotify_module = Test::MockModule->new("PVE::INotify");
 $inotify_module->mock(
     nodename => sub {
@@ -50,6 +65,11 @@ $inotify_module->mock(
     },
 );
 
+my $qemu_server_blockdev_module = Test::MockModule->new("PVE::QemuServer::Blockdev");
+$qemu_server_blockdev_module->mock(
+    mon_cmd => \&mocked_mon_cmd,
+);
+
 my $qemu_server_helpers_module = Test::MockModule->new("PVE::QemuServer::Helpers");
 $qemu_server_helpers_module->mock(
     vm_running_locally => sub {
@@ -101,20 +121,7 @@ $MigrationTest::Shared::qemu_server_module->mock(
     config_to_command => sub {
         return ['mocked_kvm_command'];
     },
-    mon_cmd => sub {
-        my ($vmid, $command, %params) = @_;
-
-        if ($command eq 'nbd-server-start') {
-            return;
-        } elsif ($command eq 'block-export-add') {
-            return;
-        } elsif ($command eq 'query-block') {
-            return [];
-        } elsif ($command eq 'qom-set') {
-            return;
-        }
-        die "mon_cmd (mocked) - implement me: $command";
-    },
+    mon_cmd => \&mocked_mon_cmd,
     nodename => sub {
         return $nodename;
     },
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 29/51] blockdev: move helper for resize into module

[Fiona Ebner]

And replace the deprecated check_running() call while at it.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/API2/Qemu.pm           |  3 ++-
 src/PVE/QemuServer.pm          | 21 ---------------------
 src/PVE/QemuServer/Blockdev.pm | 22 ++++++++++++++++++++++
 3 files changed, 24 insertions(+), 22 deletions(-)

diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 6565ce71..1aa3b358 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -28,6 +28,7 @@ use PVE::GuestImport;
 use PVE::QemuConfig;
 use PVE::QemuServer;
 use PVE::QemuServer::Agent;
+use PVE::QemuServer::Blockdev;
 use PVE::QemuServer::BlockJob;
 use PVE::QemuServer::Cloudinit;
 use PVE::QemuServer::CPUConfig;
@@ -5745,7 +5746,7 @@ __PACKAGE__->register_method({
                 "update VM $vmid: resize --disk $disk --size $sizestr",
             );
 
-            PVE::QemuServer::qemu_block_resize(
+            PVE::QemuServer::Blockdev::resize(
                 $vmid, "drive-$disk", $storecfg, $volid, $newsize,
             );
 
diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index dedb05f1..3f135fcb 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -4341,27 +4341,6 @@ sub qemu_block_set_io_throttle {
 
 }
 
-sub qemu_block_resize {
-    my ($vmid, $deviceid, $storecfg, $volid, $size) = @_;
-
-    my $running = check_running($vmid);
-
-    PVE::Storage::volume_resize($storecfg, $volid, $size, $running);
-
-    return if !$running;
-
-    my $padding = (1024 - $size % 1024) % 1024;
-    $size = $size + $padding;
-
-    mon_cmd(
-        $vmid,
-        "block_resize",
-        device => $deviceid,
-        size => int($size),
-        timeout => 60,
-    );
-}
-
 sub qemu_volume_snapshot {
     my ($vmid, $deviceid, $storecfg, $volid, $snap) = @_;
 
diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 47aa5698..1f8763a9 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -12,6 +12,7 @@ use PVE::JSONSchema qw(json_bool);
 use PVE::Storage;
 
 use PVE::QemuServer::Drive qw(drive_is_cdrom);
+use PVE::QemuServer::Helpers;
 use PVE::QemuServer::Monitor qw(mon_cmd);
 
 my sub tpm_backup_node_name {
@@ -499,4 +500,25 @@ sub detach_fleecing_block_nodes {
     }
 }
 
+sub resize {
+    my ($vmid, $deviceid, $storecfg, $volid, $size) = @_;
+
+    my $running = PVE::QemuServer::Helpers::vm_running_locally($vmid);
+
+    PVE::Storage::volume_resize($storecfg, $volid, $size, $running);
+
+    return if !$running;
+
+    my $padding = (1024 - $size % 1024) % 1024;
+    $size = $size + $padding;
+
+    mon_cmd(
+        $vmid,
+        "block_resize",
+        device => $deviceid,
+        size => int($size),
+        timeout => 60,
+    );
+}
+
 1;
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 30/51] blockdev: add helper to get node below throttle node

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

Changes in v3:
* Adapt to latest version of QEMU patch for child node name info.

 src/PVE/QemuServer/Blockdev.pm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 1f8763a9..7aacc7fc 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -139,6 +139,30 @@ sub top_node_name {
     return "drive-$drive_id";
 }
 
+sub get_node_name_below_throttle {
+    my ($vmid, $device_id, $assert_top_is_throttle) = @_;
+
+    my $block_info = get_block_info($vmid);
+    my $drive_id = $device_id =~ s/^drive-//r;
+    my $inserted = $block_info->{$drive_id}->{inserted}
+        or die "no block node inserted for drive '$drive_id'\n";
+
+    if ($inserted->{drv} ne 'throttle') {
+        die "$device_id: unexpected top node $inserted->{'node-name'} ($inserted->{drv})\n"
+            if $assert_top_is_throttle;
+        # before the switch to -blockdev, the top node was not throttle
+        return $inserted->{'node-name'};
+    }
+
+    my $children = { map { $_->{child} => $_ } $inserted->{children}->@* };
+
+    if (my $node_name = $children->{file}->{'node-name'}) {
+        return $node_name;
+    }
+
+    die "$device_id: throttle node without file child node name!\n";
+}
+
 my sub read_only_json_option {
     my ($drive, $options) = @_;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 31/51] blockdev: resize: query and use node name for resize operation

[Fiona Ebner]

This also works for -blockdev, which will be used instead of -drive
starting with machine version 10.0.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 7aacc7fc..08f99dbe 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -533,13 +533,21 @@ sub resize {
 
     return if !$running;
 
+    my $block_info = get_block_info($vmid);
+    my $drive_id = $deviceid =~ s/^drive-//r;
+    my $inserted = $block_info->{$drive_id}->{inserted}
+        or die "no block node inserted for drive '$drive_id'\n";
+
     my $padding = (1024 - $size % 1024) % 1024;
     $size = $size + $padding;
 
     mon_cmd(
         $vmid,
         "block_resize",
-        device => $deviceid,
+        # Need to use the top throttle node, not the node below, because QEMU won't update the size
+        # of the top node otherwise, even though it's a filter node (as of QEMU 10.0). For legacy
+        # -drive, there is no top throttle node, so this also is the correct node.
+        'node-name' => "$inserted->{'node-name'}",
         size => int($size),
         timeout => 60,
     );
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 32/51] blockdev: support using zeroinit filter

[Fiona Ebner]

The zeroinit filter is used for cloning/mirroring and importing with
target volumes that are known to produce zeros when reading parts that
were not written before and can be helpful for performance.

Since it is the target of the mirror, it won't have a 'throttle' node
associated with it, but be added as a top node itself. Therefore, it
requires an explicit node-name.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 08f99dbe..72ea4cdc 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -115,6 +115,8 @@ my sub get_node_name {
         $prefix = 'f';
     } elsif ($type eq 'file') {
         $prefix = 'e';
+    } elsif ($type eq 'zeroinit') {
+        $prefix = 'z';
     } else {
         die "unknown node type '$type'";
     }
@@ -347,6 +349,11 @@ sub generate_drive_blockdev {
     my $child = generate_file_blockdev($storecfg, $drive, $options);
     $child = generate_format_blockdev($storecfg, $drive, $child, $options);
 
+    if ($options->{'zero-initialized'}) {
+        my $node_name = get_node_name('zeroinit', $drive_id, $drive->{file}, $options);
+        $child = { driver => 'zeroinit', file => $child, 'node-name' => "$node_name" };
+    }
+
     # for fleecing and TPM backup, this is already the top node
     return $child if $options->{fleecing} || $options->{'tpm-backup'};
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 33/51] blockdev: make some functions private

[Fiona Ebner]

Callers outside the module should only use generate_drive_blockdev()
and specific functionality should be controlled via the $options
parameter.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 72ea4cdc..8439cf02 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -213,7 +213,7 @@ sub generate_throttle_group {
     };
 }
 
-sub generate_blockdev_drive_cache {
+my sub generate_blockdev_drive_cache {
     my ($drive, $scfg) = @_;
 
     my $cache_direct = PVE::QemuServer::Drive::drive_uses_cache_direct($drive, $scfg);
@@ -223,7 +223,7 @@ sub generate_blockdev_drive_cache {
     };
 }
 
-sub generate_file_blockdev {
+my sub generate_file_blockdev {
     my ($storecfg, $drive, $options) = @_;
 
     my $blockdev = {};
@@ -294,7 +294,7 @@ sub generate_file_blockdev {
     return $blockdev;
 }
 
-sub generate_format_blockdev {
+my sub generate_format_blockdev {
     my ($storecfg, $drive, $child, $options) = @_;
 
     die "generate_format_blockdev called without volid/path\n" if !$drive->{file};
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 34/51] blockdev: add 'no-throttle' option to skip generationg throttle top node

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 8439cf02..fcf5e1f3 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -355,7 +355,7 @@ sub generate_drive_blockdev {
     }
 
     # for fleecing and TPM backup, this is already the top node
-    return $child if $options->{fleecing} || $options->{'tpm-backup'};
+    return $child if $options->{fleecing} || $options->{'tpm-backup'} || $options->{'no-throttle'};
 
     # this is the top filter entry point, use $drive-drive_id as nodename
     return {
@@ -403,6 +403,8 @@ Parameters:
 
 =item C<< $options->{fleecing} >>: Generate and attach a block device for backup fleecing.
 
+=item C<< $options->{'no-throttle'} >>: Do not insert a throttle node as the top node.
+
 =item C<< $options->{'read-only'} >>: Attach the image as read-only irrespective of the
 configuration in C<$drive>.
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 35/51] block job: allow specifying a block node that should be detached upon completion

[Fiona Ebner]

In preparation for blockdev-mirror, where the node that is not used
anymore (either source when switching to target, or otherwise target)
needs to be detached.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/BlockJob.pm | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/src/PVE/QemuServer/BlockJob.pm b/src/PVE/QemuServer/BlockJob.pm
index d26bcb01..68d0431f 100644
--- a/src/PVE/QemuServer/BlockJob.pm
+++ b/src/PVE/QemuServer/BlockJob.pm
@@ -18,13 +18,20 @@ use PVE::QemuServer::RunState;
 # option is useful to get the error for failed jobs here. QEMU's job lock should make it impossible
 # to see a job in 'concluded' state when auto-dismiss=true.
 # $info is the 'BlockJobInfo' for the job returned by query-block-jobs.
+# $job is the information about the job recorded on the PVE-side.
+# A block node $job->{'detach-node-name'} will be detached if present.
 sub qemu_handle_concluded_blockjob {
-    my ($vmid, $job_id, $info) = @_;
+    my ($vmid, $job_id, $qmp_info, $job) = @_;
 
     eval { mon_cmd($vmid, 'job-dismiss', id => $job_id); };
     log_warn("$job_id: failed to dismiss job - $@") if $@;
 
-    die "$job_id: $info->{error} (io-status: $info->{'io-status'})\n" if $info->{error};
+    if (my $node_name = $job->{'detach-node-name'}) {
+        eval { PVE::QemuServer::Blockdev::detach($vmid, $node_name); };
+        log_warn($@) if $@;
+    }
+
+    die "$job_id: $qmp_info->{error} (io-status: $qmp_info->{'io-status'})\n" if $qmp_info->{error};
 }
 
 sub qemu_blockjobs_cancel {
@@ -47,7 +54,7 @@ sub qemu_blockjobs_cancel {
         foreach my $job (keys %$jobs) {
             my $info = $running_jobs->{$job};
             eval {
-                qemu_handle_concluded_blockjob($vmid, $job, $info)
+                qemu_handle_concluded_blockjob($vmid, $job, $info, $jobs->{$job})
                     if $info && $info->{status} eq 'concluded';
             };
             log_warn($@) if $@; # only warn and proceed with canceling other jobs
@@ -106,7 +113,7 @@ sub qemu_drive_mirror_monitor {
 
                 die "$job_id: '$op' has been cancelled\n" if !defined($job);
 
-                qemu_handle_concluded_blockjob($vmid, $job_id, $job)
+                qemu_handle_concluded_blockjob($vmid, $job_id, $job, $jobs->{$job_id})
                     if $job && $job->{status} eq 'concluded';
 
                 my $busy = $job->{busy};
@@ -322,7 +329,7 @@ sub qemu_drive_mirror_switch_to_active_mode {
 
             my $info = $running_jobs->{$job};
             if ($info->{status} eq 'concluded') {
-                qemu_handle_concluded_blockjob($vmid, $job, $info);
+                qemu_handle_concluded_blockjob($vmid, $job, $info, $jobs->{$job});
                 # The 'concluded' state should occur here if and only if the job failed, so the
                 # 'die' below should be unreachable, but play it safe.
                 die "$job: expected job to have failed, but no error was set\n";
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 36/51] block job: add blockdev mirror

[Fiona Ebner]

With blockdev-mirror, it is possible to change the aio setting on the
fly and this is useful for migrations between storages where one wants
to use io_uring by default and the other doesn't.

The node below the top throttle node needs to be replaced so that the
limits stay intact and that the top node still has the drive ID as the
node name. That node is not necessarily a format node. For example, it
could also be a zeroinit node from an earlier mirror operation. So
query QEMU itself.

QEMU automatically drops nodes after mirror only if they were
implicitly added, i.e. not explicitly added via blockdev-add. Since a
previous mirror target is explicitly added (and not just implicitly as
the child of a top throttle node), it is necessary to detach the
appropriate block node after mirror.

Already mock blockdev_mirror in the tests.

Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/BlockJob.pm            | 141 ++++++++++++++++++++++
 src/test/MigrationTest/QemuMigrateMock.pm |   8 ++
 2 files changed, 149 insertions(+)

diff --git a/src/PVE/QemuServer/BlockJob.pm b/src/PVE/QemuServer/BlockJob.pm
index 68d0431f..9131780e 100644
--- a/src/PVE/QemuServer/BlockJob.pm
+++ b/src/PVE/QemuServer/BlockJob.pm
@@ -4,12 +4,14 @@ use strict;
 use warnings;
 
 use JSON;
+use Storable qw(dclone);
 
 use PVE::Format qw(render_duration render_bytes);
 use PVE::RESTEnvironment qw(log_warn);
 use PVE::Storage;
 
 use PVE::QemuServer::Agent qw(qga_check_running);
+use PVE::QemuServer::Blockdev;
 use PVE::QemuServer::Drive qw(checked_volume_format);
 use PVE::QemuServer::Monitor qw(mon_cmd);
 use PVE::QemuServer::RunState;
@@ -187,10 +189,17 @@ sub qemu_drive_mirror_monitor {
                         print "$job_id: Completing block job...\n";
 
                         my $completion_command;
+                        # For blockdev, need to detach appropriate node. QEMU will only drop it if
+                        # it was implicitly added (e.g. as the child of a top throttle node), but
+                        # not if it was explicitly added via blockdev-add (e.g. as a previous mirror
+                        # target).
+                        my $detach_node_name;
                         if ($completion eq 'complete') {
                             $completion_command = 'block-job-complete';
+                            $detach_node_name = $jobs->{$job_id}->{'source-node-name'};
                         } elsif ($completion eq 'cancel') {
                             $completion_command = 'block-job-cancel';
+                            $detach_node_name = $jobs->{$job_id}->{'target-node-name'};
                         } else {
                             die "invalid completion value: $completion\n";
                         }
@@ -202,6 +211,9 @@ sub qemu_drive_mirror_monitor {
                         } elsif ($err) {
                             die "$job_id: block job cannot be completed - $err\n";
                         } else {
+                            $jobs->{$job_id}->{'detach-node-name'} = $detach_node_name
+                                if $detach_node_name;
+
                             print "$job_id: Completed successfully.\n";
                             $jobs->{$job_id}->{complete} = 1;
                         }
@@ -347,6 +359,135 @@ sub qemu_drive_mirror_switch_to_active_mode {
     }
 }
 
+=pod
+
+=head3 blockdev_mirror
+
+    blockdev_mirror($source, $dest, $jobs, $completion, $options)
+
+Mirrors the volume of a running VM specified by C<$source> to destination C<$dest>.
+
+=over
+
+=item C<$source>: The source information consists of:
+
+=over
+
+=item C<< $source->{vmid} >>: The ID of the running VM the source volume belongs to.
+
+=item C<< $source->{drive} >>: The drive configuration of the source volume as currently attached to
+the VM.
+
+=item C<< $source->{bitmap} >>: (optional) Use incremental mirroring based on the specified bitmap.
+
+=back
+
+=item C<$dest>: The destination information consists of:
+
+=over
+
+=item C<< $dest->{volid} >>: The volume ID of the target volume.
+
+=item C<< $dest->{vmid} >>: (optional) The ID of the VM the target volume belongs to. Defaults to
+C<< $source->{vmid} >>.
+
+=item C<< $dest->{'zero-initialized'} >>: (optional) True, if the target volume is zero-initialized.
+
+=back
+
+=item C<$jobs>: (optional) Other jobs in the transaction when multiple volumes should be mirrored.
+All jobs must be ready before completion can happen.
+
+=item C<$completion>: Completion mode, default is C<complete>:
+
+=over
+
+=item C<complete>: Wait until all jobs are ready, block-job-complete them (default). This means
+switching the orignal drive to use the new target.
+
+=item C<cancel>: Wait until all jobs are ready, block-job-cancel them. This means not switching thex
+original drive to use the new target.
+
+=item C<skip>: Wait until all jobs are ready, return with block jobs in ready state.
+
+=item C<auto>: Wait until all jobs disappear, only use for jobs which complete automatically.
+
+=back
+
+=item C<$options>: Further options:
+
+=over
+
+=item C<< $options->{'guest-agent'} >>: If the guest agent is configured for the VM. It will be used
+to freeze and thaw the filesystems for consistency when the target belongs to a different VM.
+
+=item C<< $options->{'bwlimit'} >>: The bandwidth limit to use for the mirroring operation, in
+KiB/s.
+
+=back
+
+=back
+
+=cut
+
+sub blockdev_mirror {
+    my ($source, $dest, $jobs, $completion, $options) = @_;
+
+    my $vmid = $source->{vmid};
+
+    my $drive_id = PVE::QemuServer::Drive::get_drive_id($source->{drive});
+    my $device_id = "drive-$drive_id";
+
+    my $storecfg = PVE::Storage::config();
+
+    # Need to replace the node below the top node. This is not necessarily a format node, for
+    # example, it can also be a zeroinit node by a previous mirror! So query QEMU itself.
+    my $source_node_name =
+        PVE::QemuServer::Blockdev::get_node_name_below_throttle($vmid, $device_id, 1);
+
+    # Copy original drive config (aio, cache, discard, ...):
+    my $dest_drive = dclone($source->{drive});
+    delete($dest_drive->{format}); # cannot use the source's format
+    $dest_drive->{file} = $dest->{volid};
+
+    # Mirror happens below the throttle filter, so if the target is for the same VM, it will end up
+    # below the source's throttle filter, which is inserted for the drive device.
+    my $attach_dest_opts = { 'no-throttle' => 1 };
+    $attach_dest_opts->{'zero-initialized'} = 1 if $dest->{'zero-initialized'};
+
+    # Note that if 'aio' is not explicitly set, i.e. default, it can change if source and target
+    # don't both allow or both not allow 'io_uring' as the default.
+    my $target_node_name =
+        PVE::QemuServer::Blockdev::attach($storecfg, $vmid, $dest_drive, $attach_dest_opts);
+
+    $jobs = {} if !$jobs;
+    my $jobid = "mirror-$drive_id";
+    $jobs->{$jobid} = {
+        'source-node-name' => $source_node_name,
+        'target-node-name' => $target_node_name,
+    };
+
+    my $qmp_opts = common_mirror_qmp_options(
+        $device_id, $target_node_name, $source->{bitmap}, $options->{bwlimit},
+    );
+
+    $qmp_opts->{'job-id'} = "$jobid";
+    $qmp_opts->{replaces} = "$source_node_name";
+
+    # if a job already runs for this device we get an error, catch it for cleanup
+    eval { mon_cmd($vmid, "blockdev-mirror", $qmp_opts->%*); };
+    if (my $err = $@) {
+        eval { qemu_blockjobs_cancel($vmid, $jobs) };
+        log_warn("unable to cancel block jobs - $@");
+        eval { PVE::QemuServer::Blockdev::detach($vmid, $target_node_name); };
+        log_warn("unable to delete blockdev '$target_node_name' - $@");
+        die "error starting blockdev mirrror - $err";
+    }
+    qemu_drive_mirror_monitor(
+        $vmid, $dest->{vmid}, $jobs, $completion, $options->{'guest-agent'}, 'mirror',
+    );
+}
+
 sub mirror {
     my ($source, $dest, $jobs, $completion, $options) = @_;
 
diff --git a/src/test/MigrationTest/QemuMigrateMock.pm b/src/test/MigrationTest/QemuMigrateMock.pm
index 25a4f9b2..c52df84b 100644
--- a/src/test/MigrationTest/QemuMigrateMock.pm
+++ b/src/test/MigrationTest/QemuMigrateMock.pm
@@ -9,6 +9,7 @@ use Test::MockModule;
 use MigrationTest::Shared;
 
 use PVE::API2::Qemu;
+use PVE::QemuServer::Drive;
 use PVE::Storage;
 use PVE::Tools qw(file_set_contents file_get_contents);
 
@@ -167,6 +168,13 @@ $qemu_server_blockjob_module->mock(
 
         common_mirror_mock($vmid, $drive_id);
     },
+    blockdev_mirror => sub {
+        my ($source, $dest, $jobs, $completion, $options) = @_;
+
+        my $drive_id = PVE::QemuServer::Drive::get_drive_id($source->{drive});
+
+        common_mirror_mock($source->{vmid}, $drive_id);
+    },
     qemu_drive_mirror_monitor => sub {
         my ($vmid, $vmiddst, $jobs, $completion, $qga) = @_;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 37/51] blockdev: add change_medium() helper

[Fiona Ebner]

There is a slight change in behavior for cloud-init disks, when the
file for the new cloud-init disk is 'none'. Previously, the current
drive would not be ejected, now it is. Not sure if that edge case can
even happen in practice and it is more correct, becuase the config was
already updated.

Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm          | 40 ++++++----------------------------
 src/PVE/QemuServer/Blockdev.pm | 18 +++++++++++++++
 2 files changed, 25 insertions(+), 33 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 3f135fcb..6e44132e 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -5170,30 +5170,15 @@ sub vmconfig_update_disk {
             }
 
         } else { # cdrom
+            eval { PVE::QemuServer::Blockdev::change_medium($storecfg, $vmid, $opt, $drive); };
+            my $err = $@;
 
-            if ($drive->{file} eq 'none') {
-                mon_cmd($vmid, "eject", force => JSON::true, id => "$opt");
-                if (drive_is_cloudinit($old_drive)) {
-                    vmconfig_register_unused_drive($storecfg, $vmid, $conf, $old_drive);
-                }
-            } else {
-                my ($path, $format) =
-                    PVE::QemuServer::Drive::get_path_and_format($storecfg, $drive);
-
-                # force eject if locked
-                mon_cmd($vmid, "eject", force => JSON::true, id => "$opt");
-
-                if ($path) {
-                    mon_cmd(
-                        $vmid,
-                        "blockdev-change-medium",
-                        id => "$opt",
-                        filename => "$path",
-                        format => "$format",
-                    );
-                }
+            if ($drive->{file} eq 'none' && drive_is_cloudinit($old_drive)) {
+                vmconfig_register_unused_drive($storecfg, $vmid, $conf, $old_drive);
             }
 
+            die $err if $err;
+
             return 1;
         }
     }
@@ -5230,18 +5215,7 @@ sub vmconfig_update_cloudinit_drive {
     my $running = PVE::QemuServer::check_running($vmid);
 
     if ($running) {
-        my ($path, $format) =
-            PVE::QemuServer::Drive::get_path_and_format($storecfg, $cloudinit_drive);
-        if ($path) {
-            mon_cmd($vmid, "eject", force => JSON::true, id => "$cloudinit_ds");
-            mon_cmd(
-                $vmid,
-                "blockdev-change-medium",
-                id => "$cloudinit_ds",
-                filename => "$path",
-                format => "$format",
-            );
-        }
+        PVE::QemuServer::Blockdev::change_medium($storecfg, $vmid, $cloudinit_ds, $cloudinit_drive);
     }
 }
 
diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index fcf5e1f3..1aa17b00 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -562,4 +562,22 @@ sub resize {
     );
 }
 
+sub change_medium {
+    my ($storecfg, $vmid, $qdev_id, $drive) = @_;
+
+    # force eject if locked
+    mon_cmd($vmid, "eject", force => JSON::true, id => "$qdev_id");
+
+    my ($path, $format) = PVE::QemuServer::Drive::get_path_and_format($storecfg, $drive);
+
+    if ($path) { # no path for 'none'
+        mon_cmd(
+            $vmid, "blockdev-change-medium",
+            id => "$qdev_id",
+            filename => "$path",
+            format => "$format",
+        );
+    }
+}
+
 1;
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 38/51] blockdev: add blockdev_change_medium() helper

[Fiona Ebner]

The new helper will be used after the switch to blockdev starting with
machine version 10.0.

Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 1aa17b00..3745c023 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -562,6 +562,21 @@ sub resize {
     );
 }
 
+my sub blockdev_change_medium {
+    my ($storecfg, $vmid, $qdev_id, $drive) = @_;
+
+    # force eject if locked
+    mon_cmd($vmid, "blockdev-open-tray", force => JSON::true, id => "$qdev_id");
+    mon_cmd($vmid, "blockdev-remove-medium", id => "$qdev_id");
+    detach($vmid, "drive-$qdev_id");
+
+    return if $drive->{file} eq 'none';
+
+    attach($storecfg, $vmid, $drive, {});
+    mon_cmd($vmid, "blockdev-insert-medium", id => "$qdev_id", 'node-name' => "drive-$qdev_id");
+    mon_cmd($vmid, "blockdev-close-tray", id => "$qdev_id");
+}
+
 sub change_medium {
     my ($storecfg, $vmid, $qdev_id, $drive) = @_;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 39/51] blockdev: move helper for configuring throttle limits to module

[Fiona Ebner]

Replace the deprecated check_running() call while at it.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm          | 53 +---------------------------------
 src/PVE/QemuServer/Blockdev.pm | 50 ++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 52 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 6e44132e..4cc5ea34 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -4290,57 +4290,6 @@ sub qemu_cpu_hotplug {
     }
 }
 
-sub qemu_block_set_io_throttle {
-    my (
-        $vmid,
-        $deviceid,
-        $bps,
-        $bps_rd,
-        $bps_wr,
-        $iops,
-        $iops_rd,
-        $iops_wr,
-        $bps_max,
-        $bps_rd_max,
-        $bps_wr_max,
-        $iops_max,
-        $iops_rd_max,
-        $iops_wr_max,
-        $bps_max_length,
-        $bps_rd_max_length,
-        $bps_wr_max_length,
-        $iops_max_length,
-        $iops_rd_max_length,
-        $iops_wr_max_length,
-    ) = @_;
-
-    return if !check_running($vmid);
-
-    mon_cmd(
-        $vmid, "block_set_io_throttle",
-        device => $deviceid,
-        bps => int($bps),
-        bps_rd => int($bps_rd),
-        bps_wr => int($bps_wr),
-        iops => int($iops),
-        iops_rd => int($iops_rd),
-        iops_wr => int($iops_wr),
-        bps_max => int($bps_max),
-        bps_rd_max => int($bps_rd_max),
-        bps_wr_max => int($bps_wr_max),
-        iops_max => int($iops_max),
-        iops_rd_max => int($iops_rd_max),
-        iops_wr_max => int($iops_wr_max),
-        bps_max_length => int($bps_max_length),
-        bps_rd_max_length => int($bps_rd_max_length),
-        bps_wr_max_length => int($bps_wr_max_length),
-        iops_max_length => int($iops_max_length),
-        iops_rd_max_length => int($iops_rd_max_length),
-        iops_wr_max_length => int($iops_wr_max_length),
-    );
-
-}
-
 sub qemu_volume_snapshot {
     my ($vmid, $deviceid, $storecfg, $volid, $snap) = @_;
 
@@ -5141,7 +5090,7 @@ sub vmconfig_update_disk {
                         $old_drive->{iops_wr_max_length})
                 ) {
 
-                    qemu_block_set_io_throttle(
+                    PVE::QemuServer::Blockdev::set_io_throttle(
                         $vmid,
                         "drive-$opt",
                         ($drive->{mbps} || 0) * 1024 * 1024,
diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 3745c023..c27248dd 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -595,4 +595,54 @@ sub change_medium {
     }
 }
 
+sub set_io_throttle {
+    my (
+        $vmid,
+        $deviceid,
+        $bps,
+        $bps_rd,
+        $bps_wr,
+        $iops,
+        $iops_rd,
+        $iops_wr,
+        $bps_max,
+        $bps_rd_max,
+        $bps_wr_max,
+        $iops_max,
+        $iops_rd_max,
+        $iops_wr_max,
+        $bps_max_length,
+        $bps_rd_max_length,
+        $bps_wr_max_length,
+        $iops_max_length,
+        $iops_rd_max_length,
+        $iops_wr_max_length,
+    ) = @_;
+
+    return if !PVE::QemuServer::Helpers::vm_running_locally($vmid);
+
+    mon_cmd(
+        $vmid, "block_set_io_throttle",
+        device => $deviceid,
+        bps => int($bps),
+        bps_rd => int($bps_rd),
+        bps_wr => int($bps_wr),
+        iops => int($iops),
+        iops_rd => int($iops_rd),
+        iops_wr => int($iops_wr),
+        bps_max => int($bps_max),
+        bps_rd_max => int($bps_rd_max),
+        bps_wr_max => int($bps_wr_max),
+        iops_max => int($iops_max),
+        iops_rd_max => int($iops_rd_max),
+        iops_wr_max => int($iops_wr_max),
+        bps_max_length => int($bps_max_length),
+        bps_rd_max_length => int($bps_rd_max_length),
+        bps_wr_max_length => int($bps_wr_max_length),
+        iops_max_length => int($iops_max_length),
+        iops_rd_max_length => int($iops_rd_max_length),
+        iops_wr_max_length => int($iops_wr_max_length),
+    );
+}
+
 1;
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 40/51] clone disk: skip check for aio=default (io_uring) compatibility starting with machine version 10.0

[Fiona Ebner]

With blockdev-mirror, it is possible to change the aio setting on the
fly and this is useful for migrations between storages where one wants
to use io_uring by default and the other doesn't.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 4cc5ea34..4d085ac2 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -7546,7 +7546,7 @@ sub template_create : prototype($$;$) {
 # Check for bug #4525: drive-mirror will open the target drive with the same aio setting as the
 # source, but some storages have problems with io_uring, sometimes even leading to crashes.
 my sub clone_disk_check_io_uring {
-    my ($src_drive, $storecfg, $src_storeid, $dst_storeid, $use_drive_mirror) = @_;
+    my ($vmid, $src_drive, $storecfg, $src_storeid, $dst_storeid, $use_drive_mirror) = @_;
 
     return if !$use_drive_mirror;
 
@@ -7563,6 +7563,11 @@ my sub clone_disk_check_io_uring {
     if ($src_drive->{aio}) {
         $src_uses_io_uring = $src_drive->{aio} eq 'io_uring';
     } else {
+        # With the switch to -blockdev and blockdev-mirror, the aio setting will be changed on the
+        # fly if not explicitly set.
+        my $machine_type = PVE::QemuServer::Machine::get_current_qemu_machine($vmid);
+        return if PVE::QemuServer::Machine::is_machine_version_at_least($machine_type, 10, 0);
+
         $src_uses_io_uring = storage_allows_io_uring_default($src_scfg, $cache_direct);
     }
 
@@ -7627,7 +7632,9 @@ sub clone_disk {
             $dst_format = 'raw';
             $size = PVE::QemuServer::Drive::TPMSTATE_DISK_SIZE;
         } else {
-            clone_disk_check_io_uring($drive, $storecfg, $src_storeid, $storeid, $use_drive_mirror);
+            clone_disk_check_io_uring(
+                $vmid, $drive, $storecfg, $src_storeid, $storeid, $use_drive_mirror,
+            );
 
             $size = PVE::Storage::volume_size_info($storecfg, $drive->{file}, 10);
         }
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 41/51] print drive device: don't reference any drive for 'none' starting with machine version 10.0

[Fiona Ebner]

There will be no block node for 'none' after switching to '-blockdev'.

Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
[FE: split out from larger patch
     do it also for non-SCSI cases]
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm                          | 18 +++++++++++++++---
 src/test/cfg2cmd/bootorder-empty.conf.cmd      |  2 +-
 src/test/cfg2cmd/bootorder-legacy.conf.cmd     |  2 +-
 src/test/cfg2cmd/bootorder.conf.cmd            |  2 +-
 ...cputype-icelake-client-deprecation.conf.cmd |  2 +-
 src/test/cfg2cmd/seabios_serial.conf.cmd       |  2 +-
 src/test/cfg2cmd/simple-btrfs.conf.cmd         |  2 +-
 src/test/cfg2cmd/simple-cifs.conf.cmd          |  2 +-
 src/test/cfg2cmd/simple-rbd.conf.cmd           |  2 +-
 src/test/cfg2cmd/simple-virtio-blk.conf.cmd    |  2 +-
 .../cfg2cmd/simple-zfs-over-iscsi.conf.cmd     |  2 +-
 src/test/cfg2cmd/simple1-template.conf.cmd     |  2 +-
 src/test/cfg2cmd/simple1.conf.cmd              |  2 +-
 13 files changed, 27 insertions(+), 15 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 4d085ac2..4529e270 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -1208,7 +1208,12 @@ sub print_drivedevice_full {
     my $drive_id = PVE::QemuServer::Drive::get_drive_id($drive);
     if ($drive->{interface} eq 'virtio') {
         my $pciaddr = print_pci_addr("$drive_id", $bridges, $arch);
-        $device = "virtio-blk-pci,drive=drive-$drive_id,id=${drive_id}${pciaddr}";
+        $device = 'virtio-blk-pci';
+        # for the switch to -blockdev, there is no blockdev for 'none'
+        if (!min_version($machine_version, 10, 0) || $drive->{file} ne 'none') {
+            $device .= ",drive=drive-$drive_id";
+        }
+        $device .= ",id=${drive_id}${pciaddr}";
         $device .= ",iothread=iothread-$drive_id" if $drive->{iothread};
     } elsif ($drive->{interface} eq 'scsi') {
 
@@ -1224,7 +1229,11 @@ sub print_drivedevice_full {
             $device = "scsi-$device_type,bus=$controller_prefix$controller.0,channel=0,scsi-id=0"
                 . ",lun=$drive->{index}";
         }
-        $device .= ",drive=drive-$drive_id,id=$drive_id";
+        # for the switch to -blockdev, there is no blockdev for 'none'
+        if (!min_version($machine_version, 10, 0) || $drive->{file} ne 'none') {
+            $device .= ",drive=drive-$drive_id";
+        }
+        $device .= ",id=$drive_id";
 
         if ($drive->{ssd} && ($device_type eq 'block' || $device_type eq 'hd')) {
             $device .= ",rotation_rate=1";
@@ -1264,7 +1273,10 @@ sub print_drivedevice_full {
         } else {
             $device .= ",bus=ahci$controller.$unit";
         }
-        $device .= ",drive=drive-$drive_id,id=$drive_id";
+        if (!min_version($machine_version, 10, 0) || $drive->{file} ne 'none') {
+            $device .= ",drive=drive-$drive_id";
+        }
+        $device .= ",id=$drive_id";
 
         if ($device_type eq 'hd') {
             if (my $model = $drive->{model}) {
diff --git a/src/test/cfg2cmd/bootorder-empty.conf.cmd b/src/test/cfg2cmd/bootorder-empty.conf.cmd
index e4bf4e6d..3f8fdb8e 100644
--- a/src/test/cfg2cmd/bootorder-empty.conf.cmd
+++ b/src/test/cfg2cmd/bootorder-empty.conf.cmd
@@ -28,7 +28,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2' \
   -device 'lsi,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi4,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=4,drive=drive-scsi4,id=scsi4,write-cache=on' \
diff --git a/src/test/cfg2cmd/bootorder-legacy.conf.cmd b/src/test/cfg2cmd/bootorder-legacy.conf.cmd
index 7627780c..cd990cd8 100644
--- a/src/test/cfg2cmd/bootorder-legacy.conf.cmd
+++ b/src/test/cfg2cmd/bootorder-legacy.conf.cmd
@@ -28,7 +28,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'lsi,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi4,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=4,drive=drive-scsi4,id=scsi4,write-cache=on' \
diff --git a/src/test/cfg2cmd/bootorder.conf.cmd b/src/test/cfg2cmd/bootorder.conf.cmd
index 74af37e1..3cef2161 100644
--- a/src/test/cfg2cmd/bootorder.conf.cmd
+++ b/src/test/cfg2cmd/bootorder.conf.cmd
@@ -28,7 +28,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=103' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=103' \
   -device 'lsi,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi4,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=4,drive=drive-scsi4,id=scsi4,bootindex=102,write-cache=on' \
diff --git a/src/test/cfg2cmd/cputype-icelake-client-deprecation.conf.cmd b/src/test/cfg2cmd/cputype-icelake-client-deprecation.conf.cmd
index effba2b7..e6e09278 100644
--- a/src/test/cfg2cmd/cputype-icelake-client-deprecation.conf.cmd
+++ b/src/test/cfg2cmd/cputype-icelake-client-deprecation.conf.cmd
@@ -26,7 +26,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=/var/lib/vz/images/8006/base-8006-disk-0.qcow2,if=none,id=drive-scsi0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
diff --git a/src/test/cfg2cmd/seabios_serial.conf.cmd b/src/test/cfg2cmd/seabios_serial.conf.cmd
index f901a459..0eb02459 100644
--- a/src/test/cfg2cmd/seabios_serial.conf.cmd
+++ b/src/test/cfg2cmd/seabios_serial.conf.cmd
@@ -26,7 +26,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
diff --git a/src/test/cfg2cmd/simple-btrfs.conf.cmd b/src/test/cfg2cmd/simple-btrfs.conf.cmd
index 9c3f97d2..2aa2083d 100644
--- a/src/test/cfg2cmd/simple-btrfs.conf.cmd
+++ b/src/test/cfg2cmd/simple-btrfs.conf.cmd
@@ -26,7 +26,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=/butter/bread/images/8006/vm-8006-disk-0/disk.raw,if=none,id=drive-scsi0,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
diff --git a/src/test/cfg2cmd/simple-cifs.conf.cmd b/src/test/cfg2cmd/simple-cifs.conf.cmd
index 61e8d01e..d23a046a 100644
--- a/src/test/cfg2cmd/simple-cifs.conf.cmd
+++ b/src/test/cfg2cmd/simple-cifs.conf.cmd
@@ -24,7 +24,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=/mnt/pve/cifs-store/images/8006/vm-8006-disk-0.raw,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,write-cache=on' \
diff --git a/src/test/cfg2cmd/simple-rbd.conf.cmd b/src/test/cfg2cmd/simple-rbd.conf.cmd
index ea5934c4..df7cba3f 100644
--- a/src/test/cfg2cmd/simple-rbd.conf.cmd
+++ b/src/test/cfg2cmd/simple-rbd.conf.cmd
@@ -26,7 +26,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=rbd:cpool/vm-8006-disk-0:mon_host=127.0.0.42;127.0.0.21;[\:\:1]:auth_supported=none,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
diff --git a/src/test/cfg2cmd/simple-virtio-blk.conf.cmd b/src/test/cfg2cmd/simple-virtio-blk.conf.cmd
index 2182febc..0a7eb473 100644
--- a/src/test/cfg2cmd/simple-virtio-blk.conf.cmd
+++ b/src/test/cfg2cmd/simple-virtio-blk.conf.cmd
@@ -27,7 +27,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-virtio0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
   -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,bootindex=100,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
diff --git a/src/test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd b/src/test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd
index d9a8e5e9..a90156b0 100644
--- a/src/test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd
+++ b/src/test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd
@@ -26,7 +26,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=iscsi://127.0.0.1/iqn.2019-10.org.test:foobar/0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
diff --git a/src/test/cfg2cmd/simple1-template.conf.cmd b/src/test/cfg2cmd/simple1-template.conf.cmd
index 60531b77..c736c84a 100644
--- a/src/test/cfg2cmd/simple1-template.conf.cmd
+++ b/src/test/cfg2cmd/simple1-template.conf.cmd
@@ -24,7 +24,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=/var/lib/vz/images/8006/base-8006-disk-1.qcow2,if=none,id=drive-scsi0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap,readonly=on' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,write-cache=on' \
diff --git a/src/test/cfg2cmd/simple1.conf.cmd b/src/test/cfg2cmd/simple1.conf.cmd
index aa76ca62..e657aed7 100644
--- a/src/test/cfg2cmd/simple1.conf.cmd
+++ b/src/test/cfg2cmd/simple1.conf.cmd
@@ -26,7 +26,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
+  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
   -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 42/51] blockdev: add support for NBD paths

[Fiona Ebner]

Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index c27248dd..f6495526 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -15,6 +15,18 @@ use PVE::QemuServer::Drive qw(drive_is_cdrom);
 use PVE::QemuServer::Helpers;
 use PVE::QemuServer::Monitor qw(mon_cmd);
 
+# gives ($host, $port, $export)
+my $NBD_TCP_PATH_RE_3 = qr/nbd:(\S+):(\d+):exportname=(\S+)/;
+my $NBD_UNIX_PATH_RE_2 = qr/nbd:unix:(\S+):exportname=(\S+)/;
+
+my sub is_nbd {
+    my ($drive) = @_;
+
+    return 1 if $drive->{file} =~ $NBD_TCP_PATH_RE_3;
+    return 1 if $drive->{file} =~ $NBD_UNIX_PATH_RE_2;
+    return 0;
+}
+
 my sub tpm_backup_node_name {
     my ($type, $drive_id) = @_;
 
@@ -236,7 +248,13 @@ my sub generate_file_blockdev {
 
     my $drive_id = PVE::QemuServer::Drive::get_drive_id($drive);
 
-    if ($drive->{file} eq 'cdrom') {
+    if ($drive->{file} =~ m/^$NBD_UNIX_PATH_RE_2$/) {
+        my $server = { type => 'unix', path => "$1" };
+        $blockdev = { driver => 'nbd', server => $server, export => "$2" };
+    } elsif ($drive->{file} =~ m/^$NBD_TCP_PATH_RE_3$/) {
+        my $server = { type => 'inet', host => "$1", port => "$2" }; # port is also a string in QAPI
+        $blockdev = { driver => 'nbd', server => $server, export => "$3" };
+    } elsif ($drive->{file} eq 'cdrom') {
         my $path = PVE::QemuServer::Drive::get_iso_path($storecfg, $drive->{file});
         $blockdev = { driver => 'host_cdrom', filename => "$path" };
     } elsif ($drive->{file} =~ m|^/|) {
@@ -299,6 +317,7 @@ my sub generate_format_blockdev {
 
     die "generate_format_blockdev called without volid/path\n" if !$drive->{file};
     die "generate_format_blockdev called with 'none'\n" if $drive->{file} eq 'none';
+    die "generate_format_blockdev called with NBD path\n" if is_nbd($drive);
 
     my $scfg;
     my $format;
@@ -347,7 +366,9 @@ sub generate_drive_blockdev {
     die "generate_drive_blockdev called with 'none'\n" if $drive->{file} eq 'none';
 
     my $child = generate_file_blockdev($storecfg, $drive, $options);
-    $child = generate_format_blockdev($storecfg, $drive, $child, $options);
+    if (!is_nbd($drive)) {
+        $child = generate_format_blockdev($storecfg, $drive, $child, $options);
+    }
 
     if ($options->{'zero-initialized'}) {
         my $node_name = get_node_name('zeroinit', $drive_id, $drive->{file}, $options);
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 43/51] blockdev: add helper to generate PBS block device for live restore

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index f6495526..fa946e83 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -387,6 +387,23 @@ sub generate_drive_blockdev {
     };
 }
 
+sub generate_pbs_blockdev {
+    my ($pbs_conf, $pbs_name) = @_;
+
+    my $blockdev = {
+        driver => 'pbs',
+        'node-name' => "$pbs_name",
+        'read-only' => JSON::true,
+        archive => "$pbs_conf->{archive}",
+        repository => "$pbs_conf->{repository}",
+        snapshot => "$pbs_conf->{snapshot}",
+    };
+    $blockdev->{namespace} = "$pbs_conf->{namespace}" if $pbs_conf->{namespace};
+    $blockdev->{keyfile} = "$pbs_conf->{keyfile}" if $pbs_conf->{keyfile};
+
+    return $blockdev;
+}
+
 my sub blockdev_add {
     my ($vmid, $blockdev) = @_;
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 44/51] blockdev: support alloc-track driver for live-{import, restore}

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/Blockdev.pm | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index fa946e83..29410f89 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -123,10 +123,12 @@ my sub get_node_name {
     my $hash = substr(Digest::SHA::sha256_hex($info), 0, 30);
 
     my $prefix = "";
-    if ($type eq 'fmt') {
-        $prefix = 'f';
+    if ($type eq 'alloc-track') {
+        $prefix = 'a';
     } elsif ($type eq 'file') {
         $prefix = 'e';
+    } elsif ($type eq 'fmt') {
+        $prefix = 'f';
     } elsif ($type eq 'zeroinit') {
         $prefix = 'z';
     } else {
@@ -375,6 +377,17 @@ sub generate_drive_blockdev {
         $child = { driver => 'zeroinit', file => $child, 'node-name' => "$node_name" };
     }
 
+    if (my $live_restore = $options->{'live-restore'}) {
+        my $node_name = get_node_name('alloc-track', $drive_id, $drive->{file}, $options);
+        $child = {
+            driver => 'alloc-track',
+            'auto-remove' => JSON::true,
+            backing => $live_restore->{blockdev},
+            file => $child,
+            'node-name' => "$node_name",
+        };
+    }
+
     # for fleecing and TPM backup, this is already the top node
     return $child if $options->{fleecing} || $options->{'tpm-backup'} || $options->{'no-throttle'};
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 45/51] live import: also record volid information

[Fiona Ebner]

Will be required for generating the blockdev starting with machine
version 10.0.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/API2/Qemu.pm  | 2 ++
 src/PVE/QemuServer.pm | 3 ++-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 1aa3b358..2e6358e4 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -530,6 +530,7 @@ my sub create_disks : prototype($$$$$$$$$$$) {
                         $live_import_mapping->{$ds} = {
                             path => $path,
                             format => $source_format,
+                            volid => $source,
                         };
                         $live_import_mapping->{$ds}->{'delete-after-finish'} = $source
                             if $needs_extraction;
@@ -574,6 +575,7 @@ my sub create_disks : prototype($$$$$$$$$$$) {
                         $live_import_mapping->{$ds} = {
                             path => $source,
                             format => $source_format,
+                            volid => $source,
                         };
                     } else {
                         (undef, $dst_volid) = PVE::QemuServer::ImportDisk::do_import(
diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 4529e270..05c19390 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -7095,8 +7095,9 @@ sub live_import_from_files {
             if !exists($conf->{$dev});
 
         my $info = $mapping->{$dev};
-        my ($format, $path) = $info->@{qw(format path)};
+        my ($format, $path, $volid) = $info->@{qw(format path volid)};
         die "missing path for '$dev' mapping\n" if !$path;
+        die "missing volid for '$dev' mapping\n" if !$volid;
         die "missing format for '$dev' mapping\n" if !$format;
         die "invalid format '$format' for '$dev' mapping\n"
             if !grep { $format eq $_ } qw(raw qcow2 vmdk);
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 46/51] live import/restore: query which node to use for operation

[Fiona Ebner]

In preparation for the switch to -blockdev.

Otherwise, there would be an error:
> An error occurred during live-restore: VM 103 qmp command
> 'block-stream' failed - Permission conflict on node
> 'a25d9b2028b5a364dddbb033603b68c': permissions 'write' are both
> required by node 'drive-ide0' (uses node
> 'a25d9b2028b5a364dddbb033603b68c' as 'file' child) and unshared
> by stream job 'restore-drive-ide0' (uses node
> 'a25d9b2028b5a364dddbb033603b68c' as 'intermediate node' child).

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 05c19390..d4154aeb 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -7037,11 +7037,12 @@ sub pbs_live_restore {
         # removes itself once all backing images vanish with 'auto-remove=on')
         my $jobs = {};
         for my $ds (sort keys %$restored_disks) {
+            my $node_name = PVE::QemuServer::Blockdev::get_node_name_below_throttle($vmid, $ds);
             my $job_id = "restore-$ds";
             mon_cmd(
                 $vmid, 'block-stream',
                 'job-id' => $job_id,
-                device => "$ds",
+                device => "$node_name",
                 'auto-dismiss' => JSON::false,
             );
             $jobs->{$job_id} = {};
@@ -7138,11 +7139,13 @@ sub live_import_from_files {
         # removes itself once all backing images vanish with 'auto-remove=on')
         my $jobs = {};
         for my $ds (sort keys %$live_restore_backing) {
+            my $node_name =
+                PVE::QemuServer::Blockdev::get_node_name_below_throttle($vmid, "drive-$ds");
             my $job_id = "restore-$ds";
             mon_cmd(
                 $vmid, 'block-stream',
                 'job-id' => $job_id,
-                device => "drive-$ds",
+                device => "$node_name",
                 'auto-dismiss' => JSON::false,
             );
             $jobs->{$job_id} = {};
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 47/51] live import/restore: use Blockdev::detach helper

[Fiona Ebner]

Which also catches scenarios where the block node is already gone,
which can happen with -blockdev.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index d4154aeb..d9284843 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -7057,7 +7057,7 @@ sub pbs_live_restore {
             . " to disconnect from Proxmox Backup Server\n";
 
         for my $ds (sort keys %$restored_disks) {
-            mon_cmd($vmid, 'blockdev-del', 'node-name' => "$ds-pbs");
+            PVE::QemuServer::Blockdev::detach($vmid, "$ds-pbs");
         }
 
         close($qmeventd_fd);
@@ -7159,7 +7159,7 @@ sub live_import_from_files {
         print "restore-drive jobs finished successfully, removing all tracking block devices\n";
 
         for my $ds (sort keys %$live_restore_backing) {
-            mon_cmd($vmid, 'blockdev-del', 'node-name' => "drive-$ds-restore");
+            PVE::QemuServer::Blockdev::detach($vmid, "drive-$ds-restore");
         }
 
         close($qmeventd_fd);
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 48/51] command line: switch to blockdev starting with machine version 10.0

[Fiona Ebner]

Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm                         | 129 +++++++++++++-----
 src/PVE/QemuServer/BlockJob.pm                |  32 +++--
 src/PVE/QemuServer/Blockdev.pm                | 101 +++++++++-----
 src/PVE/QemuServer/OVMF.pm                    |  21 ++-
 src/test/MigrationTest/QemuMigrateMock.pm     |   5 +
 src/test/cfg2cmd/aio.conf.cmd                 |  42 ++++--
 src/test/cfg2cmd/bootorder-empty.conf.cmd     |  10 +-
 src/test/cfg2cmd/bootorder-legacy.conf.cmd    |  10 +-
 src/test/cfg2cmd/bootorder.conf.cmd           |  10 +-
 ...putype-icelake-client-deprecation.conf.cmd |   4 +-
 src/test/cfg2cmd/efi-raw-template.conf.cmd    |   7 +-
 src/test/cfg2cmd/efi-raw.conf.cmd             |   7 +-
 .../cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd  |   7 +-
 src/test/cfg2cmd/efi-secboot-and-tpm.conf.cmd |   7 +-
 src/test/cfg2cmd/efidisk-on-rbd.conf.cmd      |   7 +-
 src/test/cfg2cmd/ide.conf.cmd                 |  15 +-
 src/test/cfg2cmd/q35-ide.conf.cmd             |  15 +-
 .../q35-linux-hostpci-mapping.conf.cmd        |   7 +-
 .../q35-linux-hostpci-multifunction.conf.cmd  |   7 +-
 .../q35-linux-hostpci-template.conf.cmd       |  10 +-
 ...q35-linux-hostpci-x-pci-overrides.conf.cmd |   7 +-
 src/test/cfg2cmd/q35-linux-hostpci.conf.cmd   |   7 +-
 src/test/cfg2cmd/q35-simple.conf.cmd          |   7 +-
 src/test/cfg2cmd/seabios_serial.conf.cmd      |   4 +-
 src/test/cfg2cmd/sev-es.conf.cmd              |   7 +-
 src/test/cfg2cmd/sev-std.conf.cmd             |   7 +-
 src/test/cfg2cmd/simple-btrfs.conf.cmd        |  13 +-
 src/test/cfg2cmd/simple-cifs.conf.cmd         |  13 +-
 .../cfg2cmd/simple-disk-passthrough.conf.cmd  |   9 +-
 src/test/cfg2cmd/simple-lvm.conf.cmd          |  12 +-
 src/test/cfg2cmd/simple-lvmthin.conf.cmd      |  12 +-
 src/test/cfg2cmd/simple-rbd.conf.cmd          |  25 ++--
 src/test/cfg2cmd/simple-virtio-blk.conf.cmd   |   4 +-
 .../cfg2cmd/simple-zfs-over-iscsi.conf.cmd    |  13 +-
 src/test/cfg2cmd/simple1-template.conf.cmd    |   7 +-
 src/test/cfg2cmd/simple1.conf.cmd             |   4 +-
 src/test/run_config2command_tests.pl          |  19 +++
 37 files changed, 417 insertions(+), 206 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index d9284843..d4ad4a25 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -3640,19 +3640,40 @@ sub config_to_command {
             }
 
             my $live_restore = $live_restore_backing->{$ds};
-            my $live_blockdev_name = undef;
-            if ($live_restore) {
-                $live_blockdev_name = $live_restore->{name};
-                push @$devices, '-blockdev', $live_restore->{blockdev};
+
+            if (min_version($machine_version, 10, 0)) { # for the switch to -blockdev
+                if ($drive->{file} ne 'none') {
+                    my $throttle_group =
+                        PVE::QemuServer::Blockdev::generate_throttle_group($drive);
+                    push @$cmd, '-object', to_json($throttle_group, { canonical => 1 });
+
+                    my $extra_blockdev_options = {};
+                    $extra_blockdev_options->{'live-restore'} = $live_restore if $live_restore;
+                    # extra protection for templates, but SATA and IDE don't support it..
+                    $extra_blockdev_options->{'read-only'} = 1
+                        if drive_is_read_only($conf, $drive);
+
+                    my $blockdev = PVE::QemuServer::Blockdev::generate_drive_blockdev(
+                        $storecfg, $drive, $extra_blockdev_options,
+                    );
+                    push @$devices, '-blockdev', to_json($blockdev, { canonical => 1 });
+                }
+            } else {
+                my $live_blockdev_name = undef;
+                if ($live_restore) {
+                    $live_blockdev_name = $live_restore->{name};
+                    push @$devices, '-blockdev', $live_restore->{blockdev};
+                }
+
+                my $drive_cmd =
+                    print_drive_commandline_full($storecfg, $vmid, $drive, $live_blockdev_name);
+
+                # extra protection for templates, but SATA and IDE don't support it..
+                $drive_cmd .= ',readonly=on' if drive_is_read_only($conf, $drive);
+
+                push @$devices, '-drive', $drive_cmd;
             }
 
-            my $drive_cmd =
-                print_drive_commandline_full($storecfg, $vmid, $drive, $live_blockdev_name);
-
-            # extra protection for templates, but SATA and IDE don't support it..
-            $drive_cmd .= ',readonly=on' if drive_is_read_only($conf, $drive);
-
-            push @$devices, '-drive', $drive_cmd;
             push @$devices, '-device',
                 print_drivedevice_full(
                     $storecfg, $conf, $vmid, $drive, $bridges, $arch, $machine_type,
@@ -4050,28 +4071,44 @@ sub qemu_iothread_del {
 sub qemu_driveadd {
     my ($storecfg, $vmid, $device) = @_;
 
-    my $drive = print_drive_commandline_full($storecfg, $vmid, $device, undef);
-    $drive =~ s/\\/\\\\/g;
-    my $ret = PVE::QemuServer::Monitor::hmp_cmd($vmid, "drive_add auto \"$drive\"", 60);
+    my $machine_type = PVE::QemuServer::Machine::get_current_qemu_machine($vmid);
 
-    # If the command succeeds qemu prints: "OK"
-    return 1 if $ret =~ m/OK/s;
+    # for the switch to -blockdev
+    if (PVE::QemuServer::Machine::is_machine_version_at_least($machine_type, 10, 0)) {
+        PVE::QemuServer::Blockdev::attach($storecfg, $vmid, $device, {});
+        return 1;
+    } else {
+        my $drive = print_drive_commandline_full($storecfg, $vmid, $device, undef);
+        $drive =~ s/\\/\\\\/g;
+        my $ret = PVE::QemuServer::Monitor::hmp_cmd($vmid, "drive_add auto \"$drive\"", 60);
 
-    die "adding drive failed: $ret\n";
+        # If the command succeeds qemu prints: "OK"
+        return 1 if $ret =~ m/OK/s;
+
+        die "adding drive failed: $ret\n";
+    }
 }
 
 sub qemu_drivedel {
     my ($vmid, $deviceid) = @_;
 
-    my $ret = PVE::QemuServer::Monitor::hmp_cmd($vmid, "drive_del drive-$deviceid", 10 * 60);
-    $ret =~ s/^\s+//;
+    my $machine_type = PVE::QemuServer::Machine::get_current_qemu_machine($vmid);
 
-    return 1 if $ret eq "";
+    # for the switch to -blockdev
+    if (PVE::QemuServer::Machine::is_machine_version_at_least($machine_type, 10, 0)) {
+        PVE::QemuServer::Blockdev::detach($vmid, "drive-$deviceid");
+        return 1;
+    } else {
+        my $ret = PVE::QemuServer::Monitor::hmp_cmd($vmid, "drive_del drive-$deviceid", 10 * 60);
+        $ret =~ s/^\s+//;
 
-    # NB: device not found errors mean the drive was auto-deleted and we ignore the error
-    return 1 if $ret =~ m/Device \'.*?\' not found/s;
+        return 1 if $ret eq "";
 
-    die "deleting drive $deviceid failed : $ret\n";
+        # NB: device not found errors mean the drive was auto-deleted and we ignore the error
+        return 1 if $ret =~ m/Device \'.*?\' not found/s;
+
+        die "deleting drive $deviceid failed : $ret\n";
+    }
 }
 
 sub qemu_deviceaddverify {
@@ -7006,10 +7043,22 @@ sub pbs_live_restore {
         print "restoring '$ds' to '$drive->{file}'\n";
 
         my $pbs_name = "drive-${confname}-pbs";
-        $live_restore_backing->{$confname} = {
-            name => $pbs_name,
-            blockdev => print_pbs_blockdev($pbs_conf, $pbs_name),
-        };
+
+        $live_restore_backing->{$confname} = { name => $pbs_name };
+
+        # add blockdev information
+        my $machine_type = PVE::QemuServer::Machine::get_vm_machine($conf, undef, $conf->{arch});
+        my $machine_version = PVE::QemuServer::Machine::extract_version(
+            $machine_type,
+            PVE::QemuServer::Helpers::kvm_user_version(),
+        );
+        if (min_version($machine_version, 10, 0)) { # for the switch to -blockdev
+            $live_restore_backing->{$confname}->{blockdev} =
+                PVE::QemuServer::Blockdev::generate_pbs_blockdev($pbs_conf, $pbs_name);
+        } else {
+            $live_restore_backing->{$confname}->{blockdev} =
+                print_pbs_blockdev($pbs_conf, $pbs_name);
+        }
     }
 
     my $drives_streamed = 0;
@@ -7086,6 +7135,8 @@ sub pbs_live_restore {
 sub live_import_from_files {
     my ($mapping, $vmid, $conf, $restore_options) = @_;
 
+    my $storecfg = PVE::Storage::config();
+
     my $live_restore_backing = {};
     my $sources_to_remove = [];
     for my $dev (keys %$mapping) {
@@ -7103,18 +7154,30 @@ sub live_import_from_files {
         die "invalid format '$format' for '$dev' mapping\n"
             if !grep { $format eq $_ } qw(raw qcow2 vmdk);
 
-        $live_restore_backing->{$dev} = {
-            name => "drive-$dev-restore",
-            blockdev => "driver=$format,node-name=drive-$dev-restore"
+        $live_restore_backing->{$dev} = { name => "drive-$dev-restore" };
+
+        my $machine_type = PVE::QemuServer::Machine::get_vm_machine($conf, undef, $conf->{arch});
+        my $machine_version = PVE::QemuServer::Machine::extract_version(
+            $machine_type,
+            PVE::QemuServer::Helpers::kvm_user_version(),
+        );
+        if (min_version($machine_version, 10, 0)) { # for the switch to -blockdev
+            my ($interface, $index) = PVE::QemuServer::Drive::parse_drive_interface($dev);
+            my $drive = { file => $volid, interface => $interface, index => $index };
+            my $blockdev =
+                PVE::QemuServer::Blockdev::generate_drive_blockdev($storecfg, $drive, {});
+            $live_restore_backing->{$dev}->{blockdev} = $blockdev;
+        } else {
+            $live_restore_backing->{$dev}->{blockdev} =
+                "driver=$format,node-name=drive-$dev-restore"
                 . ",read-only=on"
-                . ",file.driver=file,file.filename=$path",
-        };
+                . ",file.driver=file,file.filename=$path";
+        }
 
         my $source_volid = $info->{'delete-after-finish'};
         push $sources_to_remove->@*, $source_volid if defined($source_volid);
     }
 
-    my $storecfg = PVE::Storage::config();
     eval {
 
         # make sure HA doesn't interrupt our restore by stopping the VM
diff --git a/src/PVE/QemuServer/BlockJob.pm b/src/PVE/QemuServer/BlockJob.pm
index 9131780e..fc12f310 100644
--- a/src/PVE/QemuServer/BlockJob.pm
+++ b/src/PVE/QemuServer/BlockJob.pm
@@ -492,20 +492,24 @@ sub mirror {
     my ($source, $dest, $jobs, $completion, $options) = @_;
 
     # for the switch to -blockdev
-
-    my $drive_id = PVE::QemuServer::Drive::get_drive_id($source->{drive});
-    qemu_drive_mirror(
-        $source->{vmid},
-        $drive_id,
-        $dest->{volid},
-        $dest->{vmid},
-        $dest->{'zero-initialized'},
-        $jobs,
-        $completion,
-        $options->{'guest-agent'},
-        $options->{bwlimit},
-        $source->{bitmap},
-    );
+    my $machine_type = PVE::QemuServer::Machine::get_current_qemu_machine($source->{vmid});
+    if (PVE::QemuServer::Machine::is_machine_version_at_least($machine_type, 10, 0)) {
+        blockdev_mirror($source, $dest, $jobs, $completion, $options);
+    } else {
+        my $drive_id = PVE::QemuServer::Drive::get_drive_id($source->{drive});
+        qemu_drive_mirror(
+            $source->{vmid},
+            $drive_id,
+            $dest->{volid},
+            $dest->{vmid},
+            $dest->{'zero-initialized'},
+            $jobs,
+            $completion,
+            $options->{'guest-agent'},
+            $options->{bwlimit},
+            $source->{bitmap},
+        );
+    }
 }
 
 1;
diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 29410f89..ab58e3ea 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -631,18 +631,24 @@ my sub blockdev_change_medium {
 sub change_medium {
     my ($storecfg, $vmid, $qdev_id, $drive) = @_;
 
-    # force eject if locked
-    mon_cmd($vmid, "eject", force => JSON::true, id => "$qdev_id");
+    my $machine_type = PVE::QemuServer::Machine::get_current_qemu_machine($vmid);
+    # for the switch to -blockdev
+    if (PVE::QemuServer::Machine::is_machine_version_at_least($machine_type, 10, 0)) {
+        blockdev_change_medium($storecfg, $vmid, $qdev_id, $drive);
+    } else {
+        # force eject if locked
+        mon_cmd($vmid, "eject", force => JSON::true, id => "$qdev_id");
 
-    my ($path, $format) = PVE::QemuServer::Drive::get_path_and_format($storecfg, $drive);
+        my ($path, $format) = PVE::QemuServer::Drive::get_path_and_format($storecfg, $drive);
 
-    if ($path) { # no path for 'none'
-        mon_cmd(
-            $vmid, "blockdev-change-medium",
-            id => "$qdev_id",
-            filename => "$path",
-            format => "$format",
-        );
+        if ($path) { # no path for 'none'
+            mon_cmd(
+                $vmid, "blockdev-change-medium",
+                id => "$qdev_id",
+                filename => "$path",
+                format => "$format",
+            );
+        }
     }
 }
 
@@ -672,28 +678,59 @@ sub set_io_throttle {
 
     return if !PVE::QemuServer::Helpers::vm_running_locally($vmid);
 
-    mon_cmd(
-        $vmid, "block_set_io_throttle",
-        device => $deviceid,
-        bps => int($bps),
-        bps_rd => int($bps_rd),
-        bps_wr => int($bps_wr),
-        iops => int($iops),
-        iops_rd => int($iops_rd),
-        iops_wr => int($iops_wr),
-        bps_max => int($bps_max),
-        bps_rd_max => int($bps_rd_max),
-        bps_wr_max => int($bps_wr_max),
-        iops_max => int($iops_max),
-        iops_rd_max => int($iops_rd_max),
-        iops_wr_max => int($iops_wr_max),
-        bps_max_length => int($bps_max_length),
-        bps_rd_max_length => int($bps_rd_max_length),
-        bps_wr_max_length => int($bps_wr_max_length),
-        iops_max_length => int($iops_max_length),
-        iops_rd_max_length => int($iops_rd_max_length),
-        iops_wr_max_length => int($iops_wr_max_length),
-    );
+    my $machine_type = PVE::QemuServer::Machine::get_current_qemu_machine($vmid);
+    # for the switch to -blockdev
+    if (PVE::QemuServer::Machine::is_machine_version_at_least($machine_type, 10, 0)) {
+        mon_cmd(
+            $vmid,
+            'qom-set',
+            path => "throttle-$deviceid",
+            property => "limits",
+            value => {
+                'bps-total' => int($bps),
+                'bps-read' => int($bps_rd),
+                'bps-write' => int($bps_wr),
+                'iops-total' => int($iops),
+                'iops-read' => int($iops_rd),
+                'iops-write' => int($iops_wr),
+                'bps-total-max' => int($bps_max),
+                'bps-read-max' => int($bps_rd_max),
+                'bps-write-max' => int($bps_wr_max),
+                'iops-total-max' => int($iops_max),
+                'iops-read-max' => int($iops_rd_max),
+                'iops-write-max' => int($iops_wr_max),
+                'bps-total-max-length' => int($bps_max_length),
+                'bps-read-max-length' => int($bps_rd_max_length),
+                'bps-write-max-length' => int($bps_wr_max_length),
+                'iops-total-max-length' => int($iops_max_length),
+                'iops-read-max-length' => int($iops_rd_max_length),
+                'iops-write-max-length' => int($iops_wr_max_length),
+            },
+        );
+    } else {
+        mon_cmd(
+            $vmid, "block_set_io_throttle",
+            device => $deviceid,
+            bps => int($bps),
+            bps_rd => int($bps_rd),
+            bps_wr => int($bps_wr),
+            iops => int($iops),
+            iops_rd => int($iops_rd),
+            iops_wr => int($iops_wr),
+            bps_max => int($bps_max),
+            bps_rd_max => int($bps_rd_max),
+            bps_wr_max => int($bps_wr_max),
+            iops_max => int($iops_max),
+            iops_rd_max => int($iops_rd_max),
+            iops_wr_max => int($iops_wr_max),
+            bps_max_length => int($bps_max_length),
+            bps_rd_max_length => int($bps_rd_max_length),
+            bps_wr_max_length => int($bps_wr_max_length),
+            iops_max_length => int($iops_max_length),
+            iops_rd_max_length => int($iops_rd_max_length),
+            iops_wr_max_length => int($iops_wr_max_length),
+        );
+    }
 }
 
 1;
diff --git a/src/PVE/QemuServer/OVMF.pm b/src/PVE/QemuServer/OVMF.pm
index dde81eb7..a7239614 100644
--- a/src/PVE/QemuServer/OVMF.pm
+++ b/src/PVE/QemuServer/OVMF.pm
@@ -3,7 +3,7 @@ package PVE::QemuServer::OVMF;
 use strict;
 use warnings;
 
-use JSON;
+use JSON qw(to_json);
 
 use PVE::RESTEnvironment qw(log_warn);
 use PVE::Storage;
@@ -210,10 +210,21 @@ sub print_ovmf_commandline {
         }
         push $cmd->@*, '-bios', get_ovmf_files($hw_info->{arch}, undef, undef, $amd_sev_type);
     } else {
-        my ($code_drive_str, $var_drive_str) =
-            print_ovmf_drive_commandlines($conf, $storecfg, $vmid, $hw_info, $version_guard);
-        push $cmd->@*, '-drive', $code_drive_str;
-        push $cmd->@*, '-drive', $var_drive_str;
+        if ($version_guard->(10, 0, 0)) { # for the switch to -blockdev
+            my ($code_blockdev, $vars_blockdev, $throttle_group) =
+                generate_ovmf_blockdev($conf, $storecfg, $vmid, $hw_info);
+
+            push $cmd->@*, '-object', to_json($throttle_group, { canonical => 1 });
+            push $cmd->@*, '-blockdev', to_json($code_blockdev, { canonical => 1 });
+            push $cmd->@*, '-blockdev', to_json($vars_blockdev, { canonical => 1 });
+            push $machine_flags->@*, "pflash0=$code_blockdev->{'node-name'}",
+                "pflash1=$vars_blockdev->{'node-name'}";
+        } else {
+            my ($code_drive_str, $var_drive_str) =
+                print_ovmf_drive_commandlines($conf, $storecfg, $vmid, $hw_info, $version_guard);
+            push $cmd->@*, '-drive', $code_drive_str;
+            push $cmd->@*, '-drive', $var_drive_str;
+        }
     }
 
     return ($cmd, $machine_flags);
diff --git a/src/test/MigrationTest/QemuMigrateMock.pm b/src/test/MigrationTest/QemuMigrateMock.pm
index c52df84b..b04cf78b 100644
--- a/src/test/MigrationTest/QemuMigrateMock.pm
+++ b/src/test/MigrationTest/QemuMigrateMock.pm
@@ -215,6 +215,11 @@ $qemu_server_machine_module->mock(
             if !defined($vm_status->{runningmachine});
         return $vm_status->{runningmachine};
     },
+    get_current_qemu_machine => sub {
+        die "invalid test: no runningmachine specified\n"
+            if !defined($vm_status->{runningmachine});
+        return $vm_status->{runningmachine};
+    },
 );
 
 my $qemu_server_network_module = Test::MockModule->new("PVE::QemuServer::Network");
diff --git a/src/test/cfg2cmd/aio.conf.cmd b/src/test/cfg2cmd/aio.conf.cmd
index c199bacf..272c6cd6 100644
--- a/src/test/cfg2cmd/aio.conf.cmd
+++ b/src/test/cfg2cmd/aio.conf.cmd
@@ -14,6 +14,20 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 512 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi1","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi3","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi4","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi5","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi6","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi7","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi8","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi9","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi10","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi11","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi12","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi13","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -24,33 +38,33 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -device 'lsi,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.raw,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=threads,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.raw","node-name":"e3b2553803d55d43b9986a0aac3e9a7","read-only":false},"node-name":"f3b2553803d55d43b9986a0aac3e9a7","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=0,drive=drive-scsi0,id=scsi0,write-cache=on' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-1.raw,if=none,id=drive-scsi1,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"native","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-1.raw","node-name":"e08707d013893852b3d4d42301a4298","read-only":false},"node-name":"f08707d013893852b3d4d42301a4298","read-only":false},"node-name":"drive-scsi1","throttle-group":"throttle-drive-scsi1"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=1,drive=drive-scsi1,id=scsi1,write-cache=on' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-2.raw,if=none,id=drive-scsi2,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-2.raw","node-name":"edb0854bba55e8b2544ad937c9f5afc","read-only":false},"node-name":"fdb0854bba55e8b2544ad937c9f5afc","read-only":false},"node-name":"drive-scsi2","throttle-group":"throttle-drive-scsi2"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=2,drive=drive-scsi2,id=scsi2,write-cache=on' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-3.raw,if=none,id=drive-scsi3,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-3.raw","node-name":"e9c170cb9491763cad3f31718205efc","read-only":false},"node-name":"f9c170cb9491763cad3f31718205efc","read-only":false},"node-name":"drive-scsi3","throttle-group":"throttle-drive-scsi3"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=3,drive=drive-scsi3,id=scsi3,write-cache=on' \
-  -drive 'file=/mnt/pve/cifs-store/images/8006/vm-8006-disk-4.raw,if=none,id=drive-scsi4,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"native","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/mnt/pve/cifs-store/images/8006/vm-8006-disk-4.raw","node-name":"ea34ecc24c40da0d53420ef344ced37","read-only":false},"node-name":"fa34ecc24c40da0d53420ef344ced37","read-only":false},"node-name":"drive-scsi4","throttle-group":"throttle-drive-scsi4"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=4,drive=drive-scsi4,id=scsi4,write-cache=on' \
-  -drive 'file=/mnt/pve/cifs-store/images/8006/vm-8006-disk-5.raw,if=none,id=drive-scsi5,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/mnt/pve/cifs-store/images/8006/vm-8006-disk-5.raw","node-name":"e39cacf47a4f4877072601505d90949","read-only":false},"node-name":"f39cacf47a4f4877072601505d90949","read-only":false},"node-name":"drive-scsi5","throttle-group":"throttle-drive-scsi5"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=5,drive=drive-scsi5,id=scsi5,write-cache=on' \
-  -drive 'file=/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-6,if=none,id=drive-scsi6,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-6","node-name":"e7db1ee70981087e4a2861bc7da417b","read-only":false},"node-name":"f7db1ee70981087e4a2861bc7da417b","read-only":false},"node-name":"drive-scsi6","throttle-group":"throttle-drive-scsi6"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=6,drive=drive-scsi6,id=scsi6,write-cache=on' \
   -device 'lsi,id=scsihw1,bus=pci.0,addr=0x6' \
-  -drive 'file=/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-7,if=none,id=drive-scsi7,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-7","node-name":"e2d2deac808301140a96c862fe3ea85","read-only":false},"node-name":"f2d2deac808301140a96c862fe3ea85","read-only":false},"node-name":"drive-scsi7","throttle-group":"throttle-drive-scsi7"}' \
   -device 'scsi-hd,bus=scsihw1.0,scsi-id=0,drive=drive-scsi7,id=scsi7,write-cache=on' \
-  -drive 'file=/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-8,if=none,id=drive-scsi8,cache=writeback,discard=on,format=raw,aio=threads,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-8","node-name":"e9796b73db57b8943746ede7d0d3060","read-only":false},"node-name":"f9796b73db57b8943746ede7d0d3060","read-only":false},"node-name":"drive-scsi8","throttle-group":"throttle-drive-scsi8"}' \
   -device 'scsi-hd,bus=scsihw1.0,scsi-id=1,drive=drive-scsi8,id=scsi8,write-cache=on' \
-  -drive 'file=/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-9,if=none,id=drive-scsi9,cache=writeback,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-9","node-name":"efa538892acc012edbdc5810035bf7d","read-only":false},"node-name":"ffa538892acc012edbdc5810035bf7d","read-only":false},"node-name":"drive-scsi9","throttle-group":"throttle-drive-scsi9"}' \
   -device 'scsi-hd,bus=scsihw1.0,scsi-id=2,drive=drive-scsi9,id=scsi9,write-cache=on' \
-  -drive 'file=rbd:cpool/vm-8006-disk-8:mon_host=127.0.0.42;127.0.0.21;[\:\:1]:auth_supported=none,if=none,id=drive-scsi10,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"auth-client-required":["none"],"cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"rbd","image":"vm-8006-disk-8","node-name":"e6f4cbffa741d16bba69304eb2800ef","pool":"cpool","read-only":false,"server":[{"host":"127.0.0.42","port":"3300"},{"host":"127.0.0.21","port":"3300"},{"host":"::1","port":"3300"}]},"node-name":"f6f4cbffa741d16bba69304eb2800ef","read-only":false},"node-name":"drive-scsi10","throttle-group":"throttle-drive-scsi10"}' \
   -device 'scsi-hd,bus=scsihw1.0,scsi-id=3,drive=drive-scsi10,id=scsi10,write-cache=on' \
-  -drive 'file=rbd:cpool/vm-8006-disk-8:mon_host=127.0.0.42;127.0.0.21;[\:\:1]:auth_supported=none,if=none,id=drive-scsi11,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"auth-client-required":["none"],"cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"rbd","image":"vm-8006-disk-8","node-name":"e42375c54de70f5f4be966d98c90255","pool":"cpool","read-only":false,"server":[{"host":"127.0.0.42","port":"3300"},{"host":"127.0.0.21","port":"3300"},{"host":"::1","port":"3300"}]},"node-name":"f42375c54de70f5f4be966d98c90255","read-only":false},"node-name":"drive-scsi11","throttle-group":"throttle-drive-scsi11"}' \
   -device 'scsi-hd,bus=scsihw1.0,scsi-id=4,drive=drive-scsi11,id=scsi11,write-cache=on' \
-  -drive 'file=/dev/veegee/vm-8006-disk-9,if=none,id=drive-scsi12,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"native","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/veegee/vm-8006-disk-9","node-name":"ed7b2c9e0133619fcf6cb8ce5903502","read-only":false},"node-name":"fd7b2c9e0133619fcf6cb8ce5903502","read-only":false},"node-name":"drive-scsi12","throttle-group":"throttle-drive-scsi12"}' \
   -device 'scsi-hd,bus=scsihw1.0,scsi-id=5,drive=drive-scsi12,id=scsi12,write-cache=on' \
-  -drive 'file=/dev/veegee/vm-8006-disk-9,if=none,id=drive-scsi13,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/veegee/vm-8006-disk-9","node-name":"ed85420a880203ca1401d00a8edf132","read-only":false},"node-name":"fd85420a880203ca1401d00a8edf132","read-only":false},"node-name":"drive-scsi13","throttle-group":"throttle-drive-scsi13"}' \
   -device 'scsi-hd,bus=scsihw1.0,scsi-id=6,drive=drive-scsi13,id=scsi13,write-cache=on' \
   -machine 'type=pc+pve0'
diff --git a/src/test/cfg2cmd/bootorder-empty.conf.cmd b/src/test/cfg2cmd/bootorder-empty.conf.cmd
index 3f8fdb8e..84b38221 100644
--- a/src/test/cfg2cmd/bootorder-empty.conf.cmd
+++ b/src/test/cfg2cmd/bootorder-empty.conf.cmd
@@ -15,8 +15,11 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
+  -object '{"id":"throttle-drive-scsi4","limits":{},"qom-type":"throttle-group"}' \
   -object 'iothread,id=iothread-virtio0' \
+  -object '{"id":"throttle-drive-virtio0","limits":{},"qom-type":"throttle-group"}' \
   -object 'iothread,id=iothread-virtio1' \
+  -object '{"id":"throttle-drive-virtio1","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -27,14 +30,13 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2' \
   -device 'lsi,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi4,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"e6bf62e20f6c14a2c19bd6f1f5ac36c","read-only":false},"node-name":"f6bf62e20f6c14a2c19bd6f1f5ac36c","read-only":false},"node-name":"drive-scsi4","throttle-group":"throttle-drive-scsi4"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=4,drive=drive-scsi4,id=scsi4,write-cache=on' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-virtio0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"edd19f6c1b3a6d5a6248c3376a91a16","read-only":false},"node-name":"fdd19f6c1b3a6d5a6248c3376a91a16","read-only":false},"node-name":"drive-virtio0","throttle-group":"throttle-drive-virtio0"}' \
   -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,write-cache=on' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-virtio1,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"eeb683fb9c516c1a8707c917f0d7a38","read-only":false},"node-name":"feb683fb9c516c1a8707c917f0d7a38","read-only":false},"node-name":"drive-virtio1","throttle-group":"throttle-drive-virtio1"}' \
   -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb,iothread=iothread-virtio1,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256' \
diff --git a/src/test/cfg2cmd/bootorder-legacy.conf.cmd b/src/test/cfg2cmd/bootorder-legacy.conf.cmd
index cd990cd8..5f64a62f 100644
--- a/src/test/cfg2cmd/bootorder-legacy.conf.cmd
+++ b/src/test/cfg2cmd/bootorder-legacy.conf.cmd
@@ -15,8 +15,11 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
+  -object '{"id":"throttle-drive-scsi4","limits":{},"qom-type":"throttle-group"}' \
   -object 'iothread,id=iothread-virtio0' \
+  -object '{"id":"throttle-drive-virtio0","limits":{},"qom-type":"throttle-group"}' \
   -object 'iothread,id=iothread-virtio1' \
+  -object '{"id":"throttle-drive-virtio1","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -27,14 +30,13 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'lsi,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi4,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"e6bf62e20f6c14a2c19bd6f1f5ac36c","read-only":false},"node-name":"f6bf62e20f6c14a2c19bd6f1f5ac36c","read-only":false},"node-name":"drive-scsi4","throttle-group":"throttle-drive-scsi4"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=4,drive=drive-scsi4,id=scsi4,write-cache=on' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-virtio0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"edd19f6c1b3a6d5a6248c3376a91a16","read-only":false},"node-name":"fdd19f6c1b3a6d5a6248c3376a91a16","read-only":false},"node-name":"drive-virtio0","throttle-group":"throttle-drive-virtio0"}' \
   -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,write-cache=on' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-virtio1,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"eeb683fb9c516c1a8707c917f0d7a38","read-only":false},"node-name":"feb683fb9c516c1a8707c917f0d7a38","read-only":false},"node-name":"drive-virtio1","throttle-group":"throttle-drive-virtio1"}' \
   -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb,iothread=iothread-virtio1,bootindex=302,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=100' \
diff --git a/src/test/cfg2cmd/bootorder.conf.cmd b/src/test/cfg2cmd/bootorder.conf.cmd
index 3cef2161..c0f19a04 100644
--- a/src/test/cfg2cmd/bootorder.conf.cmd
+++ b/src/test/cfg2cmd/bootorder.conf.cmd
@@ -15,8 +15,11 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
+  -object '{"id":"throttle-drive-scsi4","limits":{},"qom-type":"throttle-group"}' \
   -object 'iothread,id=iothread-virtio0' \
+  -object '{"id":"throttle-drive-virtio0","limits":{},"qom-type":"throttle-group"}' \
   -object 'iothread,id=iothread-virtio1' \
+  -object '{"id":"throttle-drive-virtio1","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -27,14 +30,13 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=103' \
   -device 'lsi,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi4,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"e6bf62e20f6c14a2c19bd6f1f5ac36c","read-only":false},"node-name":"f6bf62e20f6c14a2c19bd6f1f5ac36c","read-only":false},"node-name":"drive-scsi4","throttle-group":"throttle-drive-scsi4"}' \
   -device 'scsi-hd,bus=scsihw0.0,scsi-id=4,drive=drive-scsi4,id=scsi4,bootindex=102,write-cache=on' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-virtio0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"edd19f6c1b3a6d5a6248c3376a91a16","read-only":false},"node-name":"fdd19f6c1b3a6d5a6248c3376a91a16","read-only":false},"node-name":"drive-virtio0","throttle-group":"throttle-drive-virtio0"}' \
   -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,write-cache=on' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-virtio1,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"eeb683fb9c516c1a8707c917f0d7a38","read-only":false},"node-name":"feb683fb9c516c1a8707c917f0d7a38","read-only":false},"node-name":"drive-virtio1","throttle-group":"throttle-drive-virtio1"}' \
   -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb,iothread=iothread-virtio1,bootindex=100,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=101' \
diff --git a/src/test/cfg2cmd/cputype-icelake-client-deprecation.conf.cmd b/src/test/cfg2cmd/cputype-icelake-client-deprecation.conf.cmd
index e6e09278..5269e673 100644
--- a/src/test/cfg2cmd/cputype-icelake-client-deprecation.conf.cmd
+++ b/src/test/cfg2cmd/cputype-icelake-client-deprecation.conf.cmd
@@ -15,6 +15,7 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu 'Icelake-Server,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,vendor=GenuineIntel' \
   -m 768 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -25,9 +26,8 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/8006/base-8006-disk-0.qcow2,if=none,id=drive-scsi0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/base-8006-disk-0.qcow2","node-name":"e417d5947e69c5890b1e3ddf8a68167","read-only":false},"node-name":"f417d5947e69c5890b1e3ddf8a68167","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
   -machine 'type=pc+pve0'
diff --git a/src/test/cfg2cmd/efi-raw-template.conf.cmd b/src/test/cfg2cmd/efi-raw-template.conf.cmd
index f66cbb0d..b6064f98 100644
--- a/src/test/cfg2cmd/efi-raw-template.conf.cmd
+++ b/src/test/cfg2cmd/efi-raw-template.conf.cmd
@@ -8,8 +8,9 @@
   -mon 'chardev=qmp-event,mode=control' \
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/var/lib/vz/images/100/base-disk-100-0.raw,size=131072,readonly=on' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/base-disk-100-0.raw","node-name":"e3bd051dc2860cd423537bc00138c50","read-only":true},"node-name":"f3bd051dc2860cd423537bc00138c50","read-only":true,"size":131072},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -smp '1,sockets=1,cores=1,maxcpus=1' \
   -nodefaults \
   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
@@ -25,5 +26,5 @@
   -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -machine 'accel=tcg,type=pc+pve0' \
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,accel=tcg,type=pc+pve0' \
   -snapshot
diff --git a/src/test/cfg2cmd/efi-raw.conf.cmd b/src/test/cfg2cmd/efi-raw.conf.cmd
index 6406686d..c10df1cb 100644
--- a/src/test/cfg2cmd/efi-raw.conf.cmd
+++ b/src/test/cfg2cmd/efi-raw.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=7b10d7af-b932-4c66-b2c3-3996152ec465' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/var/lib/vz/images/100/vm-disk-100-0.raw,size=131072' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-disk-100-0.raw","node-name":"e3c77a41648168ee29008dc344126a9","read-only":false},"node-name":"f3c77a41648168ee29008dc344126a9","read-only":false,"size":131072},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -smp '1,sockets=1,cores=1,maxcpus=1' \
   -nodefaults \
   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
@@ -26,4 +27,4 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -machine 'type=pc+pve0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=pc+pve0'
diff --git a/src/test/cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd b/src/test/cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd
index 4e9a7e87..a9dcd474 100644
--- a/src/test/cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd
+++ b/src/test/cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=7b10d7af-b932-4c66-b2c3-3996152ec465' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/var/lib/vz/images/100/vm-disk-100-0.raw,size=540672' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-disk-100-0.raw","node-name":"e3c77a41648168ee29008dc344126a9","read-only":false},"node-name":"f3c77a41648168ee29008dc344126a9","read-only":false,"size":540672},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -smp '1,sockets=1,cores=1,maxcpus=1' \
   -nodefaults \
   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
@@ -27,4 +28,4 @@
   -device 'VGA,id=vga,bus=pcie.0,addr=0x1' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -machine 'type=q35+pve0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=q35+pve0'
diff --git a/src/test/cfg2cmd/efi-secboot-and-tpm.conf.cmd b/src/test/cfg2cmd/efi-secboot-and-tpm.conf.cmd
index 175d9b10..c65c74f5 100644
--- a/src/test/cfg2cmd/efi-secboot-and-tpm.conf.cmd
+++ b/src/test/cfg2cmd/efi-secboot-and-tpm.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=7b10d7af-b932-4c66-b2c3-3996152ec465' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/var/lib/vz/images/100/vm-disk-100-0.raw,size=540672' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE_4M.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-disk-100-0.raw","node-name":"e3c77a41648168ee29008dc344126a9","read-only":false},"node-name":"f3c77a41648168ee29008dc344126a9","read-only":false,"size":540672},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -smp '1,sockets=1,cores=1,maxcpus=1' \
   -nodefaults \
   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
@@ -29,4 +30,4 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -machine 'type=pc+pve0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=pc+pve0'
diff --git a/src/test/cfg2cmd/efidisk-on-rbd.conf.cmd b/src/test/cfg2cmd/efidisk-on-rbd.conf.cmd
index 5c55c01b..585e6ee9 100644
--- a/src/test/cfg2cmd/efidisk-on-rbd.conf.cmd
+++ b/src/test/cfg2cmd/efidisk-on-rbd.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=3dd750ce-d910-44d0-9493-525c0be4e688' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,cache=writeback,format=raw,file=rbd:cpool/vm-100-disk-1:mon_host=127.0.0.42;127.0.0.21;[\:\:1]:auth_supported=none:rbd_cache_policy=writeback,size=131072' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"auth-client-required":["none"],"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"rbd","image":"vm-100-disk-1","node-name":"eeb8f022b5551ad1d795611f112c767","pool":"cpool","read-only":false,"server":[{"host":"127.0.0.42","port":"3300"},{"host":"127.0.0.21","port":"3300"},{"host":"::1","port":"3300"}]},"node-name":"feb8f022b5551ad1d795611f112c767","read-only":false,"size":131072},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -smp '1,sockets=1,cores=1,maxcpus=1' \
   -nodefaults \
   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
@@ -31,4 +32,4 @@
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
-  -machine 'type=pc+pve0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=pc+pve0'
diff --git a/src/test/cfg2cmd/ide.conf.cmd b/src/test/cfg2cmd/ide.conf.cmd
index a0d6c3ed..78fe7550 100644
--- a/src/test/cfg2cmd/ide.conf.cmd
+++ b/src/test/cfg2cmd/ide.conf.cmd
@@ -15,6 +15,11 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 512 \
+  -object '{"id":"throttle-drive-ide0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-ide1","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-ide2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-ide3","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -25,16 +30,16 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'file=/var/lib/vz/template/iso/zero.iso,if=none,id=drive-ide0,media=cdrom,format=raw,aio=io_uring' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"driver":"file","filename":"/var/lib/vz/template/iso/zero.iso","node-name":"e19e15bf93b8cf09e2a5d1669648165","read-only":true},"node-name":"f19e15bf93b8cf09e2a5d1669648165","read-only":true},"node-name":"drive-ide0","throttle-group":"throttle-drive-ide0"}' \
   -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200' \
-  -drive 'file=/mnt/pve/cifs-store/template/iso/one.iso,if=none,id=drive-ide1,media=cdrom,format=raw,aio=threads' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"driver":"file","filename":"/mnt/pve/cifs-store/template/iso/one.iso","node-name":"e247a6535f864815c8e9dbb5a118336","read-only":true},"node-name":"f247a6535f864815c8e9dbb5a118336","read-only":true},"node-name":"drive-ide1","throttle-group":"throttle-drive-ide1"}' \
   -device 'ide-cd,bus=ide.0,unit=1,drive=drive-ide1,id=ide1,bootindex=201' \
-  -drive 'file=/mnt/pve/cifs-store/template/iso/two.iso,if=none,id=drive-ide2,media=cdrom,format=raw,aio=threads' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"driver":"file","filename":"/mnt/pve/cifs-store/template/iso/two.iso","node-name":"ec78a64f692c2fa9f873a111580aebd","read-only":true},"node-name":"fc78a64f692c2fa9f873a111580aebd","read-only":true},"node-name":"drive-ide2","throttle-group":"throttle-drive-ide2"}' \
   -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=202' \
-  -drive 'file=/mnt/pve/cifs-store/template/iso/three.iso,if=none,id=drive-ide3,media=cdrom,format=raw,aio=threads' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"driver":"file","filename":"/mnt/pve/cifs-store/template/iso/three.iso","node-name":"e35557bae4bcbf9edc9f7ff7f132f30","read-only":true},"node-name":"f35557bae4bcbf9edc9f7ff7f132f30","read-only":true},"node-name":"drive-ide3","throttle-group":"throttle-drive-ide3"}' \
   -device 'ide-cd,bus=ide.1,unit=1,drive=drive-ide3,id=ide3,bootindex=203' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/100/vm-100-disk-2.qcow2,if=none,id=drive-scsi0,format=qcow2,cache=none,aio=io_uring,detect-zeroes=on' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-2.qcow2","node-name":"ec11e0572184321efc5835152b95d5d","read-only":false},"node-name":"fc11e0572184321efc5835152b95d5d","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
diff --git a/src/test/cfg2cmd/q35-ide.conf.cmd b/src/test/cfg2cmd/q35-ide.conf.cmd
index f12fa44d..f94accb9 100644
--- a/src/test/cfg2cmd/q35-ide.conf.cmd
+++ b/src/test/cfg2cmd/q35-ide.conf.cmd
@@ -16,6 +16,11 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 512 \
+  -object '{"id":"throttle-drive-ide0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-ide1","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-ide2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-ide3","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
   -global 'ICH9-LPC.disable_s3=1' \
   -global 'ICH9-LPC.disable_s4=1' \
   -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \
@@ -24,16 +29,16 @@
   -device 'VGA,id=vga,bus=pcie.0,addr=0x1' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'file=/mnt/pve/cifs-store/template/iso/zero.iso,if=none,id=drive-ide0,media=cdrom,format=raw,aio=threads' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"driver":"file","filename":"/mnt/pve/cifs-store/template/iso/zero.iso","node-name":"e1677eafc00b7016099210662868e38","read-only":true},"node-name":"f1677eafc00b7016099210662868e38","read-only":true},"node-name":"drive-ide0","throttle-group":"throttle-drive-ide0"}' \
   -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200' \
-  -drive 'file=/mnt/pve/cifs-store/template/iso/one.iso,if=none,id=drive-ide1,media=cdrom,format=raw,aio=threads' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"driver":"file","filename":"/mnt/pve/cifs-store/template/iso/one.iso","node-name":"e247a6535f864815c8e9dbb5a118336","read-only":true},"node-name":"f247a6535f864815c8e9dbb5a118336","read-only":true},"node-name":"drive-ide1","throttle-group":"throttle-drive-ide1"}' \
   -device 'ide-cd,bus=ide.2,unit=0,drive=drive-ide1,id=ide1,bootindex=201' \
-  -drive 'file=/mnt/pve/cifs-store/template/iso/two.iso,if=none,id=drive-ide2,media=cdrom,format=raw,aio=threads' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"driver":"file","filename":"/mnt/pve/cifs-store/template/iso/two.iso","node-name":"ec78a64f692c2fa9f873a111580aebd","read-only":true},"node-name":"fc78a64f692c2fa9f873a111580aebd","read-only":true},"node-name":"drive-ide2","throttle-group":"throttle-drive-ide2"}' \
   -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=202' \
-  -drive 'file=/mnt/pve/cifs-store/template/iso/three.iso,if=none,id=drive-ide3,media=cdrom,format=raw,aio=threads' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"driver":"file","filename":"/mnt/pve/cifs-store/template/iso/three.iso","node-name":"e35557bae4bcbf9edc9f7ff7f132f30","read-only":true},"node-name":"f35557bae4bcbf9edc9f7ff7f132f30","read-only":true},"node-name":"drive-ide3","throttle-group":"throttle-drive-ide3"}' \
   -device 'ide-cd,bus=ide.3,unit=0,drive=drive-ide3,id=ide3,bootindex=203' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/100/vm-100-disk-2.qcow2,if=none,id=drive-scsi0,format=qcow2,cache=none,aio=io_uring,detect-zeroes=on' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-2.qcow2","node-name":"ec11e0572184321efc5835152b95d5d","read-only":false},"node-name":"fc11e0572184321efc5835152b95d5d","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
diff --git a/src/test/cfg2cmd/q35-linux-hostpci-mapping.conf.cmd b/src/test/cfg2cmd/q35-linux-hostpci-mapping.conf.cmd
index 717c0be4..42f1cb80 100644
--- a/src/test/cfg2cmd/q35-linux-hostpci-mapping.conf.cmd
+++ b/src/test/cfg2cmd/q35-linux-hostpci-mapping.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=3dd750ce-d910-44d0-9493-525c0be4e687' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=qcow2,file=/var/lib/vz/images/100/vm-100-disk-1.qcow2' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-1.qcow2","node-name":"e70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"f70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -global 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' \
   -smp '2,sockets=2,cores=1,maxcpus=2' \
   -nodefaults \
@@ -35,4 +36,4 @@
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
-  -machine 'type=q35+pve0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=q35+pve0'
diff --git a/src/test/cfg2cmd/q35-linux-hostpci-multifunction.conf.cmd b/src/test/cfg2cmd/q35-linux-hostpci-multifunction.conf.cmd
index 146bf3e5..e9cd47b8 100644
--- a/src/test/cfg2cmd/q35-linux-hostpci-multifunction.conf.cmd
+++ b/src/test/cfg2cmd/q35-linux-hostpci-multifunction.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=3dd750ce-d910-44d0-9493-525c0be4e687' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=qcow2,file=/var/lib/vz/images/100/vm-100-disk-1.qcow2' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-1.qcow2","node-name":"e70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"f70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -global 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' \
   -smp '2,sockets=2,cores=1,maxcpus=2' \
   -nodefaults \
@@ -35,4 +36,4 @@
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
-  -machine 'type=q35+pve0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=q35+pve0'
diff --git a/src/test/cfg2cmd/q35-linux-hostpci-template.conf.cmd b/src/test/cfg2cmd/q35-linux-hostpci-template.conf.cmd
index ce69f23a..ddc87814 100644
--- a/src/test/cfg2cmd/q35-linux-hostpci-template.conf.cmd
+++ b/src/test/cfg2cmd/q35-linux-hostpci-template.conf.cmd
@@ -8,8 +8,9 @@
   -mon 'chardev=qmp-event,mode=control' \
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=qcow2,file=/var/lib/vz/images/100/base-100-disk-1.qcow2,readonly=on' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/base-100-disk-1.qcow2","node-name":"eb6bec0e3c391fabb7fb7dd73ced9bf","read-only":true},"node-name":"fb6bec0e3c391fabb7fb7dd73ced9bf","read-only":true},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -smp '1,sockets=1,cores=1,maxcpus=1' \
   -nodefaults \
   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
@@ -17,6 +18,7 @@
   -nographic \
   -cpu qemu64 \
   -m 512 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -26,7 +28,7 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/100/base-100-disk-2.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on,readonly=on' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/base-100-disk-2.raw","node-name":"e24dfe239201bb9924fc4cfb899ca70","read-only":true},"node-name":"f24dfe239201bb9924fc4cfb899ca70","read-only":true},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,write-cache=on' \
-  -machine 'accel=tcg,type=pc+pve0' \
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,accel=tcg,type=pc+pve0' \
   -snapshot
diff --git a/src/test/cfg2cmd/q35-linux-hostpci-x-pci-overrides.conf.cmd b/src/test/cfg2cmd/q35-linux-hostpci-x-pci-overrides.conf.cmd
index 0f0cb2c0..b06dbb4f 100644
--- a/src/test/cfg2cmd/q35-linux-hostpci-x-pci-overrides.conf.cmd
+++ b/src/test/cfg2cmd/q35-linux-hostpci-x-pci-overrides.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=3dd750ce-d910-44d0-9493-525c0be4e687' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=qcow2,file=/var/lib/vz/images/100/vm-100-disk-1.qcow2' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-1.qcow2","node-name":"e70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"f70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -global 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' \
   -smp '2,sockets=2,cores=1,maxcpus=2' \
   -nodefaults \
@@ -34,4 +35,4 @@
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
-  -machine 'type=q35+pve0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=q35+pve0'
diff --git a/src/test/cfg2cmd/q35-linux-hostpci.conf.cmd b/src/test/cfg2cmd/q35-linux-hostpci.conf.cmd
index 0abb569b..014eb09c 100644
--- a/src/test/cfg2cmd/q35-linux-hostpci.conf.cmd
+++ b/src/test/cfg2cmd/q35-linux-hostpci.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=3dd750ce-d910-44d0-9493-525c0be4e687' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=qcow2,file=/var/lib/vz/images/100/vm-100-disk-1.qcow2' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-1.qcow2","node-name":"e70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"f70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -global 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' \
   -smp '2,sockets=2,cores=1,maxcpus=2' \
   -nodefaults \
@@ -40,4 +41,4 @@
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
-  -machine 'type=q35+pve0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=q35+pve0'
diff --git a/src/test/cfg2cmd/q35-simple.conf.cmd b/src/test/cfg2cmd/q35-simple.conf.cmd
index 371ea7dd..c6b38f7d 100644
--- a/src/test/cfg2cmd/q35-simple.conf.cmd
+++ b/src/test/cfg2cmd/q35-simple.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=3dd750ce-d910-44d0-9493-525c0be4e687' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=qcow2,file=/var/lib/vz/images/100/vm-100-disk-1.qcow2' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-1.qcow2","node-name":"e70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"f70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -global 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' \
   -smp '2,sockets=1,cores=2,maxcpus=2' \
   -nodefaults \
@@ -28,4 +29,4 @@
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
-  -machine 'type=q35+pve0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=q35+pve0'
diff --git a/src/test/cfg2cmd/seabios_serial.conf.cmd b/src/test/cfg2cmd/seabios_serial.conf.cmd
index 0eb02459..fc421d16 100644
--- a/src/test/cfg2cmd/seabios_serial.conf.cmd
+++ b/src/test/cfg2cmd/seabios_serial.conf.cmd
@@ -15,6 +15,7 @@
   -nographic \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -25,10 +26,9 @@
   -device 'isa-serial,chardev=serial0' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"ecd04be4259153b8293415fefa2a84c","read-only":false},"node-name":"fcd04be4259153b8293415fefa2a84c","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
diff --git a/src/test/cfg2cmd/sev-es.conf.cmd b/src/test/cfg2cmd/sev-es.conf.cmd
index 3a100306..a39b6f67 100644
--- a/src/test/cfg2cmd/sev-es.conf.cmd
+++ b/src/test/cfg2cmd/sev-es.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=7b10d7af-b932-4c66-b2c3-3996152ec465' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CVM_CODE_4M.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/var/lib/vz/images/100/vm-disk-100-0.raw,size=540672' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CVM_CODE_4M.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-disk-100-0.raw","node-name":"e3c77a41648168ee29008dc344126a9","read-only":false},"node-name":"f3c77a41648168ee29008dc344126a9","read-only":false,"size":540672},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -smp '1,sockets=1,cores=1,maxcpus=1' \
   -nodefaults \
   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
@@ -27,4 +28,4 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -object 'sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=6,policy=0xc' \
-  -machine 'type=pc+pve0,confidential-guest-support=sev0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=pc+pve0,confidential-guest-support=sev0'
diff --git a/src/test/cfg2cmd/sev-std.conf.cmd b/src/test/cfg2cmd/sev-std.conf.cmd
index 06da2ca0..3878f15c 100644
--- a/src/test/cfg2cmd/sev-std.conf.cmd
+++ b/src/test/cfg2cmd/sev-std.conf.cmd
@@ -9,8 +9,9 @@
   -pidfile /var/run/qemu-server/8006.pid \
   -daemonize \
   -smbios 'type=1,uuid=7b10d7af-b932-4c66-b2c3-3996152ec465' \
-  -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CVM_CODE_4M.fd' \
-  -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/var/lib/vz/images/100/vm-disk-100-0.raw,size=540672' \
+  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
+  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CVM_CODE_4M.fd"},"node-name":"pflash0","read-only":true}' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-disk-100-0.raw","node-name":"e3c77a41648168ee29008dc344126a9","read-only":false},"node-name":"f3c77a41648168ee29008dc344126a9","read-only":false,"size":540672},"node-name":"drive-efidisk0","throttle-group":"throttle-drive-efidisk0"}' \
   -smp '1,sockets=1,cores=1,maxcpus=1' \
   -nodefaults \
   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
@@ -27,4 +28,4 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -object 'sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=6,policy=0x8' \
-  -machine 'type=pc+pve0,confidential-guest-support=sev0'
+  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,type=pc+pve0,confidential-guest-support=sev0'
diff --git a/src/test/cfg2cmd/simple-btrfs.conf.cmd b/src/test/cfg2cmd/simple-btrfs.conf.cmd
index 2aa2083d..810f1cc9 100644
--- a/src/test/cfg2cmd/simple-btrfs.conf.cmd
+++ b/src/test/cfg2cmd/simple-btrfs.conf.cmd
@@ -15,6 +15,10 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi1","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi3","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -25,16 +29,15 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/butter/bread/images/8006/vm-8006-disk-0/disk.raw,if=none,id=drive-scsi0,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/butter/bread/images/8006/vm-8006-disk-0/disk.raw","node-name":"e99aff0ff797aa030a22e9f580076dd","read-only":false},"node-name":"f99aff0ff797aa030a22e9f580076dd","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
-  -drive 'file=/butter/bread/images/8006/vm-8006-disk-0/disk.raw,if=none,id=drive-scsi1,cache=writeback,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/butter/bread/images/8006/vm-8006-disk-0/disk.raw","node-name":"e7b2fd2a8c5dbfc550d9781e5df8841","read-only":false},"node-name":"f7b2fd2a8c5dbfc550d9781e5df8841","read-only":false},"node-name":"drive-scsi1","throttle-group":"throttle-drive-scsi1"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,write-cache=on' \
-  -drive 'file=/butter/bread/images/8006/vm-8006-disk-0/disk.raw,if=none,id=drive-scsi2,cache=writethrough,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/butter/bread/images/8006/vm-8006-disk-0/disk.raw","node-name":"ed78b07bb04c2cbd8aedc648e885569","read-only":false},"node-name":"fd78b07bb04c2cbd8aedc648e885569","read-only":false},"node-name":"drive-scsi2","throttle-group":"throttle-drive-scsi2"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,write-cache=off' \
-  -drive 'file=/butter/bread/images/8006/vm-8006-disk-0/disk.raw,if=none,id=drive-scsi3,cache=directsync,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/butter/bread/images/8006/vm-8006-disk-0/disk.raw","node-name":"e7487c01d831e2b51a5446980170ec9","read-only":false},"node-name":"f7487c01d831e2b51a5446980170ec9","read-only":false},"node-name":"drive-scsi3","throttle-group":"throttle-drive-scsi3"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3,write-cache=off' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
diff --git a/src/test/cfg2cmd/simple-cifs.conf.cmd b/src/test/cfg2cmd/simple-cifs.conf.cmd
index d23a046a..8723bfff 100644
--- a/src/test/cfg2cmd/simple-cifs.conf.cmd
+++ b/src/test/cfg2cmd/simple-cifs.conf.cmd
@@ -14,6 +14,10 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 512 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi1","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi3","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -23,15 +27,14 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/mnt/pve/cifs-store/images/8006/vm-8006-disk-0.raw,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"native","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/mnt/pve/cifs-store/images/8006/vm-8006-disk-0.raw","node-name":"e2b3b8f2d6a23adc1aa3ecd195dbaf5","read-only":false},"node-name":"f2b3b8f2d6a23adc1aa3ecd195dbaf5","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,write-cache=on' \
-  -drive 'file=/mnt/pve/cifs-store/images/8006/vm-8006-disk-0.raw,if=none,id=drive-scsi1,cache=writeback,discard=on,format=raw,aio=threads,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/mnt/pve/cifs-store/images/8006/vm-8006-disk-0.raw","node-name":"ee4d9a961200a669c1a8182632aba3e","read-only":false},"node-name":"fe4d9a961200a669c1a8182632aba3e","read-only":false},"node-name":"drive-scsi1","throttle-group":"throttle-drive-scsi1"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,write-cache=on' \
-  -drive 'file=/mnt/pve/cifs-store/images/8006/vm-8006-disk-0.raw,if=none,id=drive-scsi2,cache=writethrough,discard=on,format=raw,aio=threads,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/mnt/pve/cifs-store/images/8006/vm-8006-disk-0.raw","node-name":"e6a3bf7eee1e2636cbe31f62b537b6c","read-only":false},"node-name":"f6a3bf7eee1e2636cbe31f62b537b6c","read-only":false},"node-name":"drive-scsi2","throttle-group":"throttle-drive-scsi2"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,write-cache=off' \
-  -drive 'file=/mnt/pve/cifs-store/images/8006/vm-8006-disk-0.raw,if=none,id=drive-scsi3,cache=directsync,discard=on,format=raw,aio=native,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"native","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/mnt/pve/cifs-store/images/8006/vm-8006-disk-0.raw","node-name":"e7042ee58e764b1296ad54014cb9a03","read-only":false},"node-name":"f7042ee58e764b1296ad54014cb9a03","read-only":false},"node-name":"drive-scsi3","throttle-group":"throttle-drive-scsi3"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3,write-cache=off' \
   -machine 'type=pc+pve0'
diff --git a/src/test/cfg2cmd/simple-disk-passthrough.conf.cmd b/src/test/cfg2cmd/simple-disk-passthrough.conf.cmd
index 70ee9f6b..58368210 100644
--- a/src/test/cfg2cmd/simple-disk-passthrough.conf.cmd
+++ b/src/test/cfg2cmd/simple-disk-passthrough.conf.cmd
@@ -15,6 +15,9 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
+  -object '{"id":"throttle-drive-ide2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi1","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -25,12 +28,12 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'file=/dev/cdrom,if=none,id=drive-ide2,media=cdrom,format=raw,aio=io_uring' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"driver":"host_cdrom","filename":"/dev/cdrom","node-name":"ee50e59431a6228dc388fc821b35696","read-only":true},"node-name":"fe50e59431a6228dc388fc821b35696","read-only":true},"node-name":"drive-ide2","throttle-group":"throttle-drive-ide2"}' \
   -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/dev/sda,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/sda","node-name":"eec235c1b362ebd19d5e98959b4c171","read-only":false},"node-name":"fec235c1b362ebd19d5e98959b4c171","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
-  -drive 'file=/mnt/file.raw,if=none,id=drive-scsi1,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/mnt/file.raw","node-name":"e234a4e3b89ac3adac9bdbf0c3dd6b4","read-only":false},"node-name":"f234a4e3b89ac3adac9bdbf0c3dd6b4","read-only":false},"node-name":"drive-scsi1","throttle-group":"throttle-drive-scsi1"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
diff --git a/src/test/cfg2cmd/simple-lvm.conf.cmd b/src/test/cfg2cmd/simple-lvm.conf.cmd
index 40a6c7c8..650f0ac3 100644
--- a/src/test/cfg2cmd/simple-lvm.conf.cmd
+++ b/src/test/cfg2cmd/simple-lvm.conf.cmd
@@ -14,6 +14,10 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 512 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi1","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi3","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -24,12 +28,12 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/dev/veegee/vm-8006-disk-0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"native","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/veegee/vm-8006-disk-0","node-name":"e0378a375d635b0f473569544c7c207","read-only":false},"node-name":"f0378a375d635b0f473569544c7c207","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
-  -drive 'file=/dev/veegee/vm-8006-disk-0,if=none,id=drive-scsi1,cache=writeback,discard=on,format=raw,aio=threads,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/veegee/vm-8006-disk-0","node-name":"e2fbae024c8a771f708f4a5391211b0","read-only":false},"node-name":"f2fbae024c8a771f708f4a5391211b0","read-only":false},"node-name":"drive-scsi1","throttle-group":"throttle-drive-scsi1"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,write-cache=on' \
-  -drive 'file=/dev/veegee/vm-8006-disk-0,if=none,id=drive-scsi2,cache=writethrough,discard=on,format=raw,aio=threads,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/veegee/vm-8006-disk-0","node-name":"e4328c26b141e3efe1564cb60bf1155","read-only":false},"node-name":"f4328c26b141e3efe1564cb60bf1155","read-only":false},"node-name":"drive-scsi2","throttle-group":"throttle-drive-scsi2"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,write-cache=off' \
-  -drive 'file=/dev/veegee/vm-8006-disk-0,if=none,id=drive-scsi3,cache=directsync,discard=on,format=raw,aio=native,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"native","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/veegee/vm-8006-disk-0","node-name":"e68e10f8128f05fe5f7e85cc1f9922b","read-only":false},"node-name":"f68e10f8128f05fe5f7e85cc1f9922b","read-only":false},"node-name":"drive-scsi3","throttle-group":"throttle-drive-scsi3"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3,write-cache=off' \
   -machine 'type=pc+pve0'
diff --git a/src/test/cfg2cmd/simple-lvmthin.conf.cmd b/src/test/cfg2cmd/simple-lvmthin.conf.cmd
index 8d366aff..22251bc6 100644
--- a/src/test/cfg2cmd/simple-lvmthin.conf.cmd
+++ b/src/test/cfg2cmd/simple-lvmthin.conf.cmd
@@ -14,6 +14,10 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 512 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi1","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi3","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -24,12 +28,12 @@
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/dev/pve/vm-8006-disk-0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/pve/vm-8006-disk-0","node-name":"e6d87b01b7bb888b8426534a542ff1c","read-only":false},"node-name":"f6d87b01b7bb888b8426534a542ff1c","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
-  -drive 'file=/dev/pve/vm-8006-disk-0,if=none,id=drive-scsi1,cache=writeback,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/pve/vm-8006-disk-0","node-name":"e96d9ece81aa4271aa2d8485184f66b","read-only":false},"node-name":"f96d9ece81aa4271aa2d8485184f66b","read-only":false},"node-name":"drive-scsi1","throttle-group":"throttle-drive-scsi1"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,write-cache=on' \
-  -drive 'file=/dev/pve/vm-8006-disk-0,if=none,id=drive-scsi2,cache=writethrough,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/pve/vm-8006-disk-0","node-name":"e0b89788ef97beda10a850ab45897d9","read-only":false},"node-name":"f0b89788ef97beda10a850ab45897d9","read-only":false},"node-name":"drive-scsi2","throttle-group":"throttle-drive-scsi2"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,write-cache=off' \
-  -drive 'file=/dev/pve/vm-8006-disk-0,if=none,id=drive-scsi3,cache=directsync,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/pve/vm-8006-disk-0","node-name":"ea7b6871af66ca3e13e95bd74570aa2","read-only":false},"node-name":"fa7b6871af66ca3e13e95bd74570aa2","read-only":false},"node-name":"drive-scsi3","throttle-group":"throttle-drive-scsi3"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3,write-cache=off' \
   -machine 'type=pc+pve0'
diff --git a/src/test/cfg2cmd/simple-rbd.conf.cmd b/src/test/cfg2cmd/simple-rbd.conf.cmd
index df7cba3f..ee86afea 100644
--- a/src/test/cfg2cmd/simple-rbd.conf.cmd
+++ b/src/test/cfg2cmd/simple-rbd.conf.cmd
@@ -15,6 +15,14 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi1","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi3","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi4","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi5","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi6","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi7","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -25,24 +33,23 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=rbd:cpool/vm-8006-disk-0:mon_host=127.0.0.42;127.0.0.21;[\:\:1]:auth_supported=none,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"auth-client-required":["none"],"cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"rbd","image":"vm-8006-disk-0","node-name":"e8e1af6f55c6a2466f178045aa79710","pool":"cpool","read-only":false,"server":[{"host":"127.0.0.42","port":"3300"},{"host":"127.0.0.21","port":"3300"},{"host":"::1","port":"3300"}]},"node-name":"f8e1af6f55c6a2466f178045aa79710","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
-  -drive 'file=rbd:cpool/vm-8006-disk-0:mon_host=127.0.0.42;127.0.0.21;[\:\:1]:auth_supported=none,if=none,id=drive-scsi1,cache=writeback,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"auth-client-required":["none"],"cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"rbd","image":"vm-8006-disk-0","node-name":"e3990bba2ed1f48c5bb23e9f37b4cec","pool":"cpool","read-only":false,"server":[{"host":"127.0.0.42","port":"3300"},{"host":"127.0.0.21","port":"3300"},{"host":"::1","port":"3300"}]},"node-name":"f3990bba2ed1f48c5bb23e9f37b4cec","read-only":false},"node-name":"drive-scsi1","throttle-group":"throttle-drive-scsi1"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,write-cache=on' \
-  -drive 'file=rbd:cpool/vm-8006-disk-0:mon_host=127.0.0.42;127.0.0.21;[\:\:1]:auth_supported=none,if=none,id=drive-scsi2,cache=writethrough,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"auth-client-required":["none"],"cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"rbd","image":"vm-8006-disk-0","node-name":"e3beccc2a8f2eacb8b5df8055a7d093","pool":"cpool","read-only":false,"server":[{"host":"127.0.0.42","port":"3300"},{"host":"127.0.0.21","port":"3300"},{"host":"::1","port":"3300"}]},"node-name":"f3beccc2a8f2eacb8b5df8055a7d093","read-only":false},"node-name":"drive-scsi2","throttle-group":"throttle-drive-scsi2"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,write-cache=off' \
-  -drive 'file=rbd:cpool/vm-8006-disk-0:mon_host=127.0.0.42;127.0.0.21;[\:\:1]:auth_supported=none,if=none,id=drive-scsi3,cache=directsync,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"auth-client-required":["none"],"cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"rbd","image":"vm-8006-disk-0","node-name":"eef923d5dfcee93fbc712b03f9f21af","pool":"cpool","read-only":false,"server":[{"host":"127.0.0.42","port":"3300"},{"host":"127.0.0.21","port":"3300"},{"host":"::1","port":"3300"}]},"node-name":"fef923d5dfcee93fbc712b03f9f21af","read-only":false},"node-name":"drive-scsi3","throttle-group":"throttle-drive-scsi3"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3,write-cache=off' \
-  -drive 'file=/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-0,if=none,id=drive-scsi4,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-0","node-name":"eb2c7a292f03b9f6d015cf83ae79730","read-only":false},"node-name":"fb2c7a292f03b9f6d015cf83ae79730","read-only":false},"node-name":"drive-scsi4","throttle-group":"throttle-drive-scsi4"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=4,drive=drive-scsi4,id=scsi4,write-cache=on' \
-  -drive 'file=/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-0,if=none,id=drive-scsi5,cache=writeback,discard=on,format=raw,aio=threads,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-0","node-name":"e5258ec75558b1f102af1e20e677fd0","read-only":false},"node-name":"f5258ec75558b1f102af1e20e677fd0","read-only":false},"node-name":"drive-scsi5","throttle-group":"throttle-drive-scsi5"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=5,drive=drive-scsi5,id=scsi5,write-cache=on' \
-  -drive 'file=/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-0,if=none,id=drive-scsi6,cache=writethrough,discard=on,format=raw,aio=threads,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"threads","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-0","node-name":"edb33cdcea8ec3e2225509c4945227e","read-only":false},"node-name":"fdb33cdcea8ec3e2225509c4945227e","read-only":false},"node-name":"drive-scsi6","throttle-group":"throttle-drive-scsi6"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=6,drive=drive-scsi6,id=scsi6,write-cache=off' \
-  -drive 'file=/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-0,if=none,id=drive-scsi7,cache=directsync,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/rbd-pve/fc4181a6-56eb-4f68-b452-8ba1f381ca2a/cpool/vm-8006-disk-0","node-name":"eb0b017124a47505c97a5da052e0141","read-only":false},"node-name":"fb0b017124a47505c97a5da052e0141","read-only":false},"node-name":"drive-scsi7","throttle-group":"throttle-drive-scsi7"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=7,drive=drive-scsi7,id=scsi7,write-cache=off' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
diff --git a/src/test/cfg2cmd/simple-virtio-blk.conf.cmd b/src/test/cfg2cmd/simple-virtio-blk.conf.cmd
index 0a7eb473..886f187b 100644
--- a/src/test/cfg2cmd/simple-virtio-blk.conf.cmd
+++ b/src/test/cfg2cmd/simple-virtio-blk.conf.cmd
@@ -16,6 +16,7 @@
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
   -object 'iothread,id=iothread-virtio0' \
+  -object '{"id":"throttle-drive-virtio0","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -26,9 +27,8 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-virtio0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"edd19f6c1b3a6d5a6248c3376a91a16","read-only":false},"node-name":"fdd19f6c1b3a6d5a6248c3376a91a16","read-only":false},"node-name":"drive-virtio0","throttle-group":"throttle-drive-virtio0"}' \
   -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,bootindex=100,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
diff --git a/src/test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd b/src/test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd
index a90156b0..19302053 100644
--- a/src/test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd
+++ b/src/test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd
@@ -15,6 +15,10 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi1","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi2","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-scsi3","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -25,16 +29,15 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=iscsi://127.0.0.1/iqn.2019-10.org.test:foobar/0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"iscsi","lun":0,"node-name":"e7106ac43d4f125a1911487dd9e3e42","portal":"127.0.0.1","read-only":false,"target":"iqn.2019-10.org.test:foobar","transport":"tcp"},"node-name":"f7106ac43d4f125a1911487dd9e3e42","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
-  -drive 'file=iscsi://127.0.0.1/iqn.2019-10.org.test:foobar/0,if=none,id=drive-scsi1,cache=writeback,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"iscsi","lun":0,"node-name":"efdb73e0d0acc5a60e3ff438cb20113","portal":"127.0.0.1","read-only":false,"target":"iqn.2019-10.org.test:foobar","transport":"tcp"},"node-name":"ffdb73e0d0acc5a60e3ff438cb20113","read-only":false},"node-name":"drive-scsi1","throttle-group":"throttle-drive-scsi1"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,write-cache=on' \
-  -drive 'file=iscsi://127.0.0.1/iqn.2019-10.org.test:foobar/0,if=none,id=drive-scsi2,cache=writethrough,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"iscsi","lun":0,"node-name":"eab527a81b458aa9603dca5e2505f6e","portal":"127.0.0.1","read-only":false,"target":"iqn.2019-10.org.test:foobar","transport":"tcp"},"node-name":"fab527a81b458aa9603dca5e2505f6e","read-only":false},"node-name":"drive-scsi2","throttle-group":"throttle-drive-scsi2"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,write-cache=off' \
-  -drive 'file=iscsi://127.0.0.1/iqn.2019-10.org.test:foobar/0,if=none,id=drive-scsi3,cache=directsync,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"raw","file":{"cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"iscsi","lun":0,"node-name":"e915a332310039f7a3feed6901eb5da","portal":"127.0.0.1","read-only":false,"target":"iqn.2019-10.org.test:foobar","transport":"tcp"},"node-name":"f915a332310039f7a3feed6901eb5da","read-only":false},"node-name":"drive-scsi3","throttle-group":"throttle-drive-scsi3"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3,write-cache=off' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
diff --git a/src/test/cfg2cmd/simple1-template.conf.cmd b/src/test/cfg2cmd/simple1-template.conf.cmd
index c736c84a..d63bd68a 100644
--- a/src/test/cfg2cmd/simple1-template.conf.cmd
+++ b/src/test/cfg2cmd/simple1-template.conf.cmd
@@ -15,6 +15,8 @@
   -nographic \
   -cpu qemu64 \
   -m 512 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
+  -object '{"id":"throttle-drive-sata0","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -23,13 +25,12 @@
   -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/8006/base-8006-disk-1.qcow2,if=none,id=drive-scsi0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap,readonly=on' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/base-8006-disk-1.qcow2","node-name":"e1085774206ae4a6b6bf8426ff08f16","read-only":true},"node-name":"f1085774206ae4a6b6bf8426ff08f16","read-only":true},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,write-cache=on' \
   -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' \
-  -drive 'file=/var/lib/vz/images/8006/base-8006-disk-0.qcow2,if=none,id=drive-sata0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/base-8006-disk-0.qcow2","node-name":"eab334c2e07734480f33dd80d89871b","read-only":false},"node-name":"fab334c2e07734480f33dd80d89871b","read-only":false},"node-name":"drive-sata0","throttle-group":"throttle-drive-sata0"}' \
   -device 'ide-hd,bus=ahci0.0,drive=drive-sata0,id=sata0,write-cache=on' \
   -machine 'accel=tcg,smm=off,type=pc+pve0' \
   -snapshot
diff --git a/src/test/cfg2cmd/simple1.conf.cmd b/src/test/cfg2cmd/simple1.conf.cmd
index e657aed7..513e41fc 100644
--- a/src/test/cfg2cmd/simple1.conf.cmd
+++ b/src/test/cfg2cmd/simple1.conf.cmd
@@ -15,6 +15,7 @@
   -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
   -m 768 \
+  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
   -global 'PIIX4_PM.disable_s3=1' \
   -global 'PIIX4_PM.disable_s4=1' \
   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
@@ -25,10 +26,9 @@
   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' \
   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
-  -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
   -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=200' \
   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
-  -drive 'file=/var/lib/vz/images/8006/vm-8006-disk-0.qcow2,if=none,id=drive-scsi0,discard=on,format=qcow2,cache=none,aio=io_uring,detect-zeroes=unmap' \
+  -blockdev '{"driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"file","filename":"/var/lib/vz/images/8006/vm-8006-disk-0.qcow2","node-name":"ecd04be4259153b8293415fefa2a84c","read-only":false},"node-name":"fcd04be4259153b8293415fefa2a84c","read-only":false},"node-name":"drive-scsi0","throttle-group":"throttle-drive-scsi0"}' \
   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100,write-cache=on' \
   -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
   -device 'virtio-net-pci,mac=A2:C0:43:77:08:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300' \
diff --git a/src/test/run_config2command_tests.pl b/src/test/run_config2command_tests.pl
index 52fedd7b..1262a0df 100755
--- a/src/test/run_config2command_tests.pl
+++ b/src/test/run_config2command_tests.pl
@@ -266,6 +266,18 @@ $storage_module->mock(
     },
 );
 
+my $file_stat_module = Test::MockModule->new("File::stat");
+$file_stat_module->mock(
+    stat => sub {
+        my ($path) = @_;
+        if ($path =~ m!/dev/!) {
+            return $file_stat_module->original('stat')->('/dev/null');
+        } else {
+            return $file_stat_module->original('stat')->('./run_config2command_tests.pl');
+        }
+    },
+);
+
 my $zfsplugin_module = Test::MockModule->new("PVE::Storage::ZFSPlugin");
 $zfsplugin_module->mock(
     zfs_get_lu_name => sub {
@@ -276,6 +288,13 @@ $zfsplugin_module->mock(
     },
 );
 
+my $rbdplugin_module = Test::MockModule->new("PVE::Storage::RBDPlugin");
+$rbdplugin_module->mock(
+    rbd_volume_config_set => sub {
+        return;
+    },
+);
+
 my $qemu_server_config;
 $qemu_server_config = Test::MockModule->new('PVE::QemuConfig');
 $qemu_server_config->mock(
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 49/51] test: migration: update running machine to 10.0

[Fiona Ebner]

In particular, this also means that (mocked) blockdev_mirror() will be
used.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/test/run_qemu_migrate_tests.pl | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/src/test/run_qemu_migrate_tests.pl b/src/test/run_qemu_migrate_tests.pl
index 68f0784e..ed2f38ee 100755
--- a/src/test/run_qemu_migrate_tests.pl
+++ b/src/test/run_qemu_migrate_tests.pl
@@ -267,7 +267,7 @@ my $vm_configs = {
                 'numa' => 0,
                 'ostype' => 'l26',
                 'runningcpu' => 'kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep',
-                'runningmachine' => 'pc-i440fx-5.0+pve0',
+                'runningmachine' => 'pc-i440fx-10.0+pve0',
                 'scsi0' => 'local-dir:4567/vm-4567-disk-0.qcow2,size=4G',
                 'scsihw' => 'virtio-scsi-pci',
                 'smbios1' => 'uuid=2925fdec-a066-4228-b46b-eef8662f5e74',
@@ -288,7 +288,7 @@ my $vm_configs = {
                 'ostype' => 'l26',
                 'parent' => 'snap1',
                 'runningcpu' => 'kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep',
-                'runningmachine' => 'pc-i440fx-5.0+pve0',
+                'runningmachine' => 'pc-i440fx-10.0+pve0',
                 'scsi0' => 'local-dir:4567/vm-4567-disk-0.qcow2,size=4G',
                 'scsi1' => 'local-zfs:vm-4567-disk-0,size=1G',
                 'scsihw' => 'virtio-scsi-pci',
@@ -769,7 +769,7 @@ my $tests = [
         vmid => 4567,
         vm_status => {
             running => 1,
-            runningmachine => 'pc-i440fx-5.0+pve0',
+            runningmachine => 'pc-i440fx-10.0+pve0',
         },
         opts => {
             online => 1,
@@ -783,7 +783,7 @@ my $tests = [
             vm_config => $vm_configs->{4567},
             vm_status => {
                 running => 1,
-                runningmachine => 'pc-i440fx-5.0+pve0',
+                runningmachine => 'pc-i440fx-10.0+pve0',
             },
         },
     },
@@ -1358,7 +1358,7 @@ my $tests = [
         vmid => 105,
         vm_status => {
             running => 1,
-            runningmachine => 'pc-i440fx-5.0+pve0',
+            runningmachine => 'pc-i440fx-10.0+pve0',
         },
         opts => {
             online => 1,
@@ -1376,7 +1376,7 @@ my $tests = [
             vm_config => $vm_configs->{105},
             vm_status => {
                 running => 1,
-                runningmachine => 'pc-i440fx-5.0+pve0',
+                runningmachine => 'pc-i440fx-10.0+pve0',
             },
         },
     },
@@ -1404,7 +1404,7 @@ my $tests = [
         vmid => 105,
         vm_status => {
             running => 1,
-            runningmachine => 'pc-i440fx-5.0+pve0',
+            runningmachine => 'pc-i440fx-10.0+pve0',
         },
         config_patch => {
             snapshots => undef,
@@ -1427,7 +1427,7 @@ my $tests = [
             }),
             vm_status => {
                 running => 1,
-                runningmachine => 'pc-i440fx-5.0+pve0',
+                runningmachine => 'pc-i440fx-10.0+pve0',
             },
         },
     },
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 50/51] partially fix #3227: ensure that target image for mirror has the same size for EFI disks

[Fiona Ebner]

When the format is raw, the size can be explicitly passed. When the
format is a container format like qcow2, the image should already be
allocated with the correct virtual size.

It is not possible to resize a disk with an explicit 'size' set, so
only set this for EFI disks.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/BlockJob.pm | 18 ++++++++++++++++++
 src/PVE/QemuServer/Blockdev.pm |  2 +-
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/src/PVE/QemuServer/BlockJob.pm b/src/PVE/QemuServer/BlockJob.pm
index fc12f310..9c04600b 100644
--- a/src/PVE/QemuServer/BlockJob.pm
+++ b/src/PVE/QemuServer/BlockJob.pm
@@ -455,6 +455,24 @@ sub blockdev_mirror {
     my $attach_dest_opts = { 'no-throttle' => 1 };
     $attach_dest_opts->{'zero-initialized'} = 1 if $dest->{'zero-initialized'};
 
+    # Source and target need to have the exact same virtual size, see bug #3227.
+    # However, it won't be possible to resize a disk with 'size' explicitly set afterwards, so only
+    # set it for EFI disks.
+    if ($drive_id eq 'efidisk0' && !PVE::QemuServer::Blockdev::is_nbd($dest_drive)) {
+        my ($storeid) = PVE::Storage::parse_volume_id($dest_drive->{file}, 1);
+        if (
+            $storeid
+            && PVE::QemuServer::Drive::checked_volume_format($storecfg, $dest->{volid}) eq 'raw'
+        ) {
+            my $block_info = PVE::QemuServer::Blockdev::get_block_info($vmid);
+            if (my $size = $block_info->{$drive_id}->{inserted}->{image}->{'virtual-size'}) {
+                $attach_dest_opts->{size} = $size;
+            } else {
+                log_warn("unable to determine source block node size - continuing anyway");
+            }
+        }
+    }
+
     # Note that if 'aio' is not explicitly set, i.e. default, it can change if source and target
     # don't both allow or both not allow 'io_uring' as the default.
     my $target_node_name =
diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index ab58e3ea..334f788b 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -19,7 +19,7 @@ use PVE::QemuServer::Monitor qw(mon_cmd);
 my $NBD_TCP_PATH_RE_3 = qr/nbd:(\S+):(\d+):exportname=(\S+)/;
 my $NBD_UNIX_PATH_RE_2 = qr/nbd:unix:(\S+):exportname=(\S+)/;
 
-my sub is_nbd {
+sub is_nbd {
     my ($drive) = @_;
 
     return 1 if $drive->{file} =~ $NBD_TCP_PATH_RE_3;
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v3 51/51] blockdev: pass along machine version to storage layer

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm          |  8 +++++---
 src/PVE/QemuServer/Blockdev.pm | 13 ++++++++-----
 src/PVE/QemuServer/OVMF.pm     |  5 +++--
 3 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index d4ad4a25..50352e82 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -3258,6 +3258,7 @@ sub config_to_command {
         my $hw_info = {
             'amd-sev-type' => get_amd_sev_type($conf),
             arch => $arch,
+            'machine-version' => $machine_version,
             q35 => $q35,
         };
         my ($ovmf_cmd, $ovmf_machine_flags) = PVE::QemuServer::OVMF::print_ovmf_commandline(
@@ -3654,7 +3655,7 @@ sub config_to_command {
                         if drive_is_read_only($conf, $drive);
 
                     my $blockdev = PVE::QemuServer::Blockdev::generate_drive_blockdev(
-                        $storecfg, $drive, $extra_blockdev_options,
+                        $storecfg, $drive, $machine_version, $extra_blockdev_options,
                     );
                     push @$devices, '-blockdev', to_json($blockdev, { canonical => 1 });
                 }
@@ -7164,8 +7165,9 @@ sub live_import_from_files {
         if (min_version($machine_version, 10, 0)) { # for the switch to -blockdev
             my ($interface, $index) = PVE::QemuServer::Drive::parse_drive_interface($dev);
             my $drive = { file => $volid, interface => $interface, index => $index };
-            my $blockdev =
-                PVE::QemuServer::Blockdev::generate_drive_blockdev($storecfg, $drive, {});
+            my $blockdev = PVE::QemuServer::Blockdev::generate_drive_blockdev(
+                $storecfg, $drive, $machine_version, {},
+            );
             $live_restore_backing->{$dev}->{blockdev} = $blockdev;
         } else {
             $live_restore_backing->{$dev}->{blockdev} =
diff --git a/src/PVE/QemuServer/Blockdev.pm b/src/PVE/QemuServer/Blockdev.pm
index 334f788b..5f1fdae3 100644
--- a/src/PVE/QemuServer/Blockdev.pm
+++ b/src/PVE/QemuServer/Blockdev.pm
@@ -238,7 +238,7 @@ my sub generate_blockdev_drive_cache {
 }
 
 my sub generate_file_blockdev {
-    my ($storecfg, $drive, $options) = @_;
+    my ($storecfg, $drive, $machine_version, $options) = @_;
 
     my $blockdev = {};
     my $scfg = undef;
@@ -282,7 +282,8 @@ my sub generate_file_blockdev {
         $storage_opts->{hints}->{'efi-disk'} = 1 if $drive->{interface} eq 'efidisk';
         $storage_opts->{'snapshot-name'} = $options->{'snapshot-name'}
             if defined($options->{'snapshot-name'});
-        $blockdev = PVE::Storage::qemu_blockdev_options($storecfg, $volid, $storage_opts);
+        $blockdev =
+            PVE::Storage::qemu_blockdev_options($storecfg, $volid, $machine_version, $storage_opts);
         $scfg = PVE::Storage::storage_config($storecfg, $storeid);
     }
 
@@ -360,14 +361,14 @@ my sub generate_format_blockdev {
 }
 
 sub generate_drive_blockdev {
-    my ($storecfg, $drive, $options) = @_;
+    my ($storecfg, $drive, $machine_version, $options) = @_;
 
     my $drive_id = PVE::QemuServer::Drive::get_drive_id($drive);
 
     die "generate_drive_blockdev called without volid/path\n" if !$drive->{file};
     die "generate_drive_blockdev called with 'none'\n" if $drive->{file} eq 'none';
 
-    my $child = generate_file_blockdev($storecfg, $drive, $options);
+    my $child = generate_file_blockdev($storecfg, $drive, $machine_version, $options);
     if (!is_nbd($drive)) {
         $child = generate_format_blockdev($storecfg, $drive, $child, $options);
     }
@@ -477,7 +478,9 @@ state image.
 sub attach {
     my ($storecfg, $vmid, $drive, $options) = @_;
 
-    my $blockdev = generate_drive_blockdev($storecfg, $drive, $options);
+    my $machine_version = PVE::QemuServer::Machine::get_current_qemu_machine($vmid);
+
+    my $blockdev = generate_drive_blockdev($storecfg, $drive, $machine_version, $options);
 
     my $throttle_group_id;
     if (parse_top_node_name($blockdev->{'node-name'})) { # device top nodes need a throttle group
diff --git a/src/PVE/QemuServer/OVMF.pm b/src/PVE/QemuServer/OVMF.pm
index a7239614..aa21c40c 100644
--- a/src/PVE/QemuServer/OVMF.pm
+++ b/src/PVE/QemuServer/OVMF.pm
@@ -147,7 +147,8 @@ sub create_efidisk($$$$$$$$) {
 my sub generate_ovmf_blockdev {
     my ($conf, $storecfg, $vmid, $hw_info) = @_;
 
-    my ($amd_sev_type, $arch, $q35) = $hw_info->@{qw(amd-sev-type arch q35)};
+    my ($amd_sev_type, $arch, $machine_version, $q35) =
+        $hw_info->@{qw(amd-sev-type arch machine-version q35)};
 
     my $drive = $conf->{efidisk0} ? parse_drive('efidisk0', $conf->{efidisk0}) : undef;
 
@@ -190,7 +191,7 @@ my sub generate_ovmf_blockdev {
     my $throttle_group = PVE::QemuServer::Blockdev::generate_throttle_group($drive);
 
     my $ovmf_vars_blockdev = PVE::QemuServer::Blockdev::generate_drive_blockdev(
-        $storecfg, $drive, $extra_blockdev_options,
+        $storecfg, $drive, $machine_version, $extra_blockdev_options,
     );
 
     return ($ovmf_code_blockdev, $ovmf_vars_blockdev, $throttle_group);
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH storage v5 14/51] qemu blockdev options: restrict allowed drivers and options

[Fiona Ebner]

On 02.07.25 6:27 PM, Fiona Ebner wrote:
> @@ -733,7 +829,25 @@ sub qemu_blockdev_options {
>      die "cannot use volume of type '$vtype' as a QEMU blockdevice\n"
>          if $vtype ne 'images' && $vtype ne 'iso' && $vtype ne 'import';
>  
> -    return $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $machine_version, $options);
> +    my $blockdev =
> +        $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $machine_version, $options);
> +
> +    if (my $driver = $blockdev->{driver}) {
> +        my $allowed_opts = $allowed_qemu_blockdev_options->{$driver};

My subconscious just told me that I forgot to add a "die" here if the
driver itself is not allowed. Like this, all options will get dropped,
so it can't be abused, i.e. kinda works by accident, but should be fixed
of course.

> +        for my $opt (keys $blockdev->%*) {
> +            next if $opt eq 'driver';
> +            if (!$allowed_opts->{$opt}) {
> +                delete($blockdev->{$opt});
> +                log_warn(
> +                    "volume '$volid' - dropping block device option '$opt' set by storage plugin"
> +                        . " - not currently part of allowed schema");
> +            }
> +        }
> +    } else {
> +        die "storage plugin for '$storeid' did not return a blockdev driver\n";
> +    }
> +
> +    return $blockdev;
>  }
>  
>  # used as last resort to adapt volnames when migrating


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu/storage/qemu-server v3 00/51] let's switch to blockdev, blockdev, blockdev, part four (final)

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 13205 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH qemu/storage/qemu-server v3 00/51] let's switch to blockdev, blockdev, blockdev, part four (final)
Date: Thu, 3 Jul 2025 07:17:35 +0000
Message-ID: <17289472181f8f6f1ef58a9c59cbc6db0d464b11.camel@groupe-cyllene.com>

Hi Fiona,

I have done a lot of test yesterday, I don't have found bug

tested:
- hotplug/unplug
- cdrom swap , cdrom none->iso , iso->none  (no more error)
- cloudinit refresh
- backup to file/pbs , fleecing device
- restore from file/pbs , live restore
- move disk between different storage
- live migration with local storage through nbd (secure && insecure)
- resize (it's ok now)

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu/storage/qemu-server v3 00/51] let's switch to blockdev, blockdev, blockdev, part four (final)

[Fabian Grünbichler]

> DERUMIER, Alexandre via pve-devel <pve-devel@lists.proxmox.com> hat am 03.07.2025 09:17 CEST geschrieben:
> Hi Fiona,
> 
> I have done a lot of test yesterday, I don't have found bug
> 
> tested:
> - hotplug/unplug
> - cdrom swap , cdrom none->iso , iso->none  (no more error)
> - cloudinit refresh
> - backup to file/pbs , fleecing device
> - restore from file/pbs , live restore
> - move disk between different storage
> - live migration with local storage through nbd (secure && insecure)
> - resize (it's ok now)

nice! :)

I'll also do some more testing and review today, unless something
big comes up I will probably apply the series later today (for
master/Trixie/PVE 9).

Do you have capacity for rebasing your external qcow2 patches on
top until early next week? Else I can also do it.. I'd like to
get those into shape for more widespread internal testing over
the next week, if possible.

Fabian


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu/storage/qemu-server v3 00/51] let's switch to blockdev, blockdev, blockdev, part four (final)

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 12648 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>, "f.gruenbichler@proxmox.com" <f.gruenbichler@proxmox.com>
Subject: Re: [pve-devel] [PATCH qemu/storage/qemu-server v3 00/51] let's switch to blockdev, blockdev, blockdev, part four (final)
Date: Thu, 3 Jul 2025 08:03:01 +0000
Message-ID: <f15ab570300d8b0782197145caaf85b355478d7f.camel@groupe-cyllene.com>

>>
>>Do you have capacity for rebasing your external qcow2 patches on
>>top until early next week? Else I can also do it.. I'd like to
>>get those into shape for more widespread internal testing over
>>the next week, if possible.

yes, I'm currently working on it ! (I'm full time on it before my
holiday around the 20th July)


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH storage v5 05/51] plugin: add method to get qemu blockdevice options for volume

[Fabian Grünbichler]

Quoting Fiona Ebner (2025-07-02 18:27:38)
> This is in preparation to switch qemu-server from using '-drive' to
> the modern '-blockdev' in the QEMU commandline options as well as for
> the qemu-storage-daemon, which only supports '-blockdev'. The plugins
> know best what driver and options are needed to access an image, so
> a dedicated plugin method returning the necessary parameters for
> '-blockdev' is the most straight-forward.
> 
> There intentionally is only handling for absolute paths in the default
> plugin implementation. Any plugin requiring more needs to implement
> the method itself. With PVE 9 being a major release and most popular
> plugins not using special protocols like 'rbd://', this seems
> acceptable.
> 
> For NBD, etc. qemu-server should construct the blockdev object.
> 
> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
> ---
>  src/PVE/Storage.pm        | 17 ++++++++++++
>  src/PVE/Storage/Plugin.pm | 56 +++++++++++++++++++++++++++++++++++++++
>  2 files changed, 73 insertions(+)
> 
> diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
> index 69eb435..ec8b753 100755
> --- a/src/PVE/Storage.pm
> +++ b/src/PVE/Storage.pm
> @@ -719,6 +719,23 @@ sub abs_filesystem_path {
>      return $path;
>  }
>  
> +# see the documentation for the plugin method
> +sub qemu_blockdev_options {
> +    my ($cfg, $volid) = @_;
> +
> +    my ($storeid, $volname) = parse_volume_id($volid);
> +
> +    my $scfg = storage_config($cfg, $storeid);
> +
> +    my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
> +
> +    my ($vtype) = $plugin->parse_volname($volname);
> +    die "cannot use volume of type '$vtype' as a QEMU blockdevice\n"
> +        if $vtype ne 'images' && $vtype ne 'iso' && $vtype ne 'import';
> +
> +    return $plugin->qemu_blockdev_options($scfg, $storeid, $volname);
> +}
> +
>  # used as last resort to adapt volnames when migrating
>  my $volname_for_storage = sub {
>      my ($cfg, $storeid, $name, $vmid, $format) = @_;
> diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
> index 53b9848..680bb6b 100644
> --- a/src/PVE/Storage/Plugin.pm
> +++ b/src/PVE/Storage/Plugin.pm
> @@ -1961,6 +1961,62 @@ sub rename_volume {
>      return "${storeid}:${base}${target_vmid}/${target_volname}";
>  }
>  
> +=pod
> +
> +=head3 qemu_blockdev_options
> +
> +    $blockdev = $plugin->qemu_blockdev_options($scfg, $storeid, $volname)
> +
> +Returns a hash reference with the basic options needed to open the volume via QEMU's C<-blockdev>
> +API. This at least requires a C<< $blockdev->{driver} >> and a reference to the image, e.g.
> +C<< $blockdev->{filename} >> for the C<file> driver. For files, the C<file> driver can be used. For
> +host block devices, the C<host_device> driver can be used. The plugin must not set options like
> +C<cache> or C<aio>. Those are managed by qemu-server and will be overwritten. For other available
> +drivers and the exact specification of the options, see
> +L<https://qemu.readthedocs.io/en/master/interop/qemu-qmp-ref.html#object-QMP-block-core.BlockdevOptions>
> +
> +While Perl does not have explicit types, the result will need to be converted to JSON later and
> +match the QMP specification (see link above), so implicit types are important. In the return value,
> +use C<JSON::true> and C<JSON::false> for booleans, C<"$value"> for strings, and C<int($value)> for
> +integers.
> +
> +The volume is activated before the function is called.
> +
> +Arguments:
> +
> +=over
> +
> +=item C<$scfg>: The hash reference with the storage configuration.
> +
> +=item C<$storeid>: The storage ID.
> +
> +=item C<$volume>: The volume name.
> +
> +=back
> +
> +=cut
> +
> +sub qemu_blockdev_options {
> +    my ($class, $scfg, $storeid, $volname) = @_;
> +
> +    my $blockdev = {};
> +
> +    my ($path) = $class->filesystem_path($scfg, $volname);

sorry for missing this in the earlier revisions - but I think(!) we should call
$class->path() here, as that is the plugin API method?

see https://lore.proxmox.com/pve-devel/1743427728.0lo886zlbq.astroid@yuna.none/

for our plugin hierarchy it should make no practical difference, but an
external plugin might just have overriden `path` (as that is what gets called
via PVE::Storage)..

but if you look at this part of the series, it is quite obvious (all *our*
plugins that require overriding qemu_blockdev_options also provide their own
path, but not their own filesystem_path)

> +
> +    if ($path =~ m|^/|) {
> +        # The 'file' driver only works for regular files. The check below is taken from
> +        # block/file-posix.c:hdev_probe_device() in QEMU. Do not bother with detecting 'host_cdrom'
> +        # devices here, those are not managed by the storage layer.
> +        my $st = File::stat::stat($path) or die "stat for '$path' failed - $!\n";
> +        my $driver = (S_ISCHR($st->mode) || S_ISBLK($st->mode)) ? 'host_device' : 'file';
> +        $blockdev = { driver => $driver, filename => $path };
> +    } else {
> +        die "storage plugin doesn't implement qemu_blockdev_options() method\n";
> +    }
> +
> +    return $blockdev;
> +}
> +
>  # Used by storage plugins for external backup providers. See PVE::BackupProvider::Plugin for the API
>  # the provider needs to implement.
>  #
> -- 
> 2.47.2
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
>


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH storage v5 15/51] plugin: qemu blockdev options: parse protocol paths in default implementation

[Fabian Grünbichler]

Quoting Fiona Ebner (2025-07-02 18:27:48)
> for better backwards compatibility. This also means using path()
> rather than filesystem_path() as the latter does not return protocol
> paths.

I guess I should have read until here before replying to the other patch, sorry
for the noise ;)

> 
> Some protocol paths are not implemented (considered all that are
> listed by grepping for '\.protocol_name' in QEMU):
> - ftp(s)/http(s), which would access web servers via curl. This one
>   could be added if there is enough interest.
> - nvme://XXXX:XX:XX.X/X, which would access a host NVME device.
> - null-{aio,co}, which are mainly useful for debugging.
> - pbs, because path-based access is not used anymore for PBS,
>   live-restore in qemu-server already defines a driver-based device.
> - nfs and ssh, because the QEMU build script used by Proxmox VE does
>   not enable them.
> - blk{debug,verify}, because they are for debugging.
> - the ones used by blkio, i.e. io_uring, nvme-io_uring,
>   virtio-blk-vfio-pci, virtio-blk-vhost-user and
>   virtio-blk-vhost-vdpa, because the QEMU build script used by Proxmox
>   VE does not enable blkio.
> - backup-dump and zeroinit, because they should not be used by the
>   storage layer directly.
> - gluster, because support is dropped in Proxmox VE 9.
> - host_cdrom, because the storage layer should not access host CD-ROM
>   devices.
> - fat, because it hopefully isn't used by any third-party plugin here.
> 
> Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
> ---
> 
> New in v5.
> 
>  src/PVE/Storage/Plugin.pm | 95 ++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 94 insertions(+), 1 deletion(-)
> 
> diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
> index 3f2c638..c2f376b 100644
> --- a/src/PVE/Storage/Plugin.pm
> +++ b/src/PVE/Storage/Plugin.pm
> @@ -1961,6 +1961,34 @@ sub rename_volume {
>      return "${storeid}:${base}${target_vmid}/${target_volname}";
>  }
>  
> +my sub blockdev_options_nbd_tcp {
> +    my ($host, $port, $export) = @_;
> +
> +    die "blockdev_options_nbd_tcp: no host" if !defined($host);
> +
> +    my $blockdev = {};
> +
> +    my $server = { type => 'inet', host => "$host" };
> +    # port is also a string in QAPI, not optional, default for NBD is 10809
> +    $server->{port} = defined($port) ? "$port" : "10809";
> +    $blockdev = { driver => 'nbd', server => $server };
> +    $blockdev->{export} = "$export" if defined($export);
> +
> +    return $blockdev;
> +}
> +
> +my sub blockdev_options_nbd_unix {
> +    my ($socket_path, $export) = @_;
> +
> +    my $blockdev = {};
> +
> +    my $server = { type => 'unix', path => "$socket_path" };
> +    $blockdev = { driver => 'nbd', server => $server };
> +    $blockdev->{export} = "$export" if defined($export);
> +
> +    return $blockdev;
> +}
> +
>  =pod
>  
>  =head3 qemu_blockdev_options
> @@ -2031,7 +2059,7 @@ sub qemu_blockdev_options {
>  
>      my $blockdev = {};
>  
> -    my ($path) = $class->filesystem_path($scfg, $volname, $options->{'snapshot-name'});
> +    my ($path) = $class->path($scfg, $volname, $storeid, $options->{'snapshot-name'});
>  
>      if ($path =~ m|^/|) {
>          # For qcow2 and qed the path of a snapshot will be the same, but it's not possible to attach
> @@ -2046,6 +2074,71 @@ sub qemu_blockdev_options {
>          my $st = File::stat::stat($path) or die "stat for '$path' failed - $!\n";
>          my $driver = (S_ISCHR($st->mode) || S_ISBLK($st->mode)) ? 'host_device' : 'file';
>          $blockdev = { driver => $driver, filename => $path };
> +    } elsif ($path =~ m|^file:(\S+)|) {
> +        $blockdev = { driver => 'file', filename => "$1" };
> +    } elsif ($path =~ m|^host_device:(\S+)|) {
> +        $blockdev = { driver => 'host_device', filename => "$1" };
> +    } elsif ($path =~ m|^iscsi://(\S+)/(\S+)/(\d+)$|) {
> +        $blockdev =
> +            { driver => 'iscsi', portal => "$1", target => "$2", lun => "$3", transport => "tcp" };
> +    } elsif ($path =~ m|^iser://(\S+)/(\S+)/(\d+)$|) {
> +        $blockdev =
> +            { driver => 'iscsi', portal => "$1", target => "$2", lun => "$3", transport => "iser" };
> +    } elsif ($path =~ m|^nbd(?:\+tcp)?://(\S+?)(?::(\d+))?/(\S+)?$|) { # new style NBD TCP URI
> +        $blockdev = blockdev_options_nbd_tcp($1, $2, $3);
> +    } elsif ($path =~ m|^nbd(?:\+tcp)?:(\S+):(\d+)(?::exportname=(\S+))?$|) {
> +        # old style NBD TCP URI
> +        $blockdev = blockdev_options_nbd_tcp($1, $2, $3);
> +    } elsif ($path =~ m|^nbd\+unix:///(\S+)?\?socket=(\S+)$|) { # new style NBD unix URI
> +        $blockdev = blockdev_options_nbd_unix($2, $1); # note the order!
> +    } elsif ($path =~ m|^nbd:unix:(\S+?)(?::exportname=(\S+))?$|) { # old style NBD unix URI
> +        $blockdev = blockdev_options_nbd_unix($1, $2);
> +    } elsif ($path =~ m/^rbd:(\S+)$/) {
> +        my $rbd_options = $1;
> +        $blockdev->{driver} = 'rbd';
> +
> +        #map options to key=value pair (if not key is provided, this is the image)
> +        #options are seprated with : but we need to exclude \: used for ipv6 address
> +        my $options = {
> +            map {
> +                s/\\:/:/g;
> +                /^(.*?)=(.*)/ ? ($1 => $2) : (image => $_)
> +            } $rbd_options =~ /(?:\\:|\[[^\]]*\]|[^:\\])+/g
> +        };
> +
> +        $blockdev->{'auth-client-required'} = [$options->{'auth_supported'}]
> +            if $options->{'auth_supported'};
> +        $blockdev->{'conf'} = $options->{'conf'} if $options->{'conf'};
> +        $blockdev->{'user'} = $options->{'id'} if $options->{'id'};
> +
> +        if ($options->{'mon_host'}) {
> +            my $server = [];
> +            my @mons = split(';', $options->{'mon_host'});
> +            for my $mon (@mons) {
> +                $mon =~ s/[\[\]]//g;
> +                my ($host, $port) = PVE::Tools::parse_host_and_port($mon);
> +                $port = '3300' if !$port;
> +                push @$server, { host => $host, port => $port };
> +            }
> +            $blockdev->{server} = $server;
> +        }
> +
> +        if ($options->{'image'} =~ m|^(\S+)/(\S+)$|) {
> +            $blockdev->{pool} = $1;
> +            $blockdev->{image} = $2;
> +            if ($blockdev->{image} =~ m|^(\S+)/(\S+)$|) {
> +                $blockdev->{namespace} = $1;
> +                $blockdev->{image} = $2;
> +            }
> +        }
> +
> +        delete($options->@{qw(auth_supported conf id mon_host image)});
> +
> +        # Map rest directly. With -drive, it was possible to use arbitrary key-value-pairs. Like
> +        # this, there will be warnings for those that are not allowed via blockdev.
> +        for my $opt (keys $options->%*) {
> +            $blockdev->{$opt} = $options->{$opt};
> +        }
>      } else {
>          die "storage plugin doesn't implement qemu_blockdev_options() method\n";
>      }
> -- 
> 2.47.2
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
>


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied-series: [PATCH qemu/storage/qemu-server v3 00/51] let's switch to blockdev, blockdev, blockdev, part four (final)

[Fabian Grünbichler]

Applied, thanks!

Best regards,
-- 
Fabian Grünbichler <f.gruenbichler@proxmox.com>


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel