[pve-devel] [RFC qemu/pve-storage] storage plugin method to get qemu blockdevice options for volume

[pve-devel] [RFC qemu/pve-storage] storage plugin method to get qemu blockdevice options for volume

[Fiona Ebner]

When using -drive, storage plugins currently give us a path that QEMU
understands, some using special protocols such as 'iscsi://'. We'd
like to switch to using the more modern -blockdev for PVE 9. The
plan is to have the storage plugins return the very basic information
required to access the image, and qemu-server can then add other
settings like cache, aio, etc. on top. In fact, pretty similar to what
we have now for -drive, just with a structured hash rather than a
string.

This is also a prerequisite for qemu-storage-deamon, that would be
useful for TPM-as-qcow2 exported via NBD or FUSE or external backup
provider restore providing an NBD export for the provider to write to.

This is an early sneak peek to get feedback on the idea itself and to
keep Alexandre in the loop :)

See the individual patches for more comments.


qemu:

Fiona Ebner (1):
  block/rbd: add @keyring-file option to BlockdevOptionsRbd

 block/rbd.c          | 8 ++++++++
 qapi/block-core.json | 4 ++++
 2 files changed, 12 insertions(+)


pve-storage:

Fiona Ebner (3):
  plugin: add method to get qemu blockdevice options for volume
  iscsi direct plugin: implement method to get qemu blockdevice options
  rbd plugin: implement new method to get qemu blockdevice options

 src/PVE/Storage.pm                   | 19 +++++++++
 src/PVE/Storage/ISCSIDirectPlugin.pm | 17 ++++++++
 src/PVE/Storage/Plugin.pm            | 64 ++++++++++++++++++++++++++++
 src/PVE/Storage/RBDPlugin.pm         | 43 +++++++++++++++++++
 4 files changed, 143 insertions(+)

-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [RFC qemu 1/1] block/rbd: add @keyring-file option to BlockdevOptionsRbd

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

I'll also submit this upstream.

 block/rbd.c          | 8 ++++++++
 qapi/block-core.json | 4 ++++
 2 files changed, 12 insertions(+)

diff --git a/block/rbd.c b/block/rbd.c
index 24e820d056..53a9a785f0 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -272,6 +272,14 @@ static int qemu_rbd_set_auth(rados_t cluster, BlockdevOptionsRbd *opts,
         }
     }
 
+    if (opts->keyring_file) {
+        r = rados_conf_set(cluster, "keyring", opts->keyring_file);
+        if (r < 0) {
+            error_setg_errno(errp, -r, "Could not set 'keyring'");
+            return r;
+        }
+    }
+
     if (opts->has_auth_client_required) {
         accu = g_string_new("");
         for (auth = opts->auth_client_required; auth; auth = auth->next) {
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 02c043f0f7..100277b371 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4671,6 +4671,9 @@
 #     authentication.  This maps to Ceph configuration option "key".
 #     (Since 3.0)
 #
+# @keyring-file: File for the keyring used for cephx authentication.
+#     This maps to Ceph configuration option "keyring". (Since 10.1)
+#
 # @server: Monitor host address and port.  This maps to the "mon_host"
 #     Ceph option.
 #
@@ -4686,6 +4689,7 @@
             '*user': 'str',
             '*auth-client-required': ['RbdAuthMode'],
             '*key-secret': 'str',
+            '*keyring-file': 'str',
             '*server': ['InetSocketAddressBase'] } }
 
 ##
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [RFC storage 1/3] plugin: add method to get qemu blockdevice options for volume

[Fiona Ebner]

There intentionally is only handling for absolute paths in the default
plugin implementation. Any plugin requiring more needs to implement
the method itself.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

I discussed this with Fabian off-list. With PVE 9 being a major
release and most popular plugins not using special protocols like
'rbd://', this seems acceptable. He'll also give a heads-up to known
plugin developers with other changes for PVE 9 in time.

For NBD, etc. qemu-server should construct the blockdev object.

Still missing API bump + Changelog

Did not test snapshots yet.

 src/PVE/Storage.pm        | 19 ++++++++++++
 src/PVE/Storage/Plugin.pm | 64 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index d0a696a..5915395 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -710,6 +710,25 @@ sub abs_filesystem_path {
     return $path;
 }
 
+sub qemu_blockdev_options {
+    my ($cfg, $volid, $snapname) = @_;
+
+    my ($storeid, $volname) = parse_volume_id($volid);
+
+    my $scfg = storage_config($cfg, $storeid);
+
+    my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+    my ($vtype) = $plugin->parse_volname($volname);
+    die "cannot use volume of type '$vtype' as a QEMU blockdevice\n"
+	if $vtype ne 'images' && $vtype ne 'iso' && $vtype ne 'import';
+
+    die "QEMU blockdevice - 'snapname' argument is not supported for vtype '$vtype'"
+	if $snapname && $vtype ne 'images';
+
+    return $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $snapname);
+}
+
 # used as last resort to adapt volnames when migrating
 my $volname_for_storage = sub {
     my ($cfg, $storeid, $name, $vmid, $format) = @_;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 4e16420..dc3c6df 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1880,6 +1880,70 @@ sub rename_volume {
     return "${storeid}:${base}${target_vmid}/${target_volname}";
 }
 
+=pod
+
+=head3 qemu_blockdev_options
+
+    $blockdev = $plugin->qemu_blockdev_options($scfg, $storeid, $volname, $snapname)
+
+Returns a hash reference with the basic options needed to open the volume via QEMU's C<-blockdev>
+API. This at least requires a C<< $blockdev->{driver} >> and a reference to the image, e.g.
+C<< $blockdev->{filename} >> for the C<file> driver. For files, the C<file> driver can be used. For
+host block devices, the C<host_device> driver can be used. The plugin must not set options like
+C<cache> or C<aio>. Those are managed by qemu-server and will be overwritten. For other available
+drivers and the exact specification of the options, see
+L<https://qemu.readthedocs.io/en/master/interop/qemu-qmp-ref.html#object-QMP-block-core.BlockdevOptions>
+
+While Perl does not have explicit types, the result will need to be converted to JSON later and
+match the QMP specification (see link above), so implicit types are important. In the return value,
+use C<JSON::true> and C<JSON::false> for booleans, C<"$value"> for strings, and C<int($value)> for
+integers.
+
+Arguments:
+
+=over
+
+=item C<$scfg>
+
+The hash reference with the storage configuration.
+
+=item C<$storeid>
+
+The storage ID.
+
+=item C<$volume>
+
+The volume name.
+
+=item C<$snapname>
+
+(optional) The snapshot name. Set when the associated snapshot should be opened
+rather than the volume itself.
+
+=back
+
+=cut
+sub qemu_blockdev_options {
+    my ($class, $scfg, $storeid, $volname, $snapname) = @_;
+
+    my $blockdev = {};
+
+    my ($path) = $class->path($scfg, $volname, $storeid, $snapname);
+
+    if ($path =~ m|^/|) {
+	# The 'file' driver only works for regular files. The check below is taken from
+	# block/file-posix.c:hdev_probe_device() in QEMU. Do not bother with detecting 'host_cdrom'
+	# devices here, those are not managed by the storage layer.
+	my $st = File::stat::stat($path);
+	my $driver = (S_ISCHR($st->mode) || S_ISBLK($st->mode)) ? 'host_device' : 'file';
+	$blockdev = { driver => $driver, filename => $path };
+    } else {
+	die "storage plugin doesn't implement qemu_blockdev_options() method\n";
+    }
+
+    return $blockdev;
+}
+
 # Used by storage plugins for external backup providers. See PVE::BackupProvider::Plugin for the API
 # the provider needs to implement.
 #
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [RFC storage 2/3] iscsi direct plugin: implement method to get qemu blockdevice options

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

for me, it fails with
VM 102 qmp command 'blockdev-add' failed - LUN is write protected
but adding as a drive fails. Will need to investigate.

 src/PVE/Storage/ISCSIDirectPlugin.pm | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/src/PVE/Storage/ISCSIDirectPlugin.pm b/src/PVE/Storage/ISCSIDirectPlugin.pm
index d54dcd8..a6381c2 100644
--- a/src/PVE/Storage/ISCSIDirectPlugin.pm
+++ b/src/PVE/Storage/ISCSIDirectPlugin.pm
@@ -105,6 +105,23 @@ sub path {
     return ($path, $vmid, $vtype);
 }
 
+sub qemu_blockdev_options {
+    my ($class, $scfg, $storeid, $volname, $snapname) = @_;
+
+    die "volume snapshot is not possible on iscsi device\n"
+	if defined($snapname);
+
+    my $lun = ($class->parse_volname($volname))[1];
+
+    return {
+	driver => 'iscsi',
+	transport => 'tcp',
+	portal => "$scfg->{portal}",
+	target => "$scfg->{target}",
+	lun => int($lun),
+    };
+}
+
 sub create_base {
     my ($class, $storeid, $scfg, $volname) = @_;
 
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [RFC storage 3/3] rbd plugin: implement new method to get qemu blockdevice options

[Fiona Ebner]

Co-developed-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

The mon host parsing is adapted from other places where we do that,
but I did not test it with IPv6 yet.

 src/PVE/Storage/RBDPlugin.pm | 43 ++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index 73bc97e..6ae8216 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -493,6 +493,49 @@ sub path {
     return ($path, $vmid, $vtype);
 }
 
+sub qemu_blockdev_options {
+    my ($class, $scfg, $storeid, $volname, $snapname) = @_;
+
+    my $cmd_option = PVE::CephConfig::ceph_connect_option($scfg, $storeid);
+    my ($name) = ($class->parse_volname($volname))[1];
+    $name .= '@'.$snapname if $snapname;
+
+    if ($scfg->{krbd}) {
+	my $rbd_dev_path = get_rbd_dev_path($scfg, $storeid, $name);
+	return { driver => 'host_device', filename => $rbd_dev_path };
+    }
+
+    my $blockdev = {
+	driver => 'rbd',
+	pool => $scfg->{pool} ? "$scfg->{pool}" : 'rbd',
+	image => "$name",
+    };
+    $blockdev->{namespace} = "$scfg->{namespace}" if defined($scfg->{namespace});
+
+    $blockdev->{conf} = $cmd_option->{ceph_conf} if $cmd_option->{ceph_conf};
+
+    if (my $monhost = $scfg->{'monhost'}) {
+	my $server = [];
+	my @mons = PVE::Tools::split_list($monhost);
+	for my $mon (@mons) {
+	    $mon =~ s/^\[?v\d\://; # remove beginning of vector
+	    $mon =~ s|/\d+\]?||; # remove end of vector
+	    my ($host, $port) = PVE::Tools::parse_host_and_port($mon);
+	    $port = '3300' if !$port;
+	    push @$server, { host => $host, port => $port };
+	}
+	$blockdev->{server} = $server;
+	$blockdev->{'auth-client-required'} = ["$cmd_option->{auth_supported}"];
+    }
+
+    if ($cmd_option->{keyring}) {
+	$blockdev->{user} = "$cmd_option->{userid}";
+	$blockdev->{'keyring-file'} = "$cmd_option->{keyring}";
+    }
+
+    return $blockdev;
+}
+
 sub find_free_diskname {
     my ($class, $storeid, $scfg, $vmid, $fmt, $add_fmt_suffix) = @_;
 
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC qemu/pve-storage] storage plugin method to get qemu blockdevice options for volume

[Fiona Ebner]

And here is a small script I used for testing to see if QMP blockdev-add
can deal with the results:

> [I] root@pve8a1 ~# cat test-storage-blockdev.pl 
> #!/usr/bin/perl
> 
> use strict;
> use warnings;
> 
> use JSON;
> use PVE::QemuServer::Monitor qw(mon_cmd);
> use PVE::Storage;
> 
> my $vmid = shift or die "need to specify VM ID\n";
> my $volid = shift or die "need to specify volume ID\n";
> my $snapname; # TODO
> 
> my $conf = PVE::Storage::config();
> 
> my $blockdev = PVE::Storage::qemu_blockdev_options($conf, $volid, $snapname);
> $blockdev->{'node-name'} = "a" . rand(1);
> 
> print to_json($blockdev, { canonical => 1, pretty => 1 });
> 
> mon_cmd($vmid, 'blockdev-add', $blockdev->%*);



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC qemu 1/1] block/rbd: add @keyring-file option to BlockdevOptionsRbd

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 15511 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>, "f.ebner@proxmox.com" <f.ebner@proxmox.com>
Subject: Re: [RFC qemu 1/1] block/rbd: add @keyring-file option to BlockdevOptionsRbd
Date: Mon, 12 May 2025 10:57:11 +0000
Message-ID: <dfc78aa17b9c1c8496fa74cb6e6d2517337b65c0.camel@groupe-cyllene.com>

for blockdev, do we still use a ceph config file in /var/run for
potential others rbd client options ?



-------- Message initial --------
De: Fiona Ebner <f.ebner@proxmox.com>
À: pve-devel@lists.proxmox.com
Cc: alexandre.derumier@groupe-cyllene.com
Objet: [RFC qemu 1/1] block/rbd: add @keyring-file option to
BlockdevOptionsRbd
Date: 09/05/2025 16:15:29

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

I'll also submit this upstream.

 block/rbd.c          | 8 ++++++++
 qapi/block-core.json | 4 ++++
 2 files changed, 12 insertions(+)

diff --git a/block/rbd.c b/block/rbd.c
index 24e820d056..53a9a785f0 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -272,6 +272,14 @@ static int qemu_rbd_set_auth(rados_t cluster,
BlockdevOptionsRbd *opts,
         }
     }
 
+    if (opts->keyring_file) {
+        r = rados_conf_set(cluster, "keyring", opts->keyring_file);
+        if (r < 0) {
+            error_setg_errno(errp, -r, "Could not set 'keyring'");
+            return r;
+        }
+    }
+
     if (opts->has_auth_client_required) {
         accu = g_string_new("");
         for (auth = opts->auth_client_required; auth; auth = auth-
>next) {
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 02c043f0f7..100277b371 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4671,6 +4671,9 @@
 #     authentication.  This maps to Ceph configuration option "key".
 #     (Since 3.0)
 #
+# @keyring-file: File for the keyring used for cephx authentication.
+#     This maps to Ceph configuration option "keyring". (Since 10.1)
+#
 # @server: Monitor host address and port.  This maps to the "mon_host"
 #     Ceph option.
 #
@@ -4686,6 +4689,7 @@
             '*user': 'str',
             '*auth-client-required': ['RbdAuthMode'],
             '*key-secret': 'str',
+            '*keyring-file': 'str',
             '*server': ['InetSocketAddressBase'] } }
 
 ##


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC qemu 1/1] block/rbd: add @keyring-file option to BlockdevOptionsRbd

[Fiona Ebner]

Am 12.05.25 um 12:57 schrieb DERUMIER, Alexandre:
> for blockdev, do we still use a ceph config file in /var/run for
> potential others rbd client options ?

Not currently, but we can add that later if we consider it worth it. We
would need to merge with the storage's already existing ceph.conf and
not only write the new options. For now, users can adapt their storage's
ceph.conf as desired.

Best Regards,
Fiona


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel