From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id E60C864907 for ; Fri, 28 Jan 2022 12:23:13 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id DBFB32E013 for ; Fri, 28 Jan 2022 12:22:43 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id DC9C82E00A for ; Fri, 28 Jan 2022 12:22:42 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id B85D34180A for ; Fri, 28 Jan 2022 12:22:42 +0100 (CET) From: Aaron Lauterer To: pve-devel@lists.proxmox.com Date: Fri, 28 Jan 2022 12:22:41 +0100 Message-Id: <20220128112241.3435277-1-a.lauterer@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.004 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_SHORT 0.001 Use of a URL Shortener for very short URL SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [ceph.io, rbdplugin.pm] Subject: [pve-devel] [PATCH v2 storage] fix #1816: rbd: add support for erasure coded ec pools X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2022 11:23:13 -0000 The first step is to allocate rbd images correctly. The metadata objects still need to be stored in a replicated pool, but by providing the --data-pool parameter on image creation, we can place the data objects on the erasure coded (EC) pool. Signed-off-by: Aaron Lauterer --- changes: add data-pool parameter in clone_image() if present Right now this only this only affects disk image creation and cloning. The EC pool needs to be created manually to test this. The Ceph blog about EC with RBD + CephFS gives a nice introduction and the necessary steps to set up such a pool [0]. The steps needed are: - create EC profile (a 21 profile is only useful for testing purposes in a 3 node cluster, not something that should be considered for production use!) # ceph osd erasure-code-profile set ec-21-profile k=2 m=1 crush-failure-domain=host - create a new pool with that profile # ceph osd pool create ec21pool erasure ec-21-profile - allow overwrite # ceph osd pool set ec21pool allow_ec_overwrites true - enable application rbd on the pool (the command in the blog seems to have gotten the order of parameters a bit wrong here) # ceph osd pool application enable ec21pool rbd - add storage configuration # pvesm add rbd ectest --pool --data-pool ec21pool For the replicated pool, either create a new one without adding the PVE storage config or use a namespace to separate it from the existing pool. To create a namespace: # rbd namespace create / add the '--namespace' parameter in the pvesm add command. To check if the objects are stored correclty you can run rados: # rados -p ls This should only show metadata objects # rados -p ls This should then show only `rbd_data.xxx` objects. If you configured a namespace, you also need to add the `--namespace` parameter to the rados command. [0] https://ceph.io/en/news/blog/2017/new-luminous-erasure-coding-rbd-cephfs/ PVE/Storage/RBDPlugin.pm | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/PVE/Storage/RBDPlugin.pm b/PVE/Storage/RBDPlugin.pm index 2607d25..efb2187 100644 --- a/PVE/Storage/RBDPlugin.pm +++ b/PVE/Storage/RBDPlugin.pm @@ -289,6 +289,10 @@ sub properties { description => "Pool.", type => 'string', }, + 'data-pool' => { + description => "Data Pool (for erasure coding only)", + type => 'string', + }, namespace => { description => "RBD Namespace.", type => 'string', @@ -318,6 +322,7 @@ sub options { disable => { optional => 1 }, monhost => { optional => 1}, pool => { optional => 1 }, + 'data-pool' => { optional => 1 }, namespace => { optional => 1 }, username => { optional => 1 }, content => { optional => 1 }, @@ -492,15 +497,10 @@ sub clone_image { my $newvol = "$basename/$name"; $newvol = $name if length($snapname); - my $cmd = $rbd_cmd->( - $scfg, - $storeid, - 'clone', - get_rbd_path($scfg, $basename), - '--snap', - $snap, - get_rbd_path($scfg, $name), - ); + my @options = ('clone', get_rbd_path($scfg, $basename), '--snap', $snap); + push @options, ('--data-pool', $scfg->{'data-pool'}) if $scfg->{'data-pool'}; + push @options, get_rbd_path($scfg, $name); + my $cmd = $rbd_cmd->($scfg, $storeid, @options); run_rbd_command($cmd, errmsg => "rbd clone '$basename' error"); @@ -516,7 +516,10 @@ sub alloc_image { $name = $class->find_free_diskname($storeid, $scfg, $vmid) if !$name; - my $cmd = $rbd_cmd->($scfg, $storeid, 'create', '--image-format' , 2, '--size', int(($size+1023)/1024), $name); + my @options = ('create', '--image-format' , 2, '--size', int(($size+1023)/1024)); + push @options, ('--data-pool', $scfg->{'data-pool'}) if $scfg->{'data-pool'}; + push @options, $name; + my $cmd = $rbd_cmd->($scfg, $storeid, @options); run_rbd_command($cmd, errmsg => "rbd create '$name' error"); return $name; -- 2.30.2