From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 53256639BE for ; Wed, 26 Jan 2022 17:07:37 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 495B51A274 for ; Wed, 26 Jan 2022 17:07:37 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 54C091A26A for ; Wed, 26 Jan 2022 17:07:36 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 1E3A342487 for ; Wed, 26 Jan 2022 17:07:36 +0100 (CET) From: Aaron Lauterer To: pve-devel@lists.proxmox.com Date: Wed, 26 Jan 2022 17:07:34 +0100 Message-Id: <20220126160734.2868618-1-a.lauterer@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.001 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_SHORT 0.001 Use of a URL Shortener for very short URL SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [rbdplugin.pm, ceph.io] Subject: [pve-devel] [PATCH storage] rbd: add support for erasure coded ec pools X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jan 2022 16:07:37 -0000 The first step is to allocate rbd images correctly. The metadata objects still need to be stored in a replicated pool, but by providing the --data-pool parameter on image creation, we can place the data objects on the erasure coded (EC) pool. Signed-off-by: Aaron Lauterer --- Right now this only this only affects disk image creation and the EC pool needs to be created manually to test this. The Ceph blog about EC with RBD + CephFS gives a nice introduction and the necessary steps to set up such a pool [0]. The steps needed are: - create EC profile (a 21 profile is only useful for testing purposes in a 3 node cluster, not something that should be considered for production use!) # ceph osd erasure-code-profile set ec-21-profile k=2 m=1 crush-failure-domain=host - create a new pool with that profile # ceph osd pool create ec21pool erasure ec-21-profile - allow overwrite # ceph osd pool set ec21pool allow_ec_overwrites true - enable application rbd on the pool (the command in the blog seems to have gotten the order of parameters a bit wrong here) # ceph osd pool application enable ec21pool rbd - add storage configuration # pvesm add rbd ectest --pool --data-pool ec21pool For the replicated pool, either create a new one without adding the PVE storage config or use a namespace to separate it from the existing pool. To create a namespace: # rbd namespace create / add the '--namespace' parameter in the pvesm add command. To check if the objects are stored correclty you can run rados: # rados -p ls This should only show metadata objects # rados -p ls This should then show only `rbd_data.xxx` objects. If you configured a namespace, you also need to add the `--namespace` parameter to the rados command. [0] https://ceph.io/en/news/blog/2017/new-luminous-erasure-coding-rbd-cephfs/ PVE/Storage/RBDPlugin.pm | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/PVE/Storage/RBDPlugin.pm b/PVE/Storage/RBDPlugin.pm index 2607d25..1ea3418 100644 --- a/PVE/Storage/RBDPlugin.pm +++ b/PVE/Storage/RBDPlugin.pm @@ -289,6 +289,10 @@ sub properties { description => "Pool.", type => 'string', }, + 'data-pool' => { + description => "Data Pool (for erasure coding only)", + type => 'string', + }, namespace => { description => "RBD Namespace.", type => 'string', @@ -318,6 +322,7 @@ sub options { disable => { optional => 1 }, monhost => { optional => 1}, pool => { optional => 1 }, + 'data-pool' => { optional => 1 }, namespace => { optional => 1 }, username => { optional => 1 }, content => { optional => 1 }, @@ -516,7 +521,10 @@ sub alloc_image { $name = $class->find_free_diskname($storeid, $scfg, $vmid) if !$name; - my $cmd = $rbd_cmd->($scfg, $storeid, 'create', '--image-format' , 2, '--size', int(($size+1023)/1024), $name); + my @options = ('create', '--image-format' , 2, '--size', int(($size+1023)/1024)); + push @options, ('--data-pool', $scfg->{'data-pool'}) if $scfg->{'data-pool'}; + push @options, $name; + my $cmd = $rbd_cmd->($scfg, $storeid, @options); run_rbd_command($cmd, errmsg => "rbd create '$name' error"); return $name; -- 2.30.2