From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id EFD4569C9D for ; Thu, 25 Feb 2021 09:45:58 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E436C31859 for ; Thu, 25 Feb 2021 09:45:58 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id B42CD3184A for ; Thu, 25 Feb 2021 09:45:57 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 849DB4631B; Thu, 25 Feb 2021 09:45:57 +0100 (CET) Message-ID: <0d734e31-fd08-237c-4cf3-0604030fed1f@proxmox.com> Date: Thu, 25 Feb 2021 09:45:56 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:86.0) Gecko/20100101 Thunderbird/86.0 Content-Language: en-US To: Mark Adams , PVE User List References: From: Thomas Lamprecht In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.056 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.001 Looks like a legit reply (A) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] KRBD vs librbd for qemu kvm vms X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2021 08:45:59 -0000 Hi, On 24.02.21 20:25, Mark Adams wrote: > Hi All, > > I have done some research on this, but I can't find any resources that > answer my specific questions - so I am hoping that everyone on the list can > help out. > > In one specific use case, I have a pve 5.4-11 cluster using ceph 12.2.12. > The main reason I haven't upgraded this is because of the very manual > process to upgrade ceph and this cluster has quite a few osds. Also it will > be replaced in the not too distant future. > > On this version, should I be using KRBD or librbd for my qemu vms? I have > read a number of posts/articles that say it performs better as KRBD, but > then also see some which say it should only be for containers or bare metal. > General Tradeoff: * KRBD: a bit faster, some operations (e.g., snapshots) may see more improvements than others * librbd: matches ceph version in use, so normally its supports all current ceph RBD features, currently I'm actually not to sure if there are any not supported by the kernel, IIRC the 5.4 kernel from PVE 6.x works out fine with all features. NOTE: you talk about PVE 5.4, as it was replaced by 6.x over 1.5 years ago and is EOL since over 6 months I really do not have all version specific pitfalls in mind, so please have that in mind when reading this, I tried hard to remember any issues but at that time span my brain gets untrustworthy to be sure, so I'd highly recommend to test any change before you employ it in production. > Additionally if I were to switch (Tick the KRBD box for the specific RBD > storage) what is the correct procedure to apply this? Simply select it, > then reboot each node? It will the be used for the next VM start or disk hotplug, no reboot necessary. > > This is an HA cluster, so, are KRBD and librbd interchangeable between > hosts in the cluster? can I live migrate the VMs on to another node after I > enable KRBD then simply live migrate them back after I reboot before I then > do the same for the next node in the cluster? You should be able to do so, reboot is actually not required. Enable flag and to a merry round of migrations to get it started up with KRBD. But, this is so easy to test that it would be unwise to not do so bevorehand, setup a test VM, if you don't have on lying around already, and test with that one. > > Lastly, is the performance of KRBD really that much better? Is there any > downsides to using it vs librbd? I guess I still don't understand why there > are 2 options if KRBD performs better? Why don't you compare yourself? Start a modern version of any distro shipping fio and compare a boot with KRBD on and off. Performance is not everything ;-) User-space crashes won't bring down your whole host, for example. And as said the user-space library can adopt new features better, which can include performance relevant features but also such having other advantages. cheers, Thomas