From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pve-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 035F31FF16B for <inbox@lore.proxmox.com>; Thu, 3 Apr 2025 09:25:26 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id DA6B138E8C; Thu, 3 Apr 2025 09:25:14 +0200 (CEST) Date: Thu, 3 Apr 2025 09:24:41 +0200 From: Wolfgang Bumiller <w.bumiller@proxmox.com> To: Andreas Rogge <andreas.rogge@bareos.com> Message-ID: <ey5nze36xsp7iaipishhlsvxo5xyllyvze2sevxutbcttofyye@bxzkiqxcm2no> References: <20241114150754.374376-1-f.ebner@proxmox.com> <20241114150754.374376-10-f.ebner@proxmox.com> <66969dc5-7587-46fc-8b36-8ecf0d5d744a@bareos.com> <866f1cdd-780c-4012-b4fa-709ae1165891@proxmox.com> <dktagxdpeqlvvmqhv2hr5n3dxmdh7nfyctluo64mwqgsimread@jbjcwstbk5fn> <c79787cf-4cc7-4ac6-9e8f-f864f77c1812@bareos.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <c79787cf-4cc7-4ac6-9e8f-f864f77c1812@bareos.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.080 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [PATCH storage v4 09/27] plugin: introduce new_backup_provider() method X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> Cc: Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Thomas Lamprecht <t.lamprecht@proxmox.com>, Philipp Storz <philipp.storz@bareos.com> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com> On Wed, Apr 02, 2025 at 06:16:57PM +0200, Andreas Rogge wrote: > Am 02.04.25 um 10:30 schrieb Wolfgang Bumiller: > > On Tue, Apr 01, 2025 at 08:21:30PM +0200, Thomas Lamprecht wrote: > > > > This sounds pretty inefficient - especially when > > > > comparing with qmrestore's ability to just read read from stdin. > > > > The reading from stdin is quite limited, does not support sparse files > > efficiently, and does not support our live-restore feature. (PS: I may have had a different case in mind when thinking of sparse files - the restore from stdin uses the VMA format after all) > > > > If we can *pull* data out-of-order from the backup provider via a better > > protocol (like NBD which Thomas mentioned), holes in the disks don't > > need to be transferred over the network, and we could probably support > > live-restore, where the VMs immediately start running *during* the > > restore process. (qemu would simply treat read-requests from the guest > > which have not yet been restored with a higher priority while otherwise > > continuing to copy the data in-order in the background) > Neither pulling nor out-of-order is an option in the Bareos architecture. Fair enough. While this is not currently done, I'd still like to know: what about *backing up* out of order? PVE's original "VMA" format was specifically designed to allow what is essentially a "copy-before-write" style backup while a VM is running. The current provider-API uses a fleecing-approach to support running VMS while the provider plugin can choose to perform backups in its desired order. But I do wonder if - to reduce space-requirements for backing up running VMs - at some point we might also add the ability for qemu to provide some kind of queue containing the offset-length pairs of blocks which have been stored in the temporary fleecing image. The provider could consume this queue to keep the temporary storage at a minimum by doing out-of-order backups. This only makes sense if there are backup systems which can benefit from it. (And it should be a simple-enough api extension to add this in the future, from what I can tell) > > > Some alternatives btw. are providing a fuse or ublk device on the PVE > > side which pulls data from bareos (or to which bareos can push data > > which, like qemu's "fleecing" mode, could store the not-yet restored > > portions in temporary files, discarding them as they are read by/moved > > to qemu). > That's basically like staging the image just that you can start reading > before we finish writing. I'll keep it in mind, even tough I don't think it > is really feasible. > > > *Technically* we could have a mode where we allocate the disks and "map" > > them onto the system (potentially via nbd, or `rbd map` for ceph etc.) > Yes, please do. I guess it makes sense for us to not expect/require random access, as any feature like that already imposes limitations on how the data can be stored. I'd expect different backup solutions to have different limitations in that regard. The API design in this series (or at least the current version of it) wouldn't prevent pure sequential restores, just not make them "easy" right now, but that's something we should be able to accommodate with another mechanism. I *believe* `qemu-nbd` should be able to bind all the storage types we want to restore to to /dev/nbdXY devices, which would give the provider a bunch of block devices to write to in whichever way they want to, so the provider would then only need to figure out how to receive the data and forward that to the devices. We'll need to try. > > > > *But* it would make live-restore impossible with that plugin. > There will be no live-restore with Bareos. > If we would ever consider implementing this, on the Bareos side it would be > pretty complicated and full of limitations. In that case we would probably > just implement yet another plugin for PVE. Fair enough. > > > Which is why the most flexible thing to do is to use a `qemu-img` call > > and giving it the paths, or more precisely, the URLs to the disks. > > I understand how this makes sense. However, if you don't have the data in a > format that qemu-img can consume, things become complicated. It can also just read data from a stream (but without any smarter protocol, this would make holes/unallocated blocks extremely inefficient), where the provider would create that stream in whichever way they want to. (Although I'm not actually sure the current invocation can deal with a stream - we do have patches on qemu-img's `dd` subcommand to specifically allow this, though...) _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel