From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 4DAFF1FF176 for ; Mon, 29 Jul 2024 10:15:14 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0D8AB1898C; Mon, 29 Jul 2024 10:15:13 +0200 (CEST) Message-ID: <226084e0-f7a8-4d9c-9b0b-f8cdb1871549@proxmox.com> Date: Mon, 29 Jul 2024 10:15:04 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Proxmox VE development discussion References: Content-Language: en-US From: Fiona Ebner In-Reply-To: X-SPAM-LEVEL: Spam detection results: 0 AWL -0.060 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Cc: Jonathan Nicklin Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" Hi, Am 26.07.24 um 21:47 schrieb Jonathan Nicklin via pve-devel: > > Hi Fiona, > > Would adding support for offloading incremental difference detection > to the underlying storage be feasible with the API updates? The QEMU > bitmap strategy works for all storage devices but is far from > optimal. If backup coordinated a storage snapshot, the underlying > storage could enumerate the differences (or generate a bitmap). > > This would allow PBS to connect directly to storage and retrieve > incremental differences, which could remove the PVE hosts from the > equation. This "storage-direct" approach for backup would improve > performance, reduce resources, and support incremental backups in all > cases (i.e., power failues, shutdowns, etc.). It would also eliminate > the dependency on QEMU bitmaps and the overhead of fleecing. > > Theoretically, this should be possible with any shared storage that > can enumerate incremental differences between snapshots: Ceph, > Blockbridge, iSCSi/ZFS? > > Thoughts? > The two big advantages of the current mechanism are: 1. it's completely storage-agnostic, so you can even use it with raw files on a directory storage. It follows in the same spirit as existing backup. Prohibiting backup for users when they use certain kinds of storages for VMs is not nice. 2. it's been battle-tested with PBS and works nicely. I don't see why your suggestion can't be implemented in principle. Feature requests for (non-incremental) "storage-snapshot" mode backup have been around since a while. It was not a priority for development yet and is totally different from the current "snapshot" backup mode, so will need to be developed from the ground up. That said, AFAICS, it's orthogonal to the series here. When an implementation like you outlined exists, it can just be added as a new backup mechanism for external providers (and PBS). See also the related discussion over at: https://bugzilla.proxmox.com/show_bug.cgi?id=3233#c19 Best Regards, Fiona _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel