Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API

From: Fiona Ebner <f.ebner@proxmox.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Cc: Jonathan Nicklin <jnicklin@blockbridge.com>
Subject: Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API
Date: Mon, 29 Jul 2024 10:15:04 +0200	[thread overview]
Message-ID: <226084e0-f7a8-4d9c-9b0b-f8cdb1871549@proxmox.com> (raw)
In-Reply-To: <mailman.0.1722092937.302.pve-devel@lists.proxmox.com>

Hi,

Am 26.07.24 um 21:47 schrieb Jonathan Nicklin via pve-devel:
> 
> Hi Fiona,
> 
> Would adding support for offloading incremental difference detection
> to the underlying storage be feasible with the API updates? The QEMU
> bitmap strategy works for all storage devices but is far from
> optimal. If backup coordinated a storage snapshot, the underlying
> storage could enumerate the differences (or generate a bitmap).
> 
> This would allow PBS to connect directly to storage and retrieve
> incremental differences, which could remove the PVE hosts from the
> equation. This "storage-direct" approach for backup would improve
> performance, reduce resources, and support incremental backups in all
> cases (i.e., power failues, shutdowns, etc.). It would also eliminate
> the dependency on QEMU bitmaps and the overhead of fleecing.
> 
> Theoretically, this should be possible with any shared storage that
> can enumerate incremental differences between snapshots: Ceph,
> Blockbridge, iSCSi/ZFS?
> 
> Thoughts?
> 

The two big advantages of the current mechanism are:

1. it's completely storage-agnostic, so you can even use it with raw
files on a directory storage. It follows in the same spirit as existing
backup. Prohibiting backup for users when they use certain kinds of
storages for VMs is not nice.
2. it's been battle-tested with PBS and works nicely.

I don't see why your suggestion can't be implemented in principle.
Feature requests for (non-incremental) "storage-snapshot" mode backup
have been around since a while. It was not a priority for development
yet and is totally different from the current "snapshot" backup mode, so
will need to be developed from the ground up.

That said, AFAICS, it's orthogonal to the series here. When an
implementation like you outlined exists, it can just be added as a new
backup mechanism for external providers (and PBS).

See also the related discussion over at:
https://bugzilla.proxmox.com/show_bug.cgi?id=3233#c19

Best Regards,
Fiona

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel