Re: [pve-devel] [PATCH storage v4 09/27] plugin: introduce new_backup_provider() method

[Wolfgang Bumiller <w.bumiller@proxmox.com>]

On Tue, Apr 01, 2025 at 08:21:30PM +0200, Thomas Lamprecht wrote:
> Hi!
> 
> Am 01.04.25 um 18:02 schrieb Andreas Rogge:
> > sorry for digging up that ancient mail, but I feel that's the best 
> > starting point for me.
> 
> For more current discussion it might be best to check out the recently
> posted v7 of this series, if nothing bigger comes up it should be very
> close to what gets applied for an initial version – i.e., one that will
> be supported for a long time, which does not necessarily mean that it
> cannot evolve.
> 
> https://lore.proxmox.com/pve-devel/20250401173435.221892-1-f.ebner@proxmox.com/T/#t
> 
> > Bareos investigates how to better integrate with Proxmox VE, so users 
> > can take backups of virtual machines and restore them using Bareos.
> 
> Great to hear!
> 
> > Am 14.11.24 um 16:07 schrieb Fiona Ebner:
> >> The new_backup_provider() method can be used by storage plugins for
> >> external backup providers. If the method returns a provider, Proxmox
> >> VE will use callbacks to that provider for backups and restore instead
> >> of using its usual backup/restore mechanisms.
> > So we as a backup provider need to provide plugin code that will run as 
> > part of PVE. If I understand correctly this must be implemented as a 
> > Perl module. Also, PVE wants to trigger the backup itself.
> 
> The interface is in Perl, the actual backup code can be whatever runs
> on the Proxmox VE base system, as that is based on Debian this means
> that there is a lot of flexibility. A colleague wrote a plugin in rust
> as proof of concept, we probably release that too as example.
> 
> We can also decouple this more once the dust settles, there are some
> ideas floating around already, but they can be implemented later so
> ensuring that the actual base functionality works out is more important,
> as whatever the interface is, they will work similar but just using a
> different transport (e.g. say varlink over an FD instead of perl wrapper
> module).
> 
> > This introduces a few hurdles for us:
> > First of all, we don't usually program in Perl, so that's going to be a 
> > problem for us. We can probably implement something, but we probably 
> > don't have any developer that can write Perl code that meets any kind of 
> > professional standard.
> 
> The Perl code required for a production quality interface is not that
> much, and certainly does not require in-depth perl experience. Perl
> is a mature language, for better or worse, that allowed and still
> allows all kind of users to create tools that can work fine if one keeps
> them simple, and as the core design is rather fixed here, it should not
> be that much work, and we certainly can help on questions or problems;
> just reach out on the developer list and point to your public plugin
> code.
> 
> > To do its job[1], Bareos must be able to schedule backup and restore 
> > jobs all by itself. If I understood correctly, backups utilizing one of 
> > the plugins can be run by calling `vzdump`, so that this is at least 
> > possible.
> 
> Exactly, we want a native PVE implementation so that such plug-ins are
> actually nicely integrated and feel almost native (in the long run,
> missing a few bits for that), so users get a first-class experience
> on "both sides" of the backup and can also see what's going on at the
> source side of things, if they want.
> 
> > However, what we've seen other vendors do is that they provide an API 
> > where you just tell the system what VM you want to back up and then you 
> > are provided with access to metadata (i.e. VM configuration, disk 
> > layouts, nvram content, etc) and disk contents.
> 
> You can send an API request that basically results in exactly that, just
> that the plugin is providing you the info then. A big advantage is that
> the user can also trigger backups from the PVE side and thus get a much
> better integration.
> 
> > For restore it is mostly the same: you provide the metadata, get a set 
> > of raw disks that you can write your data to and finally you just tell 
> > the system that the VM restore is done.
> 
> It's different but ensures that we can provide backup providers and
> their user a quasi native integration into PVE, ideally as close as
> possible to how PBS is integrated, this can IMO be also benefit for
> providers, and nothing that is possible with the methods you describe
> should become impossible with the methods we provide.
> 
> 
> >> 2. Restore API
> >>
> >> 2.1. Restore Mechanisms
> >>
> >> VM mechanism 'qemu-img':
> >>
> >> The backup provider gives a path to the disk image that will be
> >> restored. The path needs to be something 'qemu-img' can deal with,
> >> e.g. can also be an NBD URI or similar.
> > Um... that has nothing to do with what you provided when we take the 
> > backup. Is there a reason PVE cannot provide a writeable block device to 
> > restore to?
> > For Bareos this requirement would imply that we need an unpleasantly 
> > large staging area on the PVE node to facilitate a restore: As we can 
> > only stream the data we'd have to create a temporary disk image that PVE 
> > can then read.

qemu-img directly supports various protocols, and a "path" in this
case is not necessarily a *file*, I'll go into more detail below.

> > This sounds pretty inefficient - especially when 
> > comparing with qmrestore's ability to just read read from stdin.

The reading from stdin is quite limited, does not support sparse files
efficiently, and does not support our live-restore feature.

If we can *pull* data out-of-order from the backup provider via a better
protocol (like NBD which Thomas mentioned), holes in the disks  don't
need to be transferred over the network, and we could probably support
live-restore, where the VMs immediately start running *during* the
restore process. (qemu would simply treat read-requests from the guest
which have not yet been restored with a higher priority while otherwise
continuing to copy the data in-order in the background)

> 
> Bareos could provide a NBD that is streamed directly from the backup
> repository, this should be quite efficient w.r.t. space usage.
> But as this was v4 and a few things changed, and I'm less involved as
> the actual author it might make more sense for her to answer this.

Some alternatives btw. are providing a fuse or ublk device on the PVE
side which pulls data from bareos (or to which bareos can push data
which, like qemu's "fleecing" mode, could store the not-yet restored
portions in temporary files, discarding them as they are read by/moved
to qemu).

*Technically* we could have a mode where we allocate the disks and "map"
them onto the system (potentially via nbd, or `rbd map` for ceph etc.)

*But* it would make live-restore impossible with that plugin.

Which is why the most flexible thing to do is to use a `qemu-img` call
and giving it the paths, or more precisely, the URLs to the disks. 

So, for instance, if bareos allowed "pull" based access via nbd, this
would translate to:

    $ qemu-img convert 'nbd://the.bareos.host/drive-scsi0' 'rbd:pool/vm-100-disk-0'

This would transfer from the backup host directly into the ceph storage.

While for live-restore, the regular `qemu` process would perform this in
the background while the VM is already running.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel