From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 12EEC1FF163 for ; Thu, 12 Sep 2024 14:44:34 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2024330F31; Thu, 12 Sep 2024 14:44:34 +0200 (CEST) Date: Thu, 12 Sep 2024 14:43:57 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Proxmox VE development discussion References: <20240813132829.117460-1-f.ebner@proxmox.com> <20240813132829.117460-11-f.ebner@proxmox.com> In-Reply-To: <20240813132829.117460-11-f.ebner@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.16.0 (https://github.com/astroidmail/astroid) Message-Id: <1726143164.cctor3lqz7.astroid@yuna.none> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.050 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [RFC storage v2 10/25] plugin: introduce new_backup_provider() method X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" high-level comment: nicely documented! but wow is POD annoying to read in source form ;) On August 13, 2024 3:28 pm, Fiona Ebner wrote: > The new_backup_provider() method can be used by storage plugins for > external backup providers. If the method returns a provider, Proxmox > VE will use callbacks to that provider for backups and restore instead > of using its usual backup/restore mechanisms. > > API age and version are both bumped. > > The backup provider API is split into two parts, both of which again > need different implementations for VM and LXC guests: > > 1. Backup API > > There are two hook callback functions, namely: > 1. job_hook() is called during the start/end/abort phases of the > whole backup job. > 2. backup_hook() is called during the start/end/abort phases of the > backup of an individual guest. > > The backup_get_mechanism() method is used to decide on the backup > mechanism. Currently, 'block-device' or 'nbd' for VMs, and 'directory' > for containers is possible. The method also let's the plugin indicate > whether to use a bitmap for incremental VM backup or not. It is enough > to implement one mechanism for VMs and one mechanism for containers. > > Next, there are methods for backing up the guest's configuration and > data, backup_vm() for VM backup and backup_container() for container > backup. > > Finally, some helpers like getting the provider name or volume ID for > the backup target, as well as for handling the backup log. > > 1.1 Backup Mechanisms > > VM: > > Access to the data on the VM's disk from the time the backup started > is made available via a so-called "snapshot access". This is either > the full image, or in case a bitmap is used, the dirty parts of the > image since the last time the bitmap was used for a successful backup. > Reading outside of the dirty parts will result in an error. After > backing up each part of the disk, it should be discarded in the export > to avoid unnecessary space usage on the Proxmox VE side (there is an > associated fleecing image). > > VM mechanism 'block-device': > > The snapshot access is exposed as a block device. If used, a bitmap is > passed along. > > VM mechanism 'nbd': > > The snapshot access and, if used, bitmap are exported via NBD. > > Container mechanism 'directory': > > A copy or snapshot of the container's filesystem state is made > available as a directory. > > 2. Restore API > > The restore_get_mechanism() method is used to decide on the restore > mechanism. Currently, 'qemu-img' for VMs, and 'directory' or 'tar' for > containers are possible. It is enough to implement one mechanism for > VMs and one mechanism for containers. > > Next, methods for extracting the guest and firewall configuration and > the implementations of the restore mechanism via a pair of methods: an > init method, for making the data available to Proxmox VE and a cleanup > method that is called after restore. > > For VMs, there also is a restore_vm_get_device_info() helper required, > to get the disks included in the backup and their sizes. > > 2.1. Restore Mechanisms > > VM mechanism 'qemu-img': > > The backup provider gives a path to the disk image that will be > restored. The path needs to be something 'qemu-img' can deal with, > e.g. can also be an NBD URI or similar. > > Container mechanism 'directory': > > The backup provider gives the path to a directory with the full > filesystem structure of the container. > > Container mechanism 'directory': > > The backup provider gives the path to a (potentially compressed) tar > archive with the full filesystem structure of the container. same as in the cover letter ;) base on the code here I assume the second one should be tar. I wonder whether just tar wouldn't be enough (easier to not get ACLs/xattrs/ownership/.. right)? > > See the PVE::BackupProvider::Plugin module for the full API > documentation. > > Signed-off-by: Fiona Ebner > --- > [snip..] > diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm > index 6444390..d5b76ae 100644 > --- a/src/PVE/Storage/Plugin.pm > +++ b/src/PVE/Storage/Plugin.pm > @@ -1755,6 +1755,21 @@ sub rename_volume { > return "${storeid}:${base}${target_vmid}/${target_volname}"; > } > > +# Used by storage plugins for external backup providers. See PVE::BackupProvider::Plugin for the API > +# the provider needs to implement. > +# > +# $scfg - the storage configuration > +# $storeid - the storage ID > +# $log_function($log_level, $message) - this log function can be used to write to the backup task > +# log in Proxmox VE. $log_level is 'info', 'warn' or 'err', $message is the message to be printed. > +# > +# Returns a blessed reference to the backup provider class. > +sub new_backup_provider { > + my ($class, $scfg, $storeid, $log_function) = @_; > + > + return; > +} would it maybe make sense to make this a "die implement me" and make the opt-in via the storage plugin features? it would be more in line with what we do in other parts and less subtle.. > + > sub config_aware_base_mkdir { > my ($class, $scfg, $path) = @_; > > -- > 2.39.2 > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel