From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 649B51FF161 for ; Tue, 13 Aug 2024 15:29:16 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 60F0B203F; Tue, 13 Aug 2024 15:28:48 +0200 (CEST) From: Fiona Ebner To: pve-devel@lists.proxmox.com Date: Tue, 13 Aug 2024 15:28:04 +0200 Message-Id: <20240813132829.117460-1-f.ebner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.054 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pve-devel] [RFC qemu/storage/qemu-server/container/manager v2 00/25] backup provider API X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" Changes in v2: * Add 'block-device' backup mechansim for VMs. The NBD export is mounted by Proxmox VE and only the block device path (as well as a callback to get the next dirty range for bitmaps) is passed to the backup provider. * Add POC example for Borg - note that I tested with borg 1.2.4 in Debian and only tested with a local repository, not SSH yet. * Merge hook API into a single function for backup and for jobs. * Add restore_vm_init() and restore_vm_cleanup() for better flexibility to allow preparing the whole restore. Question is if restore_vm_volume_init() and restore_vm_volume_cleanup() should be dropped (but certain providers might prefer using only those)? Having both is more flexible, but makes the API longer of course. * Switch to backup_vm() (was per-volume backup_vm_volume() before) and backup_container(), passing along the configuration files, rather than having dedicated methods for the configuration files, for giving the backup provider more flexibility. * Some renames in API methods/params to improve clarity. * Pass backup time to backup 'start' hook and use that in the directory example rather than the job start time. * Use POD for base plugin documentation and flesh out documentation. * Use 'BackupProvider::Plugin::' namespace. * Various smaller improvements in the directory provider example. ====== A backup provider needs to implement a storage plugin as well as a backup provider plugin. The storage plugin is for integration in Proxmox VE's front-end, so users can manage the backups via UI/API/CLI. The backup provider plugin is for interfacing with the backup provider's backend to integrate backup and restore with that backend into Proxmox VE. This is an initial draft of an API and required changes to the backup stack in Proxmox VE to make it work. Depending on feedback from other developers and interested parties, it can still substantially change. ====== The backup provider API is split into two parts, both of which again need different implementations for VM and LXC guests: 1. Backup API There are two hook callback functions, namely: 1. job_hook() is called during the start/end/abort phases of the whole backup job. 2. backup_hook() is called during the start/end/abort phases of the backup of an individual guest. The backup_get_mechanism() method is used to decide on the backup mechanism. Currently, 'block-device' or 'nbd' for VMs, and 'directory' for containers is possible. The method also let's the plugin indicate whether to use a bitmap for incremental VM backup or not. It is enough to implement one mechanism for VMs and one mechanism for containers. Next, there are methods for backing up the guest's configuration and data, backup_vm() for VM backup and backup_container() for container backup. Finally, some helpers like getting the provider name or volume ID for the backup target, as well as for handling the backup log. 1.1 Backup Mechanisms VM: Access to the data on the VM's disk from the time the backup started is made available via a so-called "snapshot access". This is either the full image, or in case a bitmap is used, the dirty parts of the image since the last time the bitmap was used for a successful backup. Reading outside of the dirty parts will result in an error. After backing up each part of the disk, it should be discarded in the export to avoid unnecessary space usage on the Proxmox VE side (there is an associated fleecing image). VM mechanism 'block-device': The snapshot access is exposed as a block device. If used, a bitmap is passed along. VM mechanism 'nbd': The snapshot access and, if used, bitmap are exported via NBD. Container mechanism 'directory': A copy or snapshot of the container's filesystem state is made available as a directory. 2. Restore API The restore_get_mechanism() method is used to decide on the restore mechanism. Currently, 'qemu-img' for VMs, and 'directory' or 'tar' for containers are possible. It is enough to implement one mechanism for VMs and one mechanism for containers. Next, methods for extracting the guest and firewall configuration and the implementations of the restore mechanism via a pair of methods: an init method, for making the data available to Proxmox VE and a cleanup method that is called after restore. For VMs, there also is a restore_vm_get_device_info() helper required, to get the disks included in the backup and their sizes. 2.1. Restore Mechanisms VM mechanism 'qemu-img': The backup provider gives a path to the disk image that will be restored. The path needs to be something 'qemu-img' can deal with, e.g. can also be an NBD URI or similar. Container mechanism 'directory': The backup provider gives the path to a directory with the full filesystem structure of the container. Container mechanism 'directory': The backup provider gives the path to a (potentially compressed) tar archive with the full filesystem structure of the container. See the PVE::BackupProvider::Plugin module for the full API documentation. ====== This series adapts the backup stack in Proxmox VE to allow using the above API. For QEMU, backup access setup and teardown QMP commands are implemented to be able to provide access to a consistent disk state to the backup provider. The series also provides an example implementation for a backup provider as a proof-of-concept, exposing the different features. ====== Open questions: Should the backup provider plugin system also follow the same API age+version schema with a Custom/ directory for external plugins derived from the base plugin? Should the bitmap action be passed directly to the backup provider? I.e. have 'not-used', 'not-used-removed', 'new', 'used', 'invalid', instead of only 'none', 'new' and 'reuse'. It makes API slightly more complicated. Is there any situation where backup provider could care if bitmap is new, because it was the first or bitmap is new because previous was invalid? Both cases require the backup provider to do a full backup. ====== The patches marked as PATCH rather than RFC can make sense independently, with QEMU patches 02 and 03 having been sent already before (touching same code, so included here): https://lore.proxmox.com/pve-devel/20240625133551.210636-1-f.ebner@proxmox.com/#r ====== Feedback is very welcome, especially from people wishing to implement such a backup provider plugin! Please tell me what issues you see with the proposed API, what would and wouldn't work from your perspective? ====== Dependencies: pve-manager, pve-container and qemu-server all depend on new libpve-storage-perl. pve-manager also build-depends on the new libpve-storage-perl for its tests. To keep things clean, pve-manager should also depend on new pve-container and qemu-server. In qemu-server, there is no version guard added yet, as that depends on the QEMU version the feature will land in. ====== qemu: Fiona Ebner (9): block/reqlist: allow adding overlapping requests PVE backup: fixup error handling for fleecing PVE backup: factor out setting up snapshot access for fleecing PVE backup: save device name in device info structure PVE backup: include device name in error when setting up snapshot access fails PVE backup: add target ID in backup state PVE backup: get device info: allow caller to specify filter for which devices use fleecing PVE backup: implement backup access setup and teardown API for external providers PVE backup: implement bitmap support for external backup access block/copy-before-write.c | 3 +- block/reqlist.c | 2 - pve-backup.c | 620 +++++++++++++++++++++++++++++++++----- pve-backup.h | 16 + qapi/block-core.json | 61 ++++ system/runstate.c | 6 + 6 files changed, 637 insertions(+), 71 deletions(-) create mode 100644 pve-backup.h storage: Fiona Ebner (4): plugin: introduce new_backup_provider() method extract backup config: delegate to backup provider if there is one add backup provider example WIP Borg plugin src/PVE/BackupProvider/Makefile | 3 + src/PVE/BackupProvider/Plugin/Base.pm | 1149 +++++++++++++++++ src/PVE/BackupProvider/Plugin/Borg.pm | 373 ++++++ .../BackupProvider/Plugin/DirectoryExample.pm | 694 ++++++++++ src/PVE/BackupProvider/Plugin/Makefile | 5 + src/PVE/Makefile | 1 + src/PVE/Storage.pm | 24 +- src/PVE/Storage/BorgBackupPlugin.pm | 506 ++++++++ .../Custom/BackupProviderDirExamplePlugin.pm | 306 +++++ src/PVE/Storage/Custom/Makefile | 5 + src/PVE/Storage/Makefile | 2 + src/PVE/Storage/Plugin.pm | 15 + 12 files changed, 3081 insertions(+), 2 deletions(-) create mode 100644 src/PVE/BackupProvider/Makefile create mode 100644 src/PVE/BackupProvider/Plugin/Base.pm create mode 100644 src/PVE/BackupProvider/Plugin/Borg.pm create mode 100644 src/PVE/BackupProvider/Plugin/DirectoryExample.pm create mode 100644 src/PVE/BackupProvider/Plugin/Makefile create mode 100644 src/PVE/Storage/BorgBackupPlugin.pm create mode 100644 src/PVE/Storage/Custom/BackupProviderDirExamplePlugin.pm create mode 100644 src/PVE/Storage/Custom/Makefile qemu-server: Fiona Ebner (8): move nbd_stop helper to QMPHelpers module backup: move cleanup of fleecing images to cleanup method backup: cleanup: check if VM is running before issuing QMP commands backup: keep track of block-node size instead of volume size backup: allow adding fleecing images also for EFI and TPM backup: implement backup for external providers restore: die early when there is no size for a device backup: implement restore for external providers PVE/API2/Qemu.pm | 32 ++- PVE/CLI/qm.pm | 3 +- PVE/QemuServer.pm | 146 +++++++++++++- PVE/QemuServer/QMPHelpers.pm | 6 + PVE/VZDump/QemuServer.pm | 370 ++++++++++++++++++++++++++++++++--- 5 files changed, 519 insertions(+), 38 deletions(-) container: Fiona Ebner (2): backup: implement backup for external providers backup: implement restore for external providers src/PVE/LXC/Create.pm | 141 ++++++++++++++++++++++++++++++++++++++++++ src/PVE/VZDump/LXC.pm | 22 ++++++- 2 files changed, 162 insertions(+), 1 deletion(-) manager: Fiona Ebner (2): ui: backup: also check for backup subtype to classify archive backup: implement backup for external providers PVE/VZDump.pm | 62 ++++++++++++++++++++++++++---- test/vzdump_new_test.pl | 3 ++ www/manager6/Utils.js | 10 +++-- www/manager6/grid/BackupView.js | 4 +- www/manager6/storage/BackupView.js | 4 +- 5 files changed, 68 insertions(+), 15 deletions(-) Summary over all repositories: 30 files changed, 4467 insertions(+), 127 deletions(-) -- Generated by git-murpp 0.5.0 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel