public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Wolfgang Bumiller <w.bumiller@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [pve-devel] [PATCH v8 storage 0/9] backup provider API
Date: Thu,  3 Apr 2025 14:30:40 +0200	[thread overview]
Message-ID: <20250403123118.264974-1-w.bumiller@proxmox.com> (raw)

v7: https://lore.proxmox.com/pve-devel/20250401173435.221892-1-f.ebner@proxmox.com/

I picked this up since Fiona is currently not available.

This contains 1 "major" change (and incorporates some minor doc feedback
from Max (sans the `=head` rewrite, feel free to send follow-up patches
for this @Max).

The change is that instead of having a 

    sub backup_vm_available_bitmaps($self, $volumes)

which fills bitmap names into the passed-along $volumes array,
there is now a

    sub backup_vm_query_incremental($self, $volumes)

which should *return* a new hash (as modifying parameters is a bit
awkward if a plugin is not written in perl, eg. our rust perlmod...),
which instead of bitmap *names*, contains bitmap *modes* which are:

    - new: create a new bitmap, or *recreate* an existing one
    - use: use an existing bitmap, or create one if none exists

Bitmap names have two issues:
1. It is not clear if they are actually useful.
2. They do not provide a way to distinguish between the above modes.
   If the backup provider loses access to a drive which used a bitmap
   before, the only way to recreate it was to *rename* it, which meant
   keeping track of those names.

   So now the only thing we need to keep track of, which for some backup
   providers can *probably* happen without any extra state, is which
   drive a bitmap is relative to.
   Tracking names on the other hand is quite an awkward interface.

Changes in v8:
- storage: replace backup_vm_available_bitmaps() with backup_vm_query_incremental()
- qemu: add an explicit optional 'bitmap-mode' parameter to
  qmp_backup_access_setup()
- qemu-sever: adapt to the storage api change and pass the bitmap-mode
  to the qmp-backup-access-setup command
- storage: adapt directory example
- storage: change the previous-info file to be a json hash mapping disks
  to their last backups
- qemu-server: fix 'bwlimit'/'bwlimiit' typo
- qemu: fix a leak on error in `get_single_device_info()`

build order:
1. qemu
2. storage
3. a) qemu-server: depends on qemu + storage
   b) pve-container: depends on storage
4. manager: depends on qemu-server/container I guess


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel


             reply	other threads:[~2025-04-03 12:34 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-04-03 12:30 Wolfgang Bumiller [this message]
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 01/10] PVE backup: clean up directly in setup_snapshot_access() when it fails Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 02/10] PVE backup: factor out helper to clear backup state's bitmap list Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 03/10] PVE backup: factor out helper to initialize backup state stat struct Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 04/10] PVE backup: add target ID in backup state Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 05/10] PVE backup: get device info: allow caller to specify filter for which devices use fleecing Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 06/10] PVE backup: implement backup access setup and teardown API for external providers Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 07/10] PVE backup: factor out get_single_device_info() helper Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 08/10] PVE backup: implement bitmap support for external backup access Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 09/10] PVE backup: backup-access api: indicate situation where a bitmap was recreated Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu 10/10] PVE backup: backup-access-api: explicit bitmap-mode parameter Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 storage 1/8] add storage_has_feature() helper function Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 storage 2/8] common: add deallocate " Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 storage 3/8] plugin: introduce new_backup_provider() method Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 storage 4/8] config api/plugins: let plugins define sensitive properties themselves Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 storage 5/8] plugin api: bump api version and age Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 storage 6/8] extract backup config: delegate to backup provider for storages that support it Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [POC v8 storage 7/8] add backup provider example Wolfgang Bumiller
2025-04-04  6:58   ` Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [POC v8 storage 8/8] Borg example plugin Wolfgang Bumiller
2025-04-03 12:30 ` [pve-devel] [PATCH v8 qemu-server 01/11] backup: keep track of block-node size for fleecing Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 02/11] backup: fleecing: use exact size when allocating non-raw fleecing images Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 03/11] backup: allow adding fleecing images also for EFI and TPM Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 04/11] backup: implement backup for external providers Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 05/11] test: qemu img convert: add test cases for snapshots Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 06/11] image convert: collect options in hash argument Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 07/11] image convert: allow caller to specify the format of the source path Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 08/11] backup: implement restore for external providers Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 09/11] backup: future-proof checks for QEMU feature support Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 10/11] backup: support 'missing-recreated' bitmap action Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 qemu-server 11/11] backup: bitmap action to human: lie about TPM state Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 container 1/7] add LXC::Namespaces module Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 container 2/7] backup: implement backup for external providers Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 container 3/7] backup: implement restore " Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 container 4/7] external restore: don't use 'one-file-system' tar flag when restoring from a directory Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 container 5/7] create: factor out compression option helper Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 container 6/7] restore tar archive: check potentially untrusted archive Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 container 7/7] api: add early check against restoring privileged container from external source Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 manager 1/2] ui: backup: also check for backup subtype to classify archive Wolfgang Bumiller
2025-04-03 12:31 ` [pve-devel] [PATCH v8 manager 2/2] backup: implement backup for external providers Wolfgang Bumiller
2025-04-03 16:10 ` [pve-devel] partially-applied-series: [PATCH v8 storage 0/9] backup provider API Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250403123118.264974-1-w.bumiller@proxmox.com \
    --to=w.bumiller@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal