public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
To: Jakob-Tobias Winter <tobias@stuttgart.de>, pbs-devel@lists.proxmox.com
Subject: Re: [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter
Date: Wed, 18 Nov 2026 15:22:29 +0100	[thread overview]
Message-ID: <751f5842-78cc-49cd-9e4e-8a4950e46d8f@proxmox.com> (raw)
In-Reply-To: <20261113085637.57360-3-tobias@stuttgart.de>

Thanks for the series @Jakob and thanks for looping me in @Christian.

The overall approach looks good to me too.
I left a few additional inline comments on the individual patches.

I agree with Christian’s suggestion to supply the attributes when the
backup session starts and check each snapshot’s own attributes when
filtering.

Could we also show the stored attributes when inspecting a snapshot
under Datastore > Content? That would make it easier to see which keys
and values to use in a job filter

On 11/13/26 10:07 AM, Jakob-Tobias Winter wrote:
> Tape backup jobs on a Proxmox Backup Server select backup groups by
> type and ID only. Splitting guests over media pools by what they are,
> say production and test guests with different retention, means keeping
> lists of IDs up to date by hand in every job. In Proxmox VE the same
> guests are already organized with tags and resource pools, but the
> backup server knows about neither.
> 
> This series lets Proxmox VE store the tags and the resource pool of a
> guest with each backup, as attributes of the snapshot, and adds a
> filter on such attributes to tape backup jobs:
> 
>    # proxmox-tape backup-job create prod --store store1 --drive lto1 \
>        --pool prod --attribute-filter pve-tag=prod
>    # proxmox-tape backup-job create rest --store store1 --drive lto1 \
>        --pool rest --attribute-filter exclude:pve-tag=prod
> 
> It is an RFC, as it defines a format and names that others would have
> to live with, and because [1] is assigned already. If there is work in
> progress for it, I am happy to adapt this series to it or to drop
> whatever overlaps.
> 
> Proxmox Backup Server
> ---------------------
> 
> proxmox 1-2, proxmox-backup 3-7
> 
> Snapshots get attributes, a map from a key to a list of values. They
> are kept in the unprotected part of the manifest, as suggested in [0],
> so they can be set for encrypted backups without touching the
> signature, and they stay with a snapshot that is synced or written to
> tape. A guest with several tags gets one value per tag, nothing is
> joined into one string.
> 
> They are read and set through a new endpoint next to the notes,
> GET/PUT /admin/datastore/{store}/attributes, with the privileges of
> the notes, as a list of KEY=VALUE entries. A PUT replaces all
> attributes of the snapshot. An empty set is stored as well, to tell a
> guest without tags from a client which never set anything.
> 
> Tape backup jobs get the option 'attribute-filter', a list of
> [exclude:|include:]KEY=VALUE with the include and exclude rules of
> group filters, applied after the group filter. A group has the
> attributes of its newest finished snapshot that has some, so a single
> backup without attributes does not move a group to another media pool.
> Besides the API there is the option for 'proxmox-tape', a field in the
> job window, a column in the job list and documentation.
> 
> The server does not interpret keys or values.
> 
> Proxmox VE
> ----------
> 
> storage 8, guest-common 9, manager 10-11, docs 12
> 
> After a backup to a PBS storage, vzdump sets 'pve-tag' (one value per
> tag) and 'pve-pool', with the prefix suggested in [0], directly
> through the API like the protection flag. A guest in a nested pool
> gets that pool and each of its parent pools as values, so a filter
> for a pool also matches the guests of the pools nested in it. A
> server without the endpoint answers 404, which is only noted in the
> task log.
> 
> As the attributes are not encrypted, the new vzdump option
> 'pbs-snapshot-attributes' (default on) can turn this off for a job, a
> single call, or a node in /etc/vzdump.conf.
> 
> Compatibility
> -------------
> 
> * Jobs without the new option select the same groups as before, and
>    nothing changes on tape or in the catalogs. A tape job written by
>    4.2.9 was read and run unchanged.
> * The other way round, a version without the series cannot parse a
>    tape-job.cfg in which the option is used (unknown property), like
>    with other new job properties.
> * A backup server without the series leaves the attributes in the
>    manifest alone (tried with 4.2.9).
> * A PVE node without the series ignores the vzdump option in jobs.cfg
>    with a warning.
> 
> Open questions
> --------------
> 
> * The names: 'pve-tag', 'pve-pool', 'attribute-filter' and
>    'pbs-snapshot-attributes'.
> * A guest in the pool 'a/b' gets the 'pve-pool' values 'a/b' and 'a',
>    so one filter covers a pool with everything nested in it. A backup
>    job for the pool 'a' in Proxmox VE only takes the direct members of
>    'a'. Is that difference fine, or should only 'a/b' be stored?
>    Since nested pools are quite new I wasn't sure about that at all.
> * Setting the attributes after the backup, like notes and protection,
>    needs no change in the backup protocol or in the QEMU library. But
>    a user that may only append, as in [2], could then not set them.
>    Passing them when the backup is created would solve that, at the
>    price of changes in proxmox-backup-qemu, pve-qemu, qemu-server and
>    pve-container which I found a bit excessive for some attributes.
> * On by default with the option to opt out, or the other way round?
>    Is the switch needed at all?
> * This is a separate job option and not a new kind of group filter,
>    as group filters are shared with sync jobs and match on the name of
>    a group only. Using attributes in other jobs and showing them in the
>    web interface could follow, if wanted.
> * For a group in which no snapshot has attributes, all its manifests
>    are read when a job with a filter runs. Is that acceptable, or
>    should only the newest few be looked at?
> 
> Testing
> -------
> 
> proxmox-backup master (4.2.11) with the series: rustfmt, clippy
> without new warnings, every commit builds, 'cargo test --workspace'
> passes (162 tests, one of them new), the docs build as HTML and man
> pages. The tests of pbs-api-types pass, with five new ones for the
> filter. pbs-api-types 1.0.22 came from the proxmox repository, as it
> is not in the devel repository yet.
> 
> The PVE patches are perltidy and biome clean, and the test suites of
> the three Perl repositories give the same results with and without
> them.
> 
> End to end in an unprivileged sandbox with the real daemons of both
> products: the packages of PVE 9.2.26 with the PVE patches applied and
> a backup server built from master with the series. There is no KVM in
> it, so the guests were stopped VMs and a container.
> 
> * vzdump of VMs and a container to a plain and an encrypted PBS
>    storage, the latter with a namespace, as a user with DatastoreBackup
>    only; several tags, pools nested up to three levels, changed and
>    removed tags; the option per call, per job (also run by pvescheduler
>    and from the web interface) and in /etc/vzdump.conf; restore of an
>    encrypted backup
> * the same against 4.2.9 and plain master: noted in the task log, no
>    warning, the backup is fine
> * the API as owner, as another user, as auditor and as admin; invalid
>    values; verification, notes, protection and a local sync job keep
>    the attributes
> * tape jobs on the virtual tape drive (needs [3]) selecting by tag, by
>    pool and by the parent of a nested pool, combined with a group
>    filter and with namespaces, started by hand and by the scheduler; a
>    restore from tape keeps the attributes
> * both web interfaces in a browser
> 
> Not yet tested: real tape hardware, clusters with more than one node,
> datastores with an S3 backend.
> 
> I wasn't sure if I was supposed to send to pbs-devel and pve-devel.
> 
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7452
> [2] https://lore.proxmox.com/pbs-devel/20261106172235.890542-1-c.ebner@proxmox.com/
> [3] https://lore.proxmox.com/pbs-devel/20261111204853.531985-2-tobias@stuttgart.de/
> 
> 
> proxmox:
> 
> Jakob-Tobias Winter (2):
>    pbs-api-types: add types and schemas for snapshot attributes
>    pbs-api-types: tape: add attribute filter to backup jobs
> 
>   pbs-api-types/src/datastore.rs                |  22 ++-
>   pbs-api-types/src/jobs.rs                     |  98 ++++++++++-
>   pbs-api-types/src/lib.rs                      |   3 +
>   pbs-api-types/tests/attribute_filter_tests.rs | 155 ++++++++++++++++++
>   4 files changed, 276 insertions(+), 2 deletions(-)
>   create mode 100644 pbs-api-types/tests/attribute_filter_tests.rs
> 
> 
> proxmox-backup:
> 
> Jakob-Tobias Winter (5):
>    datastore: add attributes to backup snapshots
>    fix #7452: api: datastore: add endpoints for snapshot attributes
>    tape: backup job: allow selecting groups by snapshot attributes
>    ui: tape backup job: add attribute filter
>    docs: describe snapshot attributes and the tape job attribute filter
> 
>   docs/pve-integration.rst           |  23 ++++++
>   docs/tape-backup.rst               |  27 ++++++
>   docs/terminology.rst               |  33 ++++++++
>   pbs-datastore/src/backup_info.rs   |  18 +++-
>   pbs-datastore/src/datastore.rs     |  15 +++-
>   pbs-datastore/src/manifest.rs      |  87 +++++++++++++++++++-
>   src/api2/admin/datastore.rs        | 128 ++++++++++++++++++++++++++++-
>   src/api2/config/tape_backup_job.rs |   8 ++
>   src/api2/tape/backup.rs            |  19 ++++-
>   src/bin/proxmox-tape.rs            |  11 ++-
>   www/tape/BackupJobs.js             |   7 ++
>   www/tape/window/TapeBackupJob.js   |  27 +++++-
>   12 files changed, 391 insertions(+), 12 deletions(-)
> 
> 
> storage:
> 
> Jakob-Tobias Winter (1):
>    pbs: add helper to set the attributes of a backup snapshot
> 
>   src/PVE/Storage/PBSPlugin.pm | 23 +++++++++++++++++++++++
>   1 file changed, 23 insertions(+)
> 
> 
> guest-common:
> 
> Jakob-Tobias Winter (1):
>    vzdump: add pbs-snapshot-attributes option
> 
>   src/PVE/VZDump/Common.pm | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> 
> manager:
> 
> Jakob-Tobias Winter (2):
>    vzdump: store guest tags and pool as snapshot attributes on PBS
>    ui: backup job: add selector for PBS snapshot attributes
> 
>   PVE/VZDump.pm                               | 42 +++++++++++++++++++++
>   configs/vzdump.conf                         |  1 +
>   www/manager6/panel/BackupAdvancedOptions.js | 22 +++++++++++
>   3 files changed, 65 insertions(+)
> 
> 
> docs:
> 
> Jakob-Tobias Winter (1):
>    vzdump: describe snapshot attributes on Proxmox Backup Server
> 
>   vzdump.adoc | 36 ++++++++++++++++++++++++++++++++++++
>   1 file changed, 36 insertions(+)
> 
> 
> Summary over all repositories:
>    22 files changed, 799 insertions(+), 14 deletions(-)
> 





  parent reply	other threads:[~2026-11-18 14:22 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13  8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-18 12:54   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-18 12:55   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06   ` Samuel Rufinatscha
2026-11-18 15:19     ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07   ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha [this message]
2026-11-19 11:58   ` Jakob-Tobias Winter

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=751f5842-78cc-49cd-9e4e-8a4950e46d8f@proxmox.com \
    --to=s.rufinatscha@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    --cc=tobias@stuttgart.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal