From: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
To: Jakob-Tobias Winter <tobias@stuttgart.de>, pbs-devel@lists.proxmox.com
Subject: Re: [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter
Date: Wed, 18 Nov 2026 15:22:29 +0100 [thread overview]
Message-ID: <751f5842-78cc-49cd-9e4e-8a4950e46d8f@proxmox.com> (raw)
In-Reply-To: <20261113085637.57360-3-tobias@stuttgart.de>
Thanks for the series @Jakob and thanks for looping me in @Christian.
The overall approach looks good to me too.
I left a few additional inline comments on the individual patches.
I agree with Christian’s suggestion to supply the attributes when the
backup session starts and check each snapshot’s own attributes when
filtering.
Could we also show the stored attributes when inspecting a snapshot
under Datastore > Content? That would make it easier to see which keys
and values to use in a job filter
On 11/13/26 10:07 AM, Jakob-Tobias Winter wrote:
> Tape backup jobs on a Proxmox Backup Server select backup groups by
> type and ID only. Splitting guests over media pools by what they are,
> say production and test guests with different retention, means keeping
> lists of IDs up to date by hand in every job. In Proxmox VE the same
> guests are already organized with tags and resource pools, but the
> backup server knows about neither.
>
> This series lets Proxmox VE store the tags and the resource pool of a
> guest with each backup, as attributes of the snapshot, and adds a
> filter on such attributes to tape backup jobs:
>
> # proxmox-tape backup-job create prod --store store1 --drive lto1 \
> --pool prod --attribute-filter pve-tag=prod
> # proxmox-tape backup-job create rest --store store1 --drive lto1 \
> --pool rest --attribute-filter exclude:pve-tag=prod
>
> It is an RFC, as it defines a format and names that others would have
> to live with, and because [1] is assigned already. If there is work in
> progress for it, I am happy to adapt this series to it or to drop
> whatever overlaps.
>
> Proxmox Backup Server
> ---------------------
>
> proxmox 1-2, proxmox-backup 3-7
>
> Snapshots get attributes, a map from a key to a list of values. They
> are kept in the unprotected part of the manifest, as suggested in [0],
> so they can be set for encrypted backups without touching the
> signature, and they stay with a snapshot that is synced or written to
> tape. A guest with several tags gets one value per tag, nothing is
> joined into one string.
>
> They are read and set through a new endpoint next to the notes,
> GET/PUT /admin/datastore/{store}/attributes, with the privileges of
> the notes, as a list of KEY=VALUE entries. A PUT replaces all
> attributes of the snapshot. An empty set is stored as well, to tell a
> guest without tags from a client which never set anything.
>
> Tape backup jobs get the option 'attribute-filter', a list of
> [exclude:|include:]KEY=VALUE with the include and exclude rules of
> group filters, applied after the group filter. A group has the
> attributes of its newest finished snapshot that has some, so a single
> backup without attributes does not move a group to another media pool.
> Besides the API there is the option for 'proxmox-tape', a field in the
> job window, a column in the job list and documentation.
>
> The server does not interpret keys or values.
>
> Proxmox VE
> ----------
>
> storage 8, guest-common 9, manager 10-11, docs 12
>
> After a backup to a PBS storage, vzdump sets 'pve-tag' (one value per
> tag) and 'pve-pool', with the prefix suggested in [0], directly
> through the API like the protection flag. A guest in a nested pool
> gets that pool and each of its parent pools as values, so a filter
> for a pool also matches the guests of the pools nested in it. A
> server without the endpoint answers 404, which is only noted in the
> task log.
>
> As the attributes are not encrypted, the new vzdump option
> 'pbs-snapshot-attributes' (default on) can turn this off for a job, a
> single call, or a node in /etc/vzdump.conf.
>
> Compatibility
> -------------
>
> * Jobs without the new option select the same groups as before, and
> nothing changes on tape or in the catalogs. A tape job written by
> 4.2.9 was read and run unchanged.
> * The other way round, a version without the series cannot parse a
> tape-job.cfg in which the option is used (unknown property), like
> with other new job properties.
> * A backup server without the series leaves the attributes in the
> manifest alone (tried with 4.2.9).
> * A PVE node without the series ignores the vzdump option in jobs.cfg
> with a warning.
>
> Open questions
> --------------
>
> * The names: 'pve-tag', 'pve-pool', 'attribute-filter' and
> 'pbs-snapshot-attributes'.
> * A guest in the pool 'a/b' gets the 'pve-pool' values 'a/b' and 'a',
> so one filter covers a pool with everything nested in it. A backup
> job for the pool 'a' in Proxmox VE only takes the direct members of
> 'a'. Is that difference fine, or should only 'a/b' be stored?
> Since nested pools are quite new I wasn't sure about that at all.
> * Setting the attributes after the backup, like notes and protection,
> needs no change in the backup protocol or in the QEMU library. But
> a user that may only append, as in [2], could then not set them.
> Passing them when the backup is created would solve that, at the
> price of changes in proxmox-backup-qemu, pve-qemu, qemu-server and
> pve-container which I found a bit excessive for some attributes.
> * On by default with the option to opt out, or the other way round?
> Is the switch needed at all?
> * This is a separate job option and not a new kind of group filter,
> as group filters are shared with sync jobs and match on the name of
> a group only. Using attributes in other jobs and showing them in the
> web interface could follow, if wanted.
> * For a group in which no snapshot has attributes, all its manifests
> are read when a job with a filter runs. Is that acceptable, or
> should only the newest few be looked at?
>
> Testing
> -------
>
> proxmox-backup master (4.2.11) with the series: rustfmt, clippy
> without new warnings, every commit builds, 'cargo test --workspace'
> passes (162 tests, one of them new), the docs build as HTML and man
> pages. The tests of pbs-api-types pass, with five new ones for the
> filter. pbs-api-types 1.0.22 came from the proxmox repository, as it
> is not in the devel repository yet.
>
> The PVE patches are perltidy and biome clean, and the test suites of
> the three Perl repositories give the same results with and without
> them.
>
> End to end in an unprivileged sandbox with the real daemons of both
> products: the packages of PVE 9.2.26 with the PVE patches applied and
> a backup server built from master with the series. There is no KVM in
> it, so the guests were stopped VMs and a container.
>
> * vzdump of VMs and a container to a plain and an encrypted PBS
> storage, the latter with a namespace, as a user with DatastoreBackup
> only; several tags, pools nested up to three levels, changed and
> removed tags; the option per call, per job (also run by pvescheduler
> and from the web interface) and in /etc/vzdump.conf; restore of an
> encrypted backup
> * the same against 4.2.9 and plain master: noted in the task log, no
> warning, the backup is fine
> * the API as owner, as another user, as auditor and as admin; invalid
> values; verification, notes, protection and a local sync job keep
> the attributes
> * tape jobs on the virtual tape drive (needs [3]) selecting by tag, by
> pool and by the parent of a nested pool, combined with a group
> filter and with namespaces, started by hand and by the scheduler; a
> restore from tape keeps the attributes
> * both web interfaces in a browser
>
> Not yet tested: real tape hardware, clusters with more than one node,
> datastores with an S3 backend.
>
> I wasn't sure if I was supposed to send to pbs-devel and pve-devel.
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7452
> [2] https://lore.proxmox.com/pbs-devel/20261106172235.890542-1-c.ebner@proxmox.com/
> [3] https://lore.proxmox.com/pbs-devel/20261111204853.531985-2-tobias@stuttgart.de/
>
>
> proxmox:
>
> Jakob-Tobias Winter (2):
> pbs-api-types: add types and schemas for snapshot attributes
> pbs-api-types: tape: add attribute filter to backup jobs
>
> pbs-api-types/src/datastore.rs | 22 ++-
> pbs-api-types/src/jobs.rs | 98 ++++++++++-
> pbs-api-types/src/lib.rs | 3 +
> pbs-api-types/tests/attribute_filter_tests.rs | 155 ++++++++++++++++++
> 4 files changed, 276 insertions(+), 2 deletions(-)
> create mode 100644 pbs-api-types/tests/attribute_filter_tests.rs
>
>
> proxmox-backup:
>
> Jakob-Tobias Winter (5):
> datastore: add attributes to backup snapshots
> fix #7452: api: datastore: add endpoints for snapshot attributes
> tape: backup job: allow selecting groups by snapshot attributes
> ui: tape backup job: add attribute filter
> docs: describe snapshot attributes and the tape job attribute filter
>
> docs/pve-integration.rst | 23 ++++++
> docs/tape-backup.rst | 27 ++++++
> docs/terminology.rst | 33 ++++++++
> pbs-datastore/src/backup_info.rs | 18 +++-
> pbs-datastore/src/datastore.rs | 15 +++-
> pbs-datastore/src/manifest.rs | 87 +++++++++++++++++++-
> src/api2/admin/datastore.rs | 128 ++++++++++++++++++++++++++++-
> src/api2/config/tape_backup_job.rs | 8 ++
> src/api2/tape/backup.rs | 19 ++++-
> src/bin/proxmox-tape.rs | 11 ++-
> www/tape/BackupJobs.js | 7 ++
> www/tape/window/TapeBackupJob.js | 27 +++++-
> 12 files changed, 391 insertions(+), 12 deletions(-)
>
>
> storage:
>
> Jakob-Tobias Winter (1):
> pbs: add helper to set the attributes of a backup snapshot
>
> src/PVE/Storage/PBSPlugin.pm | 23 +++++++++++++++++++++++
> 1 file changed, 23 insertions(+)
>
>
> guest-common:
>
> Jakob-Tobias Winter (1):
> vzdump: add pbs-snapshot-attributes option
>
> src/PVE/VZDump/Common.pm | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
>
> manager:
>
> Jakob-Tobias Winter (2):
> vzdump: store guest tags and pool as snapshot attributes on PBS
> ui: backup job: add selector for PBS snapshot attributes
>
> PVE/VZDump.pm | 42 +++++++++++++++++++++
> configs/vzdump.conf | 1 +
> www/manager6/panel/BackupAdvancedOptions.js | 22 +++++++++++
> 3 files changed, 65 insertions(+)
>
>
> docs:
>
> Jakob-Tobias Winter (1):
> vzdump: describe snapshot attributes on Proxmox Backup Server
>
> vzdump.adoc | 36 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 36 insertions(+)
>
>
> Summary over all repositories:
> 22 files changed, 799 insertions(+), 14 deletions(-)
>
next prev parent reply other threads:[~2026-11-18 14:22 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:54 ` Samuel Rufinatscha
2026-11-13 8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:55 ` Samuel Rufinatscha
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06 ` Samuel Rufinatscha
2026-11-18 15:19 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07 ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha [this message]
2026-11-19 11:58 ` Jakob-Tobias Winter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=751f5842-78cc-49cd-9e4e-8a4950e46d8f@proxmox.com \
--to=s.rufinatscha@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
--cc=tobias@stuttgart.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox