From: Jakob-Tobias Winter <tobias@stuttgart.de>
To: pbs-devel@lists.proxmox.com
Subject: Re: [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter
Date: Thu, 19 Nov 2026 12:58:16 +0100 [thread overview]
Message-ID: <559fc611-2d42-40c0-ac46-fab2866e501b@stuttgart.de> (raw)
In-Reply-To: <751f5842-78cc-49cd-9e4e-8a4950e46d8f@proxmox.com>
Thanks to both for your feedback, I will be able to invest some more
time next week.
On 11/18/26 15:22, Samuel Rufinatscha wrote:
> Thanks for the series @Jakob and thanks for looping me in @Christian.
>
> The overall approach looks good to me too.
> I left a few additional inline comments on the individual patches.
>
> I agree with Christian’s suggestion to supply the attributes when the
> backup session starts and check each snapshot’s own attributes when
> filtering.
>
> Could we also show the stored attributes when inspecting a snapshot
> under Datastore > Content? That would make it easier to see which keys
> and values to use in a job filter
>
> On 11/13/26 10:07 AM, Jakob-Tobias Winter wrote:
>> Tape backup jobs on a Proxmox Backup Server select backup groups by
>> type and ID only. Splitting guests over media pools by what they are,
>> say production and test guests with different retention, means keeping
>> lists of IDs up to date by hand in every job. In Proxmox VE the same
>> guests are already organized with tags and resource pools, but the
>> backup server knows about neither.
>>
>> This series lets Proxmox VE store the tags and the resource pool of a
>> guest with each backup, as attributes of the snapshot, and adds a
>> filter on such attributes to tape backup jobs:
>>
>> # proxmox-tape backup-job create prod --store store1 --drive lto1 \
>> --pool prod --attribute-filter pve-tag=prod
>> # proxmox-tape backup-job create rest --store store1 --drive lto1 \
>> --pool rest --attribute-filter exclude:pve-tag=prod
>>
>> It is an RFC, as it defines a format and names that others would have
>> to live with, and because [1] is assigned already. If there is work in
>> progress for it, I am happy to adapt this series to it or to drop
>> whatever overlaps.
>>
>> Proxmox Backup Server
>> ---------------------
>>
>> proxmox 1-2, proxmox-backup 3-7
>>
>> Snapshots get attributes, a map from a key to a list of values. They
>> are kept in the unprotected part of the manifest, as suggested in [0],
>> so they can be set for encrypted backups without touching the
>> signature, and they stay with a snapshot that is synced or written to
>> tape. A guest with several tags gets one value per tag, nothing is
>> joined into one string.
>>
>> They are read and set through a new endpoint next to the notes,
>> GET/PUT /admin/datastore/{store}/attributes, with the privileges of
>> the notes, as a list of KEY=VALUE entries. A PUT replaces all
>> attributes of the snapshot. An empty set is stored as well, to tell a
>> guest without tags from a client which never set anything.
>>
>> Tape backup jobs get the option 'attribute-filter', a list of
>> [exclude:|include:]KEY=VALUE with the include and exclude rules of
>> group filters, applied after the group filter. A group has the
>> attributes of its newest finished snapshot that has some, so a single
>> backup without attributes does not move a group to another media pool.
>> Besides the API there is the option for 'proxmox-tape', a field in the
>> job window, a column in the job list and documentation.
>>
>> The server does not interpret keys or values.
>>
>> Proxmox VE
>> ----------
>>
>> storage 8, guest-common 9, manager 10-11, docs 12
>>
>> After a backup to a PBS storage, vzdump sets 'pve-tag' (one value per
>> tag) and 'pve-pool', with the prefix suggested in [0], directly
>> through the API like the protection flag. A guest in a nested pool
>> gets that pool and each of its parent pools as values, so a filter
>> for a pool also matches the guests of the pools nested in it. A
>> server without the endpoint answers 404, which is only noted in the
>> task log.
>>
>> As the attributes are not encrypted, the new vzdump option
>> 'pbs-snapshot-attributes' (default on) can turn this off for a job, a
>> single call, or a node in /etc/vzdump.conf.
>>
>> Compatibility
>> -------------
>>
>> * Jobs without the new option select the same groups as before, and
>> nothing changes on tape or in the catalogs. A tape job written by
>> 4.2.9 was read and run unchanged.
>> * The other way round, a version without the series cannot parse a
>> tape-job.cfg in which the option is used (unknown property), like
>> with other new job properties.
>> * A backup server without the series leaves the attributes in the
>> manifest alone (tried with 4.2.9).
>> * A PVE node without the series ignores the vzdump option in jobs.cfg
>> with a warning.
>>
>> Open questions
>> --------------
>>
>> * The names: 'pve-tag', 'pve-pool', 'attribute-filter' and
>> 'pbs-snapshot-attributes'.
>> * A guest in the pool 'a/b' gets the 'pve-pool' values 'a/b' and 'a',
>> so one filter covers a pool with everything nested in it. A backup
>> job for the pool 'a' in Proxmox VE only takes the direct members of
>> 'a'. Is that difference fine, or should only 'a/b' be stored?
>> Since nested pools are quite new I wasn't sure about that at all.
>> * Setting the attributes after the backup, like notes and protection,
>> needs no change in the backup protocol or in the QEMU library. But
>> a user that may only append, as in [2], could then not set them.
>> Passing them when the backup is created would solve that, at the
>> price of changes in proxmox-backup-qemu, pve-qemu, qemu-server and
>> pve-container which I found a bit excessive for some attributes.
>> * On by default with the option to opt out, or the other way round?
>> Is the switch needed at all?
>> * This is a separate job option and not a new kind of group filter,
>> as group filters are shared with sync jobs and match on the name of
>> a group only. Using attributes in other jobs and showing them in the
>> web interface could follow, if wanted.
>> * For a group in which no snapshot has attributes, all its manifests
>> are read when a job with a filter runs. Is that acceptable, or
>> should only the newest few be looked at?
>>
>> Testing
>> -------
>>
>> proxmox-backup master (4.2.11) with the series: rustfmt, clippy
>> without new warnings, every commit builds, 'cargo test --workspace'
>> passes (162 tests, one of them new), the docs build as HTML and man
>> pages. The tests of pbs-api-types pass, with five new ones for the
>> filter. pbs-api-types 1.0.22 came from the proxmox repository, as it
>> is not in the devel repository yet.
>>
>> The PVE patches are perltidy and biome clean, and the test suites of
>> the three Perl repositories give the same results with and without
>> them.
>>
>> End to end in an unprivileged sandbox with the real daemons of both
>> products: the packages of PVE 9.2.26 with the PVE patches applied and
>> a backup server built from master with the series. There is no KVM in
>> it, so the guests were stopped VMs and a container.
>>
>> * vzdump of VMs and a container to a plain and an encrypted PBS
>> storage, the latter with a namespace, as a user with DatastoreBackup
>> only; several tags, pools nested up to three levels, changed and
>> removed tags; the option per call, per job (also run by pvescheduler
>> and from the web interface) and in /etc/vzdump.conf; restore of an
>> encrypted backup
>> * the same against 4.2.9 and plain master: noted in the task log, no
>> warning, the backup is fine
>> * the API as owner, as another user, as auditor and as admin; invalid
>> values; verification, notes, protection and a local sync job keep
>> the attributes
>> * tape jobs on the virtual tape drive (needs [3]) selecting by tag, by
>> pool and by the parent of a nested pool, combined with a group
>> filter and with namespaces, started by hand and by the scheduler; a
>> restore from tape keeps the attributes
>> * both web interfaces in a browser
>>
>> Not yet tested: real tape hardware, clusters with more than one node,
>> datastores with an S3 backend.
>>
>> I wasn't sure if I was supposed to send to pbs-devel and pve-devel.
>>
>> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32
>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7452
>> [2] https://lore.proxmox.com/pbs-devel/20261106172235.890542-1-
>> c.ebner@proxmox.com/
>> [3] https://lore.proxmox.com/pbs-devel/20261111204853.531985-2-
>> tobias@stuttgart.de/
>>
>>
>> proxmox:
>>
>> Jakob-Tobias Winter (2):
>> pbs-api-types: add types and schemas for snapshot attributes
>> pbs-api-types: tape: add attribute filter to backup jobs
>>
>> pbs-api-types/src/datastore.rs | 22 ++-
>> pbs-api-types/src/jobs.rs | 98 ++++++++++-
>> pbs-api-types/src/lib.rs | 3 +
>> pbs-api-types/tests/attribute_filter_tests.rs | 155 ++++++++++++++++++
>> 4 files changed, 276 insertions(+), 2 deletions(-)
>> create mode 100644 pbs-api-types/tests/attribute_filter_tests.rs
>>
>>
>> proxmox-backup:
>>
>> Jakob-Tobias Winter (5):
>> datastore: add attributes to backup snapshots
>> fix #7452: api: datastore: add endpoints for snapshot attributes
>> tape: backup job: allow selecting groups by snapshot attributes
>> ui: tape backup job: add attribute filter
>> docs: describe snapshot attributes and the tape job attribute filter
>>
>> docs/pve-integration.rst | 23 ++++++
>> docs/tape-backup.rst | 27 ++++++
>> docs/terminology.rst | 33 ++++++++
>> pbs-datastore/src/backup_info.rs | 18 +++-
>> pbs-datastore/src/datastore.rs | 15 +++-
>> pbs-datastore/src/manifest.rs | 87 +++++++++++++++++++-
>> src/api2/admin/datastore.rs | 128 ++++++++++++++++++++++++++++-
>> src/api2/config/tape_backup_job.rs | 8 ++
>> src/api2/tape/backup.rs | 19 ++++-
>> src/bin/proxmox-tape.rs | 11 ++-
>> www/tape/BackupJobs.js | 7 ++
>> www/tape/window/TapeBackupJob.js | 27 +++++-
>> 12 files changed, 391 insertions(+), 12 deletions(-)
>>
>>
>> storage:
>>
>> Jakob-Tobias Winter (1):
>> pbs: add helper to set the attributes of a backup snapshot
>>
>> src/PVE/Storage/PBSPlugin.pm | 23 +++++++++++++++++++++++
>> 1 file changed, 23 insertions(+)
>>
>>
>> guest-common:
>>
>> Jakob-Tobias Winter (1):
>> vzdump: add pbs-snapshot-attributes option
>>
>> src/PVE/VZDump/Common.pm | 8 ++++++++
>> 1 file changed, 8 insertions(+)
>>
>>
>> manager:
>>
>> Jakob-Tobias Winter (2):
>> vzdump: store guest tags and pool as snapshot attributes on PBS
>> ui: backup job: add selector for PBS snapshot attributes
>>
>> PVE/VZDump.pm | 42 +++++++++++++++++++++
>> configs/vzdump.conf | 1 +
>> www/manager6/panel/BackupAdvancedOptions.js | 22 +++++++++++
>> 3 files changed, 65 insertions(+)
>>
>>
>> docs:
>>
>> Jakob-Tobias Winter (1):
>> vzdump: describe snapshot attributes on Proxmox Backup Server
>>
>> vzdump.adoc | 36 ++++++++++++++++++++++++++++++++++++
>> 1 file changed, 36 insertions(+)
>>
>>
>> Summary over all repositories:
>> 22 files changed, 799 insertions(+), 14 deletions(-)
>>
>
prev parent reply other threads:[~2026-11-19 12:07 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:54 ` Samuel Rufinatscha
2026-11-13 8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:55 ` Samuel Rufinatscha
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06 ` Samuel Rufinatscha
2026-11-18 15:19 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07 ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha
2026-11-19 11:58 ` Jakob-Tobias Winter [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=559fc611-2d42-40c0-ac46-fab2866e501b@stuttgart.de \
--to=tobias@stuttgart.de \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox