From: Jakob-Tobias Winter <tobias@stuttgart.de>
To: pbs-devel@lists.proxmox.com
Cc: Jakob-Tobias Winter <tobias@stuttgart.de>
Subject: [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter
Date: Fri, 13 Nov 2026 09:56:27 +0100 [thread overview]
Message-ID: <20261113085637.57360-3-tobias@stuttgart.de> (raw)
Tape backup jobs on a Proxmox Backup Server select backup groups by
type and ID only. Splitting guests over media pools by what they are,
say production and test guests with different retention, means keeping
lists of IDs up to date by hand in every job. In Proxmox VE the same
guests are already organized with tags and resource pools, but the
backup server knows about neither.
This series lets Proxmox VE store the tags and the resource pool of a
guest with each backup, as attributes of the snapshot, and adds a
filter on such attributes to tape backup jobs:
# proxmox-tape backup-job create prod --store store1 --drive lto1 \
--pool prod --attribute-filter pve-tag=prod
# proxmox-tape backup-job create rest --store store1 --drive lto1 \
--pool rest --attribute-filter exclude:pve-tag=prod
It is an RFC, as it defines a format and names that others would have
to live with, and because [1] is assigned already. If there is work in
progress for it, I am happy to adapt this series to it or to drop
whatever overlaps.
Proxmox Backup Server
---------------------
proxmox 1-2, proxmox-backup 3-7
Snapshots get attributes, a map from a key to a list of values. They
are kept in the unprotected part of the manifest, as suggested in [0],
so they can be set for encrypted backups without touching the
signature, and they stay with a snapshot that is synced or written to
tape. A guest with several tags gets one value per tag, nothing is
joined into one string.
They are read and set through a new endpoint next to the notes,
GET/PUT /admin/datastore/{store}/attributes, with the privileges of
the notes, as a list of KEY=VALUE entries. A PUT replaces all
attributes of the snapshot. An empty set is stored as well, to tell a
guest without tags from a client which never set anything.
Tape backup jobs get the option 'attribute-filter', a list of
[exclude:|include:]KEY=VALUE with the include and exclude rules of
group filters, applied after the group filter. A group has the
attributes of its newest finished snapshot that has some, so a single
backup without attributes does not move a group to another media pool.
Besides the API there is the option for 'proxmox-tape', a field in the
job window, a column in the job list and documentation.
The server does not interpret keys or values.
Proxmox VE
----------
storage 8, guest-common 9, manager 10-11, docs 12
After a backup to a PBS storage, vzdump sets 'pve-tag' (one value per
tag) and 'pve-pool', with the prefix suggested in [0], directly
through the API like the protection flag. A guest in a nested pool
gets that pool and each of its parent pools as values, so a filter
for a pool also matches the guests of the pools nested in it. A
server without the endpoint answers 404, which is only noted in the
task log.
As the attributes are not encrypted, the new vzdump option
'pbs-snapshot-attributes' (default on) can turn this off for a job, a
single call, or a node in /etc/vzdump.conf.
Compatibility
-------------
* Jobs without the new option select the same groups as before, and
nothing changes on tape or in the catalogs. A tape job written by
4.2.9 was read and run unchanged.
* The other way round, a version without the series cannot parse a
tape-job.cfg in which the option is used (unknown property), like
with other new job properties.
* A backup server without the series leaves the attributes in the
manifest alone (tried with 4.2.9).
* A PVE node without the series ignores the vzdump option in jobs.cfg
with a warning.
Open questions
--------------
* The names: 'pve-tag', 'pve-pool', 'attribute-filter' and
'pbs-snapshot-attributes'.
* A guest in the pool 'a/b' gets the 'pve-pool' values 'a/b' and 'a',
so one filter covers a pool with everything nested in it. A backup
job for the pool 'a' in Proxmox VE only takes the direct members of
'a'. Is that difference fine, or should only 'a/b' be stored?
Since nested pools are quite new I wasn't sure about that at all.
* Setting the attributes after the backup, like notes and protection,
needs no change in the backup protocol or in the QEMU library. But
a user that may only append, as in [2], could then not set them.
Passing them when the backup is created would solve that, at the
price of changes in proxmox-backup-qemu, pve-qemu, qemu-server and
pve-container which I found a bit excessive for some attributes.
* On by default with the option to opt out, or the other way round?
Is the switch needed at all?
* This is a separate job option and not a new kind of group filter,
as group filters are shared with sync jobs and match on the name of
a group only. Using attributes in other jobs and showing them in the
web interface could follow, if wanted.
* For a group in which no snapshot has attributes, all its manifests
are read when a job with a filter runs. Is that acceptable, or
should only the newest few be looked at?
Testing
-------
proxmox-backup master (4.2.11) with the series: rustfmt, clippy
without new warnings, every commit builds, 'cargo test --workspace'
passes (162 tests, one of them new), the docs build as HTML and man
pages. The tests of pbs-api-types pass, with five new ones for the
filter. pbs-api-types 1.0.22 came from the proxmox repository, as it
is not in the devel repository yet.
The PVE patches are perltidy and biome clean, and the test suites of
the three Perl repositories give the same results with and without
them.
End to end in an unprivileged sandbox with the real daemons of both
products: the packages of PVE 9.2.26 with the PVE patches applied and
a backup server built from master with the series. There is no KVM in
it, so the guests were stopped VMs and a container.
* vzdump of VMs and a container to a plain and an encrypted PBS
storage, the latter with a namespace, as a user with DatastoreBackup
only; several tags, pools nested up to three levels, changed and
removed tags; the option per call, per job (also run by pvescheduler
and from the web interface) and in /etc/vzdump.conf; restore of an
encrypted backup
* the same against 4.2.9 and plain master: noted in the task log, no
warning, the backup is fine
* the API as owner, as another user, as auditor and as admin; invalid
values; verification, notes, protection and a local sync job keep
the attributes
* tape jobs on the virtual tape drive (needs [3]) selecting by tag, by
pool and by the parent of a nested pool, combined with a group
filter and with namespaces, started by hand and by the scheduler; a
restore from tape keeps the attributes
* both web interfaces in a browser
Not yet tested: real tape hardware, clusters with more than one node,
datastores with an S3 backend.
I wasn't sure if I was supposed to send to pbs-devel and pve-devel.
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7452
[2] https://lore.proxmox.com/pbs-devel/20261106172235.890542-1-c.ebner@proxmox.com/
[3] https://lore.proxmox.com/pbs-devel/20261111204853.531985-2-tobias@stuttgart.de/
proxmox:
Jakob-Tobias Winter (2):
pbs-api-types: add types and schemas for snapshot attributes
pbs-api-types: tape: add attribute filter to backup jobs
pbs-api-types/src/datastore.rs | 22 ++-
pbs-api-types/src/jobs.rs | 98 ++++++++++-
pbs-api-types/src/lib.rs | 3 +
pbs-api-types/tests/attribute_filter_tests.rs | 155 ++++++++++++++++++
4 files changed, 276 insertions(+), 2 deletions(-)
create mode 100644 pbs-api-types/tests/attribute_filter_tests.rs
proxmox-backup:
Jakob-Tobias Winter (5):
datastore: add attributes to backup snapshots
fix #7452: api: datastore: add endpoints for snapshot attributes
tape: backup job: allow selecting groups by snapshot attributes
ui: tape backup job: add attribute filter
docs: describe snapshot attributes and the tape job attribute filter
docs/pve-integration.rst | 23 ++++++
docs/tape-backup.rst | 27 ++++++
docs/terminology.rst | 33 ++++++++
pbs-datastore/src/backup_info.rs | 18 +++-
pbs-datastore/src/datastore.rs | 15 +++-
pbs-datastore/src/manifest.rs | 87 +++++++++++++++++++-
src/api2/admin/datastore.rs | 128 ++++++++++++++++++++++++++++-
src/api2/config/tape_backup_job.rs | 8 ++
src/api2/tape/backup.rs | 19 ++++-
src/bin/proxmox-tape.rs | 11 ++-
www/tape/BackupJobs.js | 7 ++
www/tape/window/TapeBackupJob.js | 27 +++++-
12 files changed, 391 insertions(+), 12 deletions(-)
storage:
Jakob-Tobias Winter (1):
pbs: add helper to set the attributes of a backup snapshot
src/PVE/Storage/PBSPlugin.pm | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
guest-common:
Jakob-Tobias Winter (1):
vzdump: add pbs-snapshot-attributes option
src/PVE/VZDump/Common.pm | 8 ++++++++
1 file changed, 8 insertions(+)
manager:
Jakob-Tobias Winter (2):
vzdump: store guest tags and pool as snapshot attributes on PBS
ui: backup job: add selector for PBS snapshot attributes
PVE/VZDump.pm | 42 +++++++++++++++++++++
configs/vzdump.conf | 1 +
www/manager6/panel/BackupAdvancedOptions.js | 22 +++++++++++
3 files changed, 65 insertions(+)
docs:
Jakob-Tobias Winter (1):
vzdump: describe snapshot attributes on Proxmox Backup Server
vzdump.adoc | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
Summary over all repositories:
22 files changed, 799 insertions(+), 14 deletions(-)
--
2.53.0
next reply other threads:[~2026-11-13 9:07 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 8:56 Jakob-Tobias Winter [this message]
2026-11-13 8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:54 ` Samuel Rufinatscha
2026-11-13 8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:55 ` Samuel Rufinatscha
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06 ` Samuel Rufinatscha
2026-11-18 15:19 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07 ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha
2026-11-19 11:58 ` Jakob-Tobias Winter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261113085637.57360-3-tobias@stuttgart.de \
--to=tobias@stuttgart.de \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox