public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Jakob-Tobias Winter <tobias@stuttgart.de>
To: pbs-devel@lists.proxmox.com
Subject: Re: [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter
Date: Thu, 19 Nov 2026 12:58:16 +0100	[thread overview]
Message-ID: <559fc611-2d42-40c0-ac46-fab2866e501b@stuttgart.de> (raw)
In-Reply-To: <751f5842-78cc-49cd-9e4e-8a4950e46d8f@proxmox.com>

Thanks to both for your feedback, I will be able to invest some more 
time next week.


On 11/18/26 15:22, Samuel Rufinatscha wrote:
> Thanks for the series @Jakob and thanks for looping me in @Christian.
> 
> The overall approach looks good to me too.
> I left a few additional inline comments on the individual patches.
> 
> I agree with Christian’s suggestion to supply the attributes when the
> backup session starts and check each snapshot’s own attributes when
> filtering.
> 
> Could we also show the stored attributes when inspecting a snapshot
> under Datastore > Content? That would make it easier to see which keys
> and values to use in a job filter
> 
> On 11/13/26 10:07 AM, Jakob-Tobias Winter wrote:
>> Tape backup jobs on a Proxmox Backup Server select backup groups by
>> type and ID only. Splitting guests over media pools by what they are,
>> say production and test guests with different retention, means keeping
>> lists of IDs up to date by hand in every job. In Proxmox VE the same
>> guests are already organized with tags and resource pools, but the
>> backup server knows about neither.
>>
>> This series lets Proxmox VE store the tags and the resource pool of a
>> guest with each backup, as attributes of the snapshot, and adds a
>> filter on such attributes to tape backup jobs:
>>
>>    # proxmox-tape backup-job create prod --store store1 --drive lto1 \
>>        --pool prod --attribute-filter pve-tag=prod
>>    # proxmox-tape backup-job create rest --store store1 --drive lto1 \
>>        --pool rest --attribute-filter exclude:pve-tag=prod
>>
>> It is an RFC, as it defines a format and names that others would have
>> to live with, and because [1] is assigned already. If there is work in
>> progress for it, I am happy to adapt this series to it or to drop
>> whatever overlaps.
>>
>> Proxmox Backup Server
>> ---------------------
>>
>> proxmox 1-2, proxmox-backup 3-7
>>
>> Snapshots get attributes, a map from a key to a list of values. They
>> are kept in the unprotected part of the manifest, as suggested in [0],
>> so they can be set for encrypted backups without touching the
>> signature, and they stay with a snapshot that is synced or written to
>> tape. A guest with several tags gets one value per tag, nothing is
>> joined into one string.
>>
>> They are read and set through a new endpoint next to the notes,
>> GET/PUT /admin/datastore/{store}/attributes, with the privileges of
>> the notes, as a list of KEY=VALUE entries. A PUT replaces all
>> attributes of the snapshot. An empty set is stored as well, to tell a
>> guest without tags from a client which never set anything.
>>
>> Tape backup jobs get the option 'attribute-filter', a list of
>> [exclude:|include:]KEY=VALUE with the include and exclude rules of
>> group filters, applied after the group filter. A group has the
>> attributes of its newest finished snapshot that has some, so a single
>> backup without attributes does not move a group to another media pool.
>> Besides the API there is the option for 'proxmox-tape', a field in the
>> job window, a column in the job list and documentation.
>>
>> The server does not interpret keys or values.
>>
>> Proxmox VE
>> ----------
>>
>> storage 8, guest-common 9, manager 10-11, docs 12
>>
>> After a backup to a PBS storage, vzdump sets 'pve-tag' (one value per
>> tag) and 'pve-pool', with the prefix suggested in [0], directly
>> through the API like the protection flag. A guest in a nested pool
>> gets that pool and each of its parent pools as values, so a filter
>> for a pool also matches the guests of the pools nested in it. A
>> server without the endpoint answers 404, which is only noted in the
>> task log.
>>
>> As the attributes are not encrypted, the new vzdump option
>> 'pbs-snapshot-attributes' (default on) can turn this off for a job, a
>> single call, or a node in /etc/vzdump.conf.
>>
>> Compatibility
>> -------------
>>
>> * Jobs without the new option select the same groups as before, and
>>    nothing changes on tape or in the catalogs. A tape job written by
>>    4.2.9 was read and run unchanged.
>> * The other way round, a version without the series cannot parse a
>>    tape-job.cfg in which the option is used (unknown property), like
>>    with other new job properties.
>> * A backup server without the series leaves the attributes in the
>>    manifest alone (tried with 4.2.9).
>> * A PVE node without the series ignores the vzdump option in jobs.cfg
>>    with a warning.
>>
>> Open questions
>> --------------
>>
>> * The names: 'pve-tag', 'pve-pool', 'attribute-filter' and
>>    'pbs-snapshot-attributes'.
>> * A guest in the pool 'a/b' gets the 'pve-pool' values 'a/b' and 'a',
>>    so one filter covers a pool with everything nested in it. A backup
>>    job for the pool 'a' in Proxmox VE only takes the direct members of
>>    'a'. Is that difference fine, or should only 'a/b' be stored?
>>    Since nested pools are quite new I wasn't sure about that at all.
>> * Setting the attributes after the backup, like notes and protection,
>>    needs no change in the backup protocol or in the QEMU library. But
>>    a user that may only append, as in [2], could then not set them.
>>    Passing them when the backup is created would solve that, at the
>>    price of changes in proxmox-backup-qemu, pve-qemu, qemu-server and
>>    pve-container which I found a bit excessive for some attributes.
>> * On by default with the option to opt out, or the other way round?
>>    Is the switch needed at all?
>> * This is a separate job option and not a new kind of group filter,
>>    as group filters are shared with sync jobs and match on the name of
>>    a group only. Using attributes in other jobs and showing them in the
>>    web interface could follow, if wanted.
>> * For a group in which no snapshot has attributes, all its manifests
>>    are read when a job with a filter runs. Is that acceptable, or
>>    should only the newest few be looked at?
>>
>> Testing
>> -------
>>
>> proxmox-backup master (4.2.11) with the series: rustfmt, clippy
>> without new warnings, every commit builds, 'cargo test --workspace'
>> passes (162 tests, one of them new), the docs build as HTML and man
>> pages. The tests of pbs-api-types pass, with five new ones for the
>> filter. pbs-api-types 1.0.22 came from the proxmox repository, as it
>> is not in the devel repository yet.
>>
>> The PVE patches are perltidy and biome clean, and the test suites of
>> the three Perl repositories give the same results with and without
>> them.
>>
>> End to end in an unprivileged sandbox with the real daemons of both
>> products: the packages of PVE 9.2.26 with the PVE patches applied and
>> a backup server built from master with the series. There is no KVM in
>> it, so the guests were stopped VMs and a container.
>>
>> * vzdump of VMs and a container to a plain and an encrypted PBS
>>    storage, the latter with a namespace, as a user with DatastoreBackup
>>    only; several tags, pools nested up to three levels, changed and
>>    removed tags; the option per call, per job (also run by pvescheduler
>>    and from the web interface) and in /etc/vzdump.conf; restore of an
>>    encrypted backup
>> * the same against 4.2.9 and plain master: noted in the task log, no
>>    warning, the backup is fine
>> * the API as owner, as another user, as auditor and as admin; invalid
>>    values; verification, notes, protection and a local sync job keep
>>    the attributes
>> * tape jobs on the virtual tape drive (needs [3]) selecting by tag, by
>>    pool and by the parent of a nested pool, combined with a group
>>    filter and with namespaces, started by hand and by the scheduler; a
>>    restore from tape keeps the attributes
>> * both web interfaces in a browser
>>
>> Not yet tested: real tape hardware, clusters with more than one node,
>> datastores with an S3 backend.
>>
>> I wasn't sure if I was supposed to send to pbs-devel and pve-devel.
>>
>> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32
>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7452
>> [2] https://lore.proxmox.com/pbs-devel/20261106172235.890542-1- 
>> c.ebner@proxmox.com/
>> [3] https://lore.proxmox.com/pbs-devel/20261111204853.531985-2- 
>> tobias@stuttgart.de/
>>
>>
>> proxmox:
>>
>> Jakob-Tobias Winter (2):
>>    pbs-api-types: add types and schemas for snapshot attributes
>>    pbs-api-types: tape: add attribute filter to backup jobs
>>
>>   pbs-api-types/src/datastore.rs                |  22 ++-
>>   pbs-api-types/src/jobs.rs                     |  98 ++++++++++-
>>   pbs-api-types/src/lib.rs                      |   3 +
>>   pbs-api-types/tests/attribute_filter_tests.rs | 155 ++++++++++++++++++
>>   4 files changed, 276 insertions(+), 2 deletions(-)
>>   create mode 100644 pbs-api-types/tests/attribute_filter_tests.rs
>>
>>
>> proxmox-backup:
>>
>> Jakob-Tobias Winter (5):
>>    datastore: add attributes to backup snapshots
>>    fix #7452: api: datastore: add endpoints for snapshot attributes
>>    tape: backup job: allow selecting groups by snapshot attributes
>>    ui: tape backup job: add attribute filter
>>    docs: describe snapshot attributes and the tape job attribute filter
>>
>>   docs/pve-integration.rst           |  23 ++++++
>>   docs/tape-backup.rst               |  27 ++++++
>>   docs/terminology.rst               |  33 ++++++++
>>   pbs-datastore/src/backup_info.rs   |  18 +++-
>>   pbs-datastore/src/datastore.rs     |  15 +++-
>>   pbs-datastore/src/manifest.rs      |  87 +++++++++++++++++++-
>>   src/api2/admin/datastore.rs        | 128 ++++++++++++++++++++++++++++-
>>   src/api2/config/tape_backup_job.rs |   8 ++
>>   src/api2/tape/backup.rs            |  19 ++++-
>>   src/bin/proxmox-tape.rs            |  11 ++-
>>   www/tape/BackupJobs.js             |   7 ++
>>   www/tape/window/TapeBackupJob.js   |  27 +++++-
>>   12 files changed, 391 insertions(+), 12 deletions(-)
>>
>>
>> storage:
>>
>> Jakob-Tobias Winter (1):
>>    pbs: add helper to set the attributes of a backup snapshot
>>
>>   src/PVE/Storage/PBSPlugin.pm | 23 +++++++++++++++++++++++
>>   1 file changed, 23 insertions(+)
>>
>>
>> guest-common:
>>
>> Jakob-Tobias Winter (1):
>>    vzdump: add pbs-snapshot-attributes option
>>
>>   src/PVE/VZDump/Common.pm | 8 ++++++++
>>   1 file changed, 8 insertions(+)
>>
>>
>> manager:
>>
>> Jakob-Tobias Winter (2):
>>    vzdump: store guest tags and pool as snapshot attributes on PBS
>>    ui: backup job: add selector for PBS snapshot attributes
>>
>>   PVE/VZDump.pm                               | 42 +++++++++++++++++++++
>>   configs/vzdump.conf                         |  1 +
>>   www/manager6/panel/BackupAdvancedOptions.js | 22 +++++++++++
>>   3 files changed, 65 insertions(+)
>>
>>
>> docs:
>>
>> Jakob-Tobias Winter (1):
>>    vzdump: describe snapshot attributes on Proxmox Backup Server
>>
>>   vzdump.adoc | 36 ++++++++++++++++++++++++++++++++++++
>>   1 file changed, 36 insertions(+)
>>
>>
>> Summary over all repositories:
>>    22 files changed, 799 insertions(+), 14 deletions(-)
>>
> 




      reply	other threads:[~2026-11-19 12:07 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13  8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-18 12:54   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-18 12:55   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06   ` Samuel Rufinatscha
2026-11-18 15:19     ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07   ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha
2026-11-19 11:58   ` Jakob-Tobias Winter [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=559fc611-2d42-40c0-ac46-fab2866e501b@stuttgart.de \
    --to=tobias@stuttgart.de \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal