From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 0FB9B1FF0B7 for ; Fri, 13 Nov 2026 10:07:38 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 9EA4F2188C; Fri, 13 Nov 2026 10:06:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stuttgart.de; s=s1022; t=1794560285; bh=jQ/InOxEb605+y4306UCJLd1lPZ7850KUYH2vuOX+RE=; h=From:To:Cc:Subject:Date:From; b=GBUR74yA5rHuQZYVl523SweIqRjivKmyL/tN080Zk7tqHbL2/Zn9PBxiPeLfnwOdi itdCL6B25rS89o7P4cJLhEHwQxBki6RAzeqYX/vGWahUSL3XJcAANI3NFiieKrOdjg /laloIfR74X3gLHfR6RRwZwfc4op/737hVw+neJp/4a63f2T6l9QHl9kKezDE4qHQm P6uxrnMgtPPdo1hNFtIKKpkC+QrN3bNhDHhrio9oIe27OV1OhRWJmArjIqM5lKlCE5 TGfvWr14RzB3dJtC27HzSQZLgJNytqetRfj4hqoKRO8t9ehM+RoV00sL4jF7nYAFrq u6XAJkRz5/9gA== From: Jakob-Tobias Winter To: pbs-devel@lists.proxmox.com Subject: [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Date: Fri, 13 Nov 2026 09:56:27 +0100 Message-ID: <20261113085637.57360-3-tobias@stuttgart.de> X-Mailer: git-send-email 2.53.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.128 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: C4UDY6Y7ZWA2JX6Y3C5EQREO7AV36476 X-Message-ID-Hash: C4UDY6Y7ZWA2JX6Y3C5EQREO7AV36476 X-MailFrom: prvs=d19b210678=tobias@stuttgart.de X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Jakob-Tobias Winter X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Tape backup jobs on a Proxmox Backup Server select backup groups by type and ID only. Splitting guests over media pools by what they are, say production and test guests with different retention, means keeping lists of IDs up to date by hand in every job. In Proxmox VE the same guests are already organized with tags and resource pools, but the backup server knows about neither. This series lets Proxmox VE store the tags and the resource pool of a guest with each backup, as attributes of the snapshot, and adds a filter on such attributes to tape backup jobs: # proxmox-tape backup-job create prod --store store1 --drive lto1 \ --pool prod --attribute-filter pve-tag=prod # proxmox-tape backup-job create rest --store store1 --drive lto1 \ --pool rest --attribute-filter exclude:pve-tag=prod It is an RFC, as it defines a format and names that others would have to live with, and because [1] is assigned already. If there is work in progress for it, I am happy to adapt this series to it or to drop whatever overlaps. Proxmox Backup Server --------------------- proxmox 1-2, proxmox-backup 3-7 Snapshots get attributes, a map from a key to a list of values. They are kept in the unprotected part of the manifest, as suggested in [0], so they can be set for encrypted backups without touching the signature, and they stay with a snapshot that is synced or written to tape. A guest with several tags gets one value per tag, nothing is joined into one string. They are read and set through a new endpoint next to the notes, GET/PUT /admin/datastore/{store}/attributes, with the privileges of the notes, as a list of KEY=VALUE entries. A PUT replaces all attributes of the snapshot. An empty set is stored as well, to tell a guest without tags from a client which never set anything. Tape backup jobs get the option 'attribute-filter', a list of [exclude:|include:]KEY=VALUE with the include and exclude rules of group filters, applied after the group filter. A group has the attributes of its newest finished snapshot that has some, so a single backup without attributes does not move a group to another media pool. Besides the API there is the option for 'proxmox-tape', a field in the job window, a column in the job list and documentation. The server does not interpret keys or values. Proxmox VE ---------- storage 8, guest-common 9, manager 10-11, docs 12 After a backup to a PBS storage, vzdump sets 'pve-tag' (one value per tag) and 'pve-pool', with the prefix suggested in [0], directly through the API like the protection flag. A guest in a nested pool gets that pool and each of its parent pools as values, so a filter for a pool also matches the guests of the pools nested in it. A server without the endpoint answers 404, which is only noted in the task log. As the attributes are not encrypted, the new vzdump option 'pbs-snapshot-attributes' (default on) can turn this off for a job, a single call, or a node in /etc/vzdump.conf. Compatibility ------------- * Jobs without the new option select the same groups as before, and nothing changes on tape or in the catalogs. A tape job written by 4.2.9 was read and run unchanged. * The other way round, a version without the series cannot parse a tape-job.cfg in which the option is used (unknown property), like with other new job properties. * A backup server without the series leaves the attributes in the manifest alone (tried with 4.2.9). * A PVE node without the series ignores the vzdump option in jobs.cfg with a warning. Open questions -------------- * The names: 'pve-tag', 'pve-pool', 'attribute-filter' and 'pbs-snapshot-attributes'. * A guest in the pool 'a/b' gets the 'pve-pool' values 'a/b' and 'a', so one filter covers a pool with everything nested in it. A backup job for the pool 'a' in Proxmox VE only takes the direct members of 'a'. Is that difference fine, or should only 'a/b' be stored? Since nested pools are quite new I wasn't sure about that at all. * Setting the attributes after the backup, like notes and protection, needs no change in the backup protocol or in the QEMU library. But a user that may only append, as in [2], could then not set them. Passing them when the backup is created would solve that, at the price of changes in proxmox-backup-qemu, pve-qemu, qemu-server and pve-container which I found a bit excessive for some attributes. * On by default with the option to opt out, or the other way round? Is the switch needed at all? * This is a separate job option and not a new kind of group filter, as group filters are shared with sync jobs and match on the name of a group only. Using attributes in other jobs and showing them in the web interface could follow, if wanted. * For a group in which no snapshot has attributes, all its manifests are read when a job with a filter runs. Is that acceptable, or should only the newest few be looked at? Testing ------- proxmox-backup master (4.2.11) with the series: rustfmt, clippy without new warnings, every commit builds, 'cargo test --workspace' passes (162 tests, one of them new), the docs build as HTML and man pages. The tests of pbs-api-types pass, with five new ones for the filter. pbs-api-types 1.0.22 came from the proxmox repository, as it is not in the devel repository yet. The PVE patches are perltidy and biome clean, and the test suites of the three Perl repositories give the same results with and without them. End to end in an unprivileged sandbox with the real daemons of both products: the packages of PVE 9.2.26 with the PVE patches applied and a backup server built from master with the series. There is no KVM in it, so the guests were stopped VMs and a container. * vzdump of VMs and a container to a plain and an encrypted PBS storage, the latter with a namespace, as a user with DatastoreBackup only; several tags, pools nested up to three levels, changed and removed tags; the option per call, per job (also run by pvescheduler and from the web interface) and in /etc/vzdump.conf; restore of an encrypted backup * the same against 4.2.9 and plain master: noted in the task log, no warning, the backup is fine * the API as owner, as another user, as auditor and as admin; invalid values; verification, notes, protection and a local sync job keep the attributes * tape jobs on the virtual tape drive (needs [3]) selecting by tag, by pool and by the parent of a nested pool, combined with a group filter and with namespaces, started by hand and by the scheduler; a restore from tape keeps the attributes * both web interfaces in a browser Not yet tested: real tape hardware, clusters with more than one node, datastores with an S3 backend. I wasn't sure if I was supposed to send to pbs-devel and pve-devel. [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32 [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7452 [2] https://lore.proxmox.com/pbs-devel/20261106172235.890542-1-c.ebner@proxmox.com/ [3] https://lore.proxmox.com/pbs-devel/20261111204853.531985-2-tobias@stuttgart.de/ proxmox: Jakob-Tobias Winter (2): pbs-api-types: add types and schemas for snapshot attributes pbs-api-types: tape: add attribute filter to backup jobs pbs-api-types/src/datastore.rs | 22 ++- pbs-api-types/src/jobs.rs | 98 ++++++++++- pbs-api-types/src/lib.rs | 3 + pbs-api-types/tests/attribute_filter_tests.rs | 155 ++++++++++++++++++ 4 files changed, 276 insertions(+), 2 deletions(-) create mode 100644 pbs-api-types/tests/attribute_filter_tests.rs proxmox-backup: Jakob-Tobias Winter (5): datastore: add attributes to backup snapshots fix #7452: api: datastore: add endpoints for snapshot attributes tape: backup job: allow selecting groups by snapshot attributes ui: tape backup job: add attribute filter docs: describe snapshot attributes and the tape job attribute filter docs/pve-integration.rst | 23 ++++++ docs/tape-backup.rst | 27 ++++++ docs/terminology.rst | 33 ++++++++ pbs-datastore/src/backup_info.rs | 18 +++- pbs-datastore/src/datastore.rs | 15 +++- pbs-datastore/src/manifest.rs | 87 +++++++++++++++++++- src/api2/admin/datastore.rs | 128 ++++++++++++++++++++++++++++- src/api2/config/tape_backup_job.rs | 8 ++ src/api2/tape/backup.rs | 19 ++++- src/bin/proxmox-tape.rs | 11 ++- www/tape/BackupJobs.js | 7 ++ www/tape/window/TapeBackupJob.js | 27 +++++- 12 files changed, 391 insertions(+), 12 deletions(-) storage: Jakob-Tobias Winter (1): pbs: add helper to set the attributes of a backup snapshot src/PVE/Storage/PBSPlugin.pm | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) guest-common: Jakob-Tobias Winter (1): vzdump: add pbs-snapshot-attributes option src/PVE/VZDump/Common.pm | 8 ++++++++ 1 file changed, 8 insertions(+) manager: Jakob-Tobias Winter (2): vzdump: store guest tags and pool as snapshot attributes on PBS ui: backup job: add selector for PBS snapshot attributes PVE/VZDump.pm | 42 +++++++++++++++++++++ configs/vzdump.conf | 1 + www/manager6/panel/BackupAdvancedOptions.js | 22 +++++++++++ 3 files changed, 65 insertions(+) docs: Jakob-Tobias Winter (1): vzdump: describe snapshot attributes on Proxmox Backup Server vzdump.adoc | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) Summary over all repositories: 22 files changed, 799 insertions(+), 14 deletions(-) -- 2.53.0