From: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
To: Jakob-Tobias Winter <tobias@stuttgart.de>, pbs-devel@lists.proxmox.com
Subject: Re: [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots
Date: Wed, 18 Nov 2026 13:55:10 +0100 [thread overview]
Message-ID: <96bdde16-b17f-4b14-90cf-6830fdec5641@proxmox.com> (raw)
In-Reply-To: <20261113085637.57360-6-tobias@stuttgart.de>
On 11/13/26 10:06 AM, Jakob-Tobias Winter wrote:
> A backup server knows little about where a backup comes from. Things
> like the tags and the resource pool of a Proxmox VE guest are only
> known to the client, but they are what users want to select backups
> by, for example in tape backup jobs.
>
> Allow storing such information as attributes of a snapshot. They are
> kept in the unprotected part of the manifest, as suggested in [0].
> That way they can also be set for encrypted backups, and they stay
> with the snapshot when it is synced or written to tape.
>
> Each attribute is a key with a list of values. A guest with several
> tags gets one list element for each tag, instead of one string that
> has to be split again.
>
> A backup group has the attributes of its newest snapshot which has
> attributes. Snapshots without attributes are skipped on purpose: a
> client which does not set them, or fails to set them once, must not
> make a group lose its attributes.
>
> The unprotected part of the manifest can be uploaded by the client,
> so values with an invalid format are dropped when reading it.
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32
>
> Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
> ---
> The patches for proxmox-backup need the pbs-api-types of patches 1 and
> 2, so the dependency has to be bumped once they are released.
>
> pbs-datastore/src/backup_info.rs | 18 ++++++-
> pbs-datastore/src/datastore.rs | 15 +++++-
> pbs-datastore/src/manifest.rs | 87 +++++++++++++++++++++++++++++++-
> 3 files changed, 117 insertions(+), 3 deletions(-)
>
> diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
> index cce2b699f..ea9dc9a2a 100644
> --- a/pbs-datastore/src/backup_info.rs
> +++ b/pbs-datastore/src/backup_info.rs
> @@ -17,7 +17,7 @@ use proxmox_systemd::escape_unit;
>
> use pbs_api_types::{
> ArchiveType, Authid, BACKUP_DATE_REGEX, BackupArchiveName, BackupGroupDeleteStats,
> - BackupNamespace, BackupType, GroupFilter, MANIFEST_BLOB_NAME, VerifyState,
> + BackupNamespace, BackupType, GroupFilter, MANIFEST_BLOB_NAME, SnapshotAttributes, VerifyState,
> };
> use pbs_config::{BackupLockGuard, open_backup_lockfile};
>
> @@ -148,6 +148,22 @@ impl BackupGroup {
> .max_by_key(|item| item.backup_dir.backup_time()))
> }
>
> + /// Returns the attributes of the backup group
> + ///
> + /// These are the attributes of the newest finished snapshot which has some stored. Snapshots
> + /// without stored attributes are skipped, so a group keeps its attributes if a client did not
> + /// provide them for newer snapshots.
> + pub fn attributes(&self) -> Result<SnapshotAttributes, Error> {
> + let mut backups = self.list_backups()?;
> + BackupInfo::sort_list(&mut backups, false);
> +
> + Ok(backups
> + .iter()
> + .filter(|info| info.is_finished())
> + .find_map(|info| info.backup_dir.load_manifest().ok()?.0.attributes())
this ok() would silently skip an unreadable/corrupt manifest and
fallback to an older snapshot. i think it would make sense to
propagate it instead
> + .unwrap_or_default())
> + }
> +
> pub fn last_successful_backup(&self) -> Result<Option<i64>, Error> {
> let mut last = None;
>
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index e6766a3b9..5b094a095 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -34,7 +34,7 @@ use pbs_api_types::{
> ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder,
> DEFAULT_SNAPSHOT_TIME_SKEW_SECS, DataStoreConfig, DatastoreBackendConfig, DatastoreBackendType,
> GarbageCollectionCacheStats, GarbageCollectionStatus, MAX_NAMESPACE_DEPTH, MaintenanceMode,
> - MaintenanceType, Operation, S3Statistics, UPID,
> + MaintenanceType, Operation, S3Statistics, SnapshotAttributes, UPID,
> };
> use pbs_config::s3::S3_CFG_TYPE_ID;
> use pbs_config::{BackupLockGuard, ConfigVersionCache};
> @@ -3458,6 +3458,19 @@ impl DataStore {
> Ok(())
> }
>
> + /// Set the attributes for the given snapshot, replacing the existing ones.
> + pub fn set_attributes(
> + self: &Arc<Self>,
> + attributes: SnapshotAttributes,
> + snapshot: BackupDir,
> + ) -> Result<(), Error> {
> + let backend = self.backend().context("failed to get backend")?;
> + snapshot
> + .update_manifest(&backend, |manifest| manifest.set_attributes(&attributes))
> + .map_err(|err| format_err!("unable to update manifest blob - {err}"))?;
> + Ok(())
> + }
> +
> /// Renames a corrupt chunk, returning the new path if the chunk was renamed successfully.
> /// Returns with `Ok(None)` if the chunk source was not found.
> pub fn rename_corrupt_chunk(&self, digest: &[u8; 32]) -> Result<Option<PathBuf>, Error> {
> diff --git a/pbs-datastore/src/manifest.rs b/pbs-datastore/src/manifest.rs
> index 16f9668f2..61982d64a 100644
> --- a/pbs-datastore/src/manifest.rs
> +++ b/pbs-datastore/src/manifest.rs
> @@ -9,7 +9,7 @@ use proxmox_sys::fs::{CreateOptions, replace_file};
>
> use pbs_api_types::{
> BackupArchiveName, BackupType, CLIENT_LOG_BLOB_NAME, CryptMode, Fingerprint,
> - MANIFEST_BLOB_NAME, SnapshotVerifyState,
> + MANIFEST_BLOB_NAME, SNAPSHOT_ATTRIBUTE_REGEX, SnapshotAttributes, SnapshotVerifyState,
> };
> use pbs_tools::crypt_config::CryptConfig;
>
> @@ -273,6 +273,27 @@ impl BackupManifest {
> Ok(Some(serde_json::from_value::<SnapshotVerifyState>(verify)?))
> }
>
> + /// Get the attributes of the snapshot
> + ///
> + /// Returns `None` if no attributes were stored for the snapshot. The unprotected part of the
> + /// manifest can be provided by the client, so values with an invalid format are dropped and
> + /// attributes which are not a list of values for each key count as not stored.
> + pub fn attributes(&self) -> Option<SnapshotAttributes> {
> + let mut attributes =
> + SnapshotAttributes::deserialize(&self.unprotected["attributes"]).ok()?;
> +
> + for (key, values) in attributes.iter_mut() {
> + values.retain(|value| SNAPSHOT_ATTRIBUTE_REGEX.is_match(&format!("{key}={value}")));
> + }
> +
> + Some(attributes)
> + }
> +
> + /// Set the attributes of the snapshot, replacing the existing ones
> + pub fn set_attributes(&mut self, attributes: &SnapshotAttributes) {
> + self.unprotected["attributes"] = json!(attributes);
> + }
> +
> /// Set the sync-source-signature used to detect snapshot changes across encrypted/decrypted
> /// sync flows.
> ///
> @@ -597,3 +618,67 @@ fn test_manifest_eq_ignoring_verify_state() -> Result<(), Error> {
>
> Ok(())
> }
> +
> +#[test]
> +fn test_manifest_attributes() -> Result<(), Error> {
> + let crypt_config = CryptConfig::new([7; 32])?;
> + let mut manifest = BackupManifest::new("vm/100/2020-06-26T13:56:05Z".parse()?);
> + manifest.add_file(
> + &"drive-scsi0.img.fidx".try_into()?,
> + 200,
> + [1; 32],
> + CryptMode::Encrypt,
> + )?;
> +
> + // nothing stored yet
> + assert!(manifest.attributes().is_none());
> +
> + let signature = manifest.signature(&crypt_config)?;
> +
> + let mut attributes = SnapshotAttributes::new();
> + for (key, value) in [
> + ("pve-tag", "web"),
> + ("pve-tag", "prod"),
> + ("pve-pool", "customers/a"),
> + ] {
> + attributes
> + .entry(key.to_string())
> + .or_default()
> + .insert(value.to_string());
> + }
> + manifest.set_attributes(&attributes);
> +
> + // not covered by the signature, so attributes can be set for encrypted backups
> + assert_eq!(manifest.signature(&crypt_config)?, signature);
> +
> + // each value is a list element of its own, in a stable order
> + assert_eq!(
> + manifest.unprotected["attributes"],
> + json!({ "pve-pool": ["customers/a"], "pve-tag": ["prod", "web"] }),
> + );
> +
> + let data = manifest.to_string(Some(&crypt_config))?;
> + let mut manifest = BackupManifest::from_data(data.as_bytes(), Some(&crypt_config), false)?;
> + assert_eq!(manifest.attributes(), Some(attributes));
> +
> + // stored, but empty
> + manifest.set_attributes(&SnapshotAttributes::new());
> + assert_eq!(manifest.attributes(), Some(SnapshotAttributes::new()));
> +
> + // data which does not come from the server
> + manifest.unprotected["attributes"] = json!("pve-tag=prod");
> + assert!(manifest.attributes().is_none());
> +
> + manifest.unprotected["attributes"] = json!({ "pve-tag": "prod;web" });
> + assert!(manifest.attributes().is_none());
> +
> + manifest.unprotected["attributes"] = json!({
> + "pve-tag": ["prod", "with space", "web;db"],
> + "Invalid Key": ["prod"],
> + });
> + let attributes = manifest.attributes().unwrap();
> + assert_eq!(attributes["pve-tag"].iter().collect::<Vec<_>>(), ["prod"]);
> + assert!(attributes["Invalid Key"].is_empty());
> +
> + Ok(())
> +}
>
> base-commit: a7eb4e8a1477df960570fa72e9e7e2b8ac58bdab
next prev parent reply other threads:[~2026-11-18 12:55 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:54 ` Samuel Rufinatscha
2026-11-13 8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:55 ` Samuel Rufinatscha [this message]
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06 ` Samuel Rufinatscha
2026-11-18 15:19 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07 ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha
2026-11-19 11:58 ` Jakob-Tobias Winter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=96bdde16-b17f-4b14-90cf-6830fdec5641@proxmox.com \
--to=s.rufinatscha@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
--cc=tobias@stuttgart.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.