public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Jakob-Tobias Winter <tobias@stuttgart.de>
To: pbs-devel@lists.proxmox.com
Cc: Jakob-Tobias Winter <tobias@stuttgart.de>
Subject: [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots
Date: Fri, 13 Nov 2026 09:56:30 +0100	[thread overview]
Message-ID: <20261113085637.57360-6-tobias@stuttgart.de> (raw)
In-Reply-To: <20261113085637.57360-3-tobias@stuttgart.de>

A backup server knows little about where a backup comes from. Things
like the tags and the resource pool of a Proxmox VE guest are only
known to the client, but they are what users want to select backups
by, for example in tape backup jobs.

Allow storing such information as attributes of a snapshot. They are
kept in the unprotected part of the manifest, as suggested in [0].
That way they can also be set for encrypted backups, and they stay
with the snapshot when it is synced or written to tape.

Each attribute is a key with a list of values. A guest with several
tags gets one list element for each tag, instead of one string that
has to be split again.

A backup group has the attributes of its newest snapshot which has
attributes. Snapshots without attributes are skipped on purpose: a
client which does not set them, or fails to set them once, must not
make a group lose its attributes.

The unprotected part of the manifest can be uploaded by the client,
so values with an invalid format are dropped when reading it.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32

Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
---
The patches for proxmox-backup need the pbs-api-types of patches 1 and
2, so the dependency has to be bumped once they are released.

 pbs-datastore/src/backup_info.rs | 18 ++++++-
 pbs-datastore/src/datastore.rs   | 15 +++++-
 pbs-datastore/src/manifest.rs    | 87 +++++++++++++++++++++++++++++++-
 3 files changed, 117 insertions(+), 3 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index cce2b699f..ea9dc9a2a 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -17,7 +17,7 @@ use proxmox_systemd::escape_unit;
 
 use pbs_api_types::{
     ArchiveType, Authid, BACKUP_DATE_REGEX, BackupArchiveName, BackupGroupDeleteStats,
-    BackupNamespace, BackupType, GroupFilter, MANIFEST_BLOB_NAME, VerifyState,
+    BackupNamespace, BackupType, GroupFilter, MANIFEST_BLOB_NAME, SnapshotAttributes, VerifyState,
 };
 use pbs_config::{BackupLockGuard, open_backup_lockfile};
 
@@ -148,6 +148,22 @@ impl BackupGroup {
             .max_by_key(|item| item.backup_dir.backup_time()))
     }
 
+    /// Returns the attributes of the backup group
+    ///
+    /// These are the attributes of the newest finished snapshot which has some stored. Snapshots
+    /// without stored attributes are skipped, so a group keeps its attributes if a client did not
+    /// provide them for newer snapshots.
+    pub fn attributes(&self) -> Result<SnapshotAttributes, Error> {
+        let mut backups = self.list_backups()?;
+        BackupInfo::sort_list(&mut backups, false);
+
+        Ok(backups
+            .iter()
+            .filter(|info| info.is_finished())
+            .find_map(|info| info.backup_dir.load_manifest().ok()?.0.attributes())
+            .unwrap_or_default())
+    }
+
     pub fn last_successful_backup(&self) -> Result<Option<i64>, Error> {
         let mut last = None;
 
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index e6766a3b9..5b094a095 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -34,7 +34,7 @@ use pbs_api_types::{
     ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder,
     DEFAULT_SNAPSHOT_TIME_SKEW_SECS, DataStoreConfig, DatastoreBackendConfig, DatastoreBackendType,
     GarbageCollectionCacheStats, GarbageCollectionStatus, MAX_NAMESPACE_DEPTH, MaintenanceMode,
-    MaintenanceType, Operation, S3Statistics, UPID,
+    MaintenanceType, Operation, S3Statistics, SnapshotAttributes, UPID,
 };
 use pbs_config::s3::S3_CFG_TYPE_ID;
 use pbs_config::{BackupLockGuard, ConfigVersionCache};
@@ -3458,6 +3458,19 @@ impl DataStore {
         Ok(())
     }
 
+    /// Set the attributes for the given snapshot, replacing the existing ones.
+    pub fn set_attributes(
+        self: &Arc<Self>,
+        attributes: SnapshotAttributes,
+        snapshot: BackupDir,
+    ) -> Result<(), Error> {
+        let backend = self.backend().context("failed to get backend")?;
+        snapshot
+            .update_manifest(&backend, |manifest| manifest.set_attributes(&attributes))
+            .map_err(|err| format_err!("unable to update manifest blob - {err}"))?;
+        Ok(())
+    }
+
     /// Renames a corrupt chunk, returning the new path if the chunk was renamed successfully.
     /// Returns with `Ok(None)` if the chunk source was not found.
     pub fn rename_corrupt_chunk(&self, digest: &[u8; 32]) -> Result<Option<PathBuf>, Error> {
diff --git a/pbs-datastore/src/manifest.rs b/pbs-datastore/src/manifest.rs
index 16f9668f2..61982d64a 100644
--- a/pbs-datastore/src/manifest.rs
+++ b/pbs-datastore/src/manifest.rs
@@ -9,7 +9,7 @@ use proxmox_sys::fs::{CreateOptions, replace_file};
 
 use pbs_api_types::{
     BackupArchiveName, BackupType, CLIENT_LOG_BLOB_NAME, CryptMode, Fingerprint,
-    MANIFEST_BLOB_NAME, SnapshotVerifyState,
+    MANIFEST_BLOB_NAME, SNAPSHOT_ATTRIBUTE_REGEX, SnapshotAttributes, SnapshotVerifyState,
 };
 use pbs_tools::crypt_config::CryptConfig;
 
@@ -273,6 +273,27 @@ impl BackupManifest {
         Ok(Some(serde_json::from_value::<SnapshotVerifyState>(verify)?))
     }
 
+    /// Get the attributes of the snapshot
+    ///
+    /// Returns `None` if no attributes were stored for the snapshot. The unprotected part of the
+    /// manifest can be provided by the client, so values with an invalid format are dropped and
+    /// attributes which are not a list of values for each key count as not stored.
+    pub fn attributes(&self) -> Option<SnapshotAttributes> {
+        let mut attributes =
+            SnapshotAttributes::deserialize(&self.unprotected["attributes"]).ok()?;
+
+        for (key, values) in attributes.iter_mut() {
+            values.retain(|value| SNAPSHOT_ATTRIBUTE_REGEX.is_match(&format!("{key}={value}")));
+        }
+
+        Some(attributes)
+    }
+
+    /// Set the attributes of the snapshot, replacing the existing ones
+    pub fn set_attributes(&mut self, attributes: &SnapshotAttributes) {
+        self.unprotected["attributes"] = json!(attributes);
+    }
+
     /// Set the sync-source-signature used to detect snapshot changes across encrypted/decrypted
     /// sync flows.
     ///
@@ -597,3 +618,67 @@ fn test_manifest_eq_ignoring_verify_state() -> Result<(), Error> {
 
     Ok(())
 }
+
+#[test]
+fn test_manifest_attributes() -> Result<(), Error> {
+    let crypt_config = CryptConfig::new([7; 32])?;
+    let mut manifest = BackupManifest::new("vm/100/2020-06-26T13:56:05Z".parse()?);
+    manifest.add_file(
+        &"drive-scsi0.img.fidx".try_into()?,
+        200,
+        [1; 32],
+        CryptMode::Encrypt,
+    )?;
+
+    // nothing stored yet
+    assert!(manifest.attributes().is_none());
+
+    let signature = manifest.signature(&crypt_config)?;
+
+    let mut attributes = SnapshotAttributes::new();
+    for (key, value) in [
+        ("pve-tag", "web"),
+        ("pve-tag", "prod"),
+        ("pve-pool", "customers/a"),
+    ] {
+        attributes
+            .entry(key.to_string())
+            .or_default()
+            .insert(value.to_string());
+    }
+    manifest.set_attributes(&attributes);
+
+    // not covered by the signature, so attributes can be set for encrypted backups
+    assert_eq!(manifest.signature(&crypt_config)?, signature);
+
+    // each value is a list element of its own, in a stable order
+    assert_eq!(
+        manifest.unprotected["attributes"],
+        json!({ "pve-pool": ["customers/a"], "pve-tag": ["prod", "web"] }),
+    );
+
+    let data = manifest.to_string(Some(&crypt_config))?;
+    let mut manifest = BackupManifest::from_data(data.as_bytes(), Some(&crypt_config), false)?;
+    assert_eq!(manifest.attributes(), Some(attributes));
+
+    // stored, but empty
+    manifest.set_attributes(&SnapshotAttributes::new());
+    assert_eq!(manifest.attributes(), Some(SnapshotAttributes::new()));
+
+    // data which does not come from the server
+    manifest.unprotected["attributes"] = json!("pve-tag=prod");
+    assert!(manifest.attributes().is_none());
+
+    manifest.unprotected["attributes"] = json!({ "pve-tag": "prod;web" });
+    assert!(manifest.attributes().is_none());
+
+    manifest.unprotected["attributes"] = json!({
+        "pve-tag": ["prod", "with space", "web;db"],
+        "Invalid Key": ["prod"],
+    });
+    let attributes = manifest.attributes().unwrap();
+    assert_eq!(attributes["pve-tag"].iter().collect::<Vec<_>>(), ["prod"]);
+    assert!(attributes["Invalid Key"].is_empty());
+
+    Ok(())
+}

base-commit: a7eb4e8a1477df960570fa72e9e7e2b8ac58bdab
-- 
2.53.0




  parent reply	other threads:[~2026-11-13  9:06 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13  8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-18 12:54   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` Jakob-Tobias Winter [this message]
2026-11-16 14:57   ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Christian Ebner
2026-11-18 12:55   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06   ` Samuel Rufinatscha
2026-11-18 15:19     ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07   ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha
2026-11-19 11:58   ` Jakob-Tobias Winter

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261113085637.57360-6-tobias@stuttgart.de \
    --to=tobias@stuttgart.de \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal