all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: Jakob-Tobias Winter <tobias@stuttgart.de>, pbs-devel@lists.proxmox.com
Subject: Re: [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots
Date: Mon, 16 Nov 2026 15:57:58 +0100	[thread overview]
Message-ID: <b460a85f-858f-4072-8850-b9c2b2897694@proxmox.com> (raw)
In-Reply-To: <20261113085637.57360-6-tobias@stuttgart.de>

On 11/13/26 10:06 AM, Jakob-Tobias Winter wrote:
> A backup server knows little about where a backup comes from. Things
> like the tags and the resource pool of a Proxmox VE guest are only
> known to the client, but they are what users want to select backups
> by, for example in tape backup jobs.
> 
> Allow storing such information as attributes of a snapshot. They are
> kept in the unprotected part of the manifest, as suggested in [0].
> That way they can also be set for encrypted backups, and they stay
> with the snapshot when it is synced or written to tape.
> 
> Each attribute is a key with a list of values. A guest with several
> tags gets one list element for each tag, instead of one string that
> has to be split again.
> 
> A backup group has the attributes of its newest snapshot which has
> attributes. Snapshots without attributes are skipped on purpose: a
> client which does not set them, or fails to set them once, must not
> make a group lose its attributes.
> 
> The unprotected part of the manifest can be uploaded by the client,
> so values with an invalid format are dropped when reading it.
> 
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32
> 
> Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
> ---
> The patches for proxmox-backup need the pbs-api-types of patches 1 and
> 2, so the dependency has to be bumped once they are released.
> 
>   pbs-datastore/src/backup_info.rs | 18 ++++++-
>   pbs-datastore/src/datastore.rs   | 15 +++++-
>   pbs-datastore/src/manifest.rs    | 87 +++++++++++++++++++++++++++++++-
>   3 files changed, 117 insertions(+), 3 deletions(-)
> 
> diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
> index cce2b699f..ea9dc9a2a 100644
> --- a/pbs-datastore/src/backup_info.rs
> +++ b/pbs-datastore/src/backup_info.rs
> @@ -17,7 +17,7 @@ use proxmox_systemd::escape_unit;
>   
>   use pbs_api_types::{
>       ArchiveType, Authid, BACKUP_DATE_REGEX, BackupArchiveName, BackupGroupDeleteStats,
> -    BackupNamespace, BackupType, GroupFilter, MANIFEST_BLOB_NAME, VerifyState,
> +    BackupNamespace, BackupType, GroupFilter, MANIFEST_BLOB_NAME, SnapshotAttributes, VerifyState,
>   };
>   use pbs_config::{BackupLockGuard, open_backup_lockfile};
>   
> @@ -148,6 +148,22 @@ impl BackupGroup {
>               .max_by_key(|item| item.backup_dir.backup_time()))
>       }
>   
> +    /// Returns the attributes of the backup group
> +    ///
> +    /// These are the attributes of the newest finished snapshot which has some stored. Snapshots
> +    /// without stored attributes are skipped, so a group keeps its attributes if a client did not
> +    /// provide them for newer snapshots.
> +    pub fn attributes(&self) -> Result<SnapshotAttributes, Error> {
> +        let mut backups = self.list_backups()?;
> +        BackupInfo::sort_list(&mut backups, false);
> +
> +        Ok(backups
> +            .iter()
> +            .filter(|info| info.is_finished())
> +            .find_map(|info| info.backup_dir.load_manifest().ok()?.0.attributes())
> +            .unwrap_or_default())
> +    }

while I see the intention here, I think this is not ideal: What if you 
have a malicious client which writes a new backup to the group with some 
bogus/unrelated attributes? That would lead for the group to now present 
completely unrelated attributes, potentially excluding/including it once 
filters are being applied to it.

It might make more sense to do the filtering on the snapshot level only, 
although that comes with its own downsides.

> +
>       pub fn last_successful_backup(&self) -> Result<Option<i64>, Error> {
>           let mut last = None;
>   
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index e6766a3b9..5b094a095 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -34,7 +34,7 @@ use pbs_api_types::{
>       ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder,
>       DEFAULT_SNAPSHOT_TIME_SKEW_SECS, DataStoreConfig, DatastoreBackendConfig, DatastoreBackendType,
>       GarbageCollectionCacheStats, GarbageCollectionStatus, MAX_NAMESPACE_DEPTH, MaintenanceMode,
> -    MaintenanceType, Operation, S3Statistics, UPID,
> +    MaintenanceType, Operation, S3Statistics, SnapshotAttributes, UPID,
>   };
>   use pbs_config::s3::S3_CFG_TYPE_ID;
>   use pbs_config::{BackupLockGuard, ConfigVersionCache};
> @@ -3458,6 +3458,19 @@ impl DataStore {
>           Ok(())
>       }
>   
> +    /// Set the attributes for the given snapshot, replacing the existing ones.
> +    pub fn set_attributes(
> +        self: &Arc<Self>,
> +        attributes: SnapshotAttributes,
> +        snapshot: BackupDir,
> +    ) -> Result<(), Error> {
> +        let backend = self.backend().context("failed to get backend")?;
> +        snapshot
> +            .update_manifest(&backend, |manifest| manifest.set_attributes(&attributes))
> +            .map_err(|err| format_err!("unable to update manifest blob - {err}"))?;
> +        Ok(())
> +    }
> +
>       /// Renames a corrupt chunk, returning the new path if the chunk was renamed successfully.
>       /// Returns with `Ok(None)` if the chunk source was not found.
>       pub fn rename_corrupt_chunk(&self, digest: &[u8; 32]) -> Result<Option<PathBuf>, Error> {
> diff --git a/pbs-datastore/src/manifest.rs b/pbs-datastore/src/manifest.rs
> index 16f9668f2..61982d64a 100644
> --- a/pbs-datastore/src/manifest.rs
> +++ b/pbs-datastore/src/manifest.rs
> @@ -9,7 +9,7 @@ use proxmox_sys::fs::{CreateOptions, replace_file};
>   
>   use pbs_api_types::{
>       BackupArchiveName, BackupType, CLIENT_LOG_BLOB_NAME, CryptMode, Fingerprint,
> -    MANIFEST_BLOB_NAME, SnapshotVerifyState,
> +    MANIFEST_BLOB_NAME, SNAPSHOT_ATTRIBUTE_REGEX, SnapshotAttributes, SnapshotVerifyState,
>   };
>   use pbs_tools::crypt_config::CryptConfig;
>   
> @@ -273,6 +273,27 @@ impl BackupManifest {
>           Ok(Some(serde_json::from_value::<SnapshotVerifyState>(verify)?))
>       }
>   
> +    /// Get the attributes of the snapshot
> +    ///
> +    /// Returns `None` if no attributes were stored for the snapshot. The unprotected part of the
> +    /// manifest can be provided by the client, so values with an invalid format are dropped and
> +    /// attributes which are not a list of values for each key count as not stored.
> +    pub fn attributes(&self) -> Option<SnapshotAttributes> {
> +        let mut attributes =
> +            SnapshotAttributes::deserialize(&self.unprotected["attributes"]).ok()?;
> +
> +        for (key, values) in attributes.iter_mut() {
> +            values.retain(|value| SNAPSHOT_ATTRIBUTE_REGEX.is_match(&format!("{key}={value}")));
> +        }
> +
> +        Some(attributes)
> +    }
> +
> +    /// Set the attributes of the snapshot, replacing the existing ones
> +    pub fn set_attributes(&mut self, attributes: &SnapshotAttributes) {
> +        self.unprotected["attributes"] = json!(attributes);
> +    }
> +
>       /// Set the sync-source-signature used to detect snapshot changes across encrypted/decrypted
>       /// sync flows.
>       ///
> @@ -597,3 +618,67 @@ fn test_manifest_eq_ignoring_verify_state() -> Result<(), Error> {
>   
>       Ok(())
>   }
> +
> +#[test]
> +fn test_manifest_attributes() -> Result<(), Error> {
> +    let crypt_config = CryptConfig::new([7; 32])?;
> +    let mut manifest = BackupManifest::new("vm/100/2020-06-26T13:56:05Z".parse()?);
> +    manifest.add_file(
> +        &"drive-scsi0.img.fidx".try_into()?,
> +        200,
> +        [1; 32],
> +        CryptMode::Encrypt,
> +    )?;
> +
> +    // nothing stored yet
> +    assert!(manifest.attributes().is_none());
> +
> +    let signature = manifest.signature(&crypt_config)?;
> +
> +    let mut attributes = SnapshotAttributes::new();
> +    for (key, value) in [
> +        ("pve-tag", "web"),
> +        ("pve-tag", "prod"),
> +        ("pve-pool", "customers/a"),
> +    ] {
> +        attributes
> +            .entry(key.to_string())
> +            .or_default()
> +            .insert(value.to_string());
> +    }
> +    manifest.set_attributes(&attributes);
> +
> +    // not covered by the signature, so attributes can be set for encrypted backups
> +    assert_eq!(manifest.signature(&crypt_config)?, signature);
> +
> +    // each value is a list element of its own, in a stable order
> +    assert_eq!(
> +        manifest.unprotected["attributes"],
> +        json!({ "pve-pool": ["customers/a"], "pve-tag": ["prod", "web"] }),
> +    );
> +
> +    let data = manifest.to_string(Some(&crypt_config))?;
> +    let mut manifest = BackupManifest::from_data(data.as_bytes(), Some(&crypt_config), false)?;
> +    assert_eq!(manifest.attributes(), Some(attributes));
> +
> +    // stored, but empty
> +    manifest.set_attributes(&SnapshotAttributes::new());
> +    assert_eq!(manifest.attributes(), Some(SnapshotAttributes::new()));
> +
> +    // data which does not come from the server
> +    manifest.unprotected["attributes"] = json!("pve-tag=prod");
> +    assert!(manifest.attributes().is_none());
> +
> +    manifest.unprotected["attributes"] = json!({ "pve-tag": "prod;web" });
> +    assert!(manifest.attributes().is_none());
> +
> +    manifest.unprotected["attributes"] = json!({
> +        "pve-tag": ["prod", "with space", "web;db"],
> +        "Invalid Key": ["prod"],
> +    });
> +    let attributes = manifest.attributes().unwrap();
> +    assert_eq!(attributes["pve-tag"].iter().collect::<Vec<_>>(), ["prod"]);
> +    assert!(attributes["Invalid Key"].is_empty());
> +
> +    Ok(())
> +}
> 
> base-commit: a7eb4e8a1477df960570fa72e9e7e2b8ac58bdab





  reply	other threads:[~2026-11-16 14:58 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13  8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-18 12:54   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner [this message]
2026-11-18 12:55   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06   ` Samuel Rufinatscha
2026-11-18 15:19     ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07   ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha
2026-11-19 11:58   ` Jakob-Tobias Winter

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=b460a85f-858f-4072-8850-b9c2b2897694@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    --cc=tobias@stuttgart.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal