public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: Jakob-Tobias Winter <tobias@stuttgart.de>, pbs-devel@lists.proxmox.com
Subject: Re: [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots
Date: Mon, 16 Nov 2026 15:57:58 +0100	[thread overview]
Message-ID: <b460a85f-858f-4072-8850-b9c2b2897694@proxmox.com> (raw)
In-Reply-To: <20261113085637.57360-6-tobias@stuttgart.de>

On 11/13/26 10:06 AM, Jakob-Tobias Winter wrote:
> A backup server knows little about where a backup comes from. Things
> like the tags and the resource pool of a Proxmox VE guest are only
> known to the client, but they are what users want to select backups
> by, for example in tape backup jobs.
> 
> Allow storing such information as attributes of a snapshot. They are
> kept in the unprotected part of the manifest, as suggested in [0].
> That way they can also be set for encrypted backups, and they stay
> with the snapshot when it is synced or written to tape.
> 
> Each attribute is a key with a list of values. A guest with several
> tags gets one list element for each tag, instead of one string that
> has to be split again.
> 
> A backup group has the attributes of its newest snapshot which has
> attributes. Snapshots without attributes are skipped on purpose: a
> client which does not set them, or fails to set them once, must not
> make a group lose its attributes.
> 
> The unprotected part of the manifest can be uploaded by the client,
> so values with an invalid format are dropped when reading it.
> 
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4186#c32
> 
> Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
> ---
> The patches for proxmox-backup need the pbs-api-types of patches 1 and
> 2, so the dependency has to be bumped once they are released.
> 
>   pbs-datastore/src/backup_info.rs | 18 ++++++-
>   pbs-datastore/src/datastore.rs   | 15 +++++-
>   pbs-datastore/src/manifest.rs    | 87 +++++++++++++++++++++++++++++++-
>   3 files changed, 117 insertions(+), 3 deletions(-)
> 
> diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
> index cce2b699f..ea9dc9a2a 100644
> --- a/pbs-datastore/src/backup_info.rs
> +++ b/pbs-datastore/src/backup_info.rs
> @@ -17,7 +17,7 @@ use proxmox_systemd::escape_unit;
>   
>   use pbs_api_types::{
>       ArchiveType, Authid, BACKUP_DATE_REGEX, BackupArchiveName, BackupGroupDeleteStats,
> -    BackupNamespace, BackupType, GroupFilter, MANIFEST_BLOB_NAME, VerifyState,
> +    BackupNamespace, BackupType, GroupFilter, MANIFEST_BLOB_NAME, SnapshotAttributes, VerifyState,
>   };
>   use pbs_config::{BackupLockGuard, open_backup_lockfile};
>   
> @@ -148,6 +148,22 @@ impl BackupGroup {
>               .max_by_key(|item| item.backup_dir.backup_time()))
>       }
>   
> +    /// Returns the attributes of the backup group
> +    ///
> +    /// These are the attributes of the newest finished snapshot which has some stored. Snapshots
> +    /// without stored attributes are skipped, so a group keeps its attributes if a client did not
> +    /// provide them for newer snapshots.
> +    pub fn attributes(&self) -> Result<SnapshotAttributes, Error> {
> +        let mut backups = self.list_backups()?;
> +        BackupInfo::sort_list(&mut backups, false);
> +
> +        Ok(backups
> +            .iter()
> +            .filter(|info| info.is_finished())
> +            .find_map(|info| info.backup_dir.load_manifest().ok()?.0.attributes())
> +            .unwrap_or_default())
> +    }

while I see the intention here, I think this is not ideal: What if you 
have a malicious client which writes a new backup to the group with some 
bogus/unrelated attributes? That would lead for the group to now present 
completely unrelated attributes, potentially excluding/including it once 
filters are being applied to it.

It might make more sense to do the filtering on the snapshot level only, 
although that comes with its own downsides.

> +
>       pub fn last_successful_backup(&self) -> Result<Option<i64>, Error> {
>           let mut last = None;
>   
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index e6766a3b9..5b094a095 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -34,7 +34,7 @@ use pbs_api_types::{
>       ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder,
>       DEFAULT_SNAPSHOT_TIME_SKEW_SECS, DataStoreConfig, DatastoreBackendConfig, DatastoreBackendType,
>       GarbageCollectionCacheStats, GarbageCollectionStatus, MAX_NAMESPACE_DEPTH, MaintenanceMode,
> -    MaintenanceType, Operation, S3Statistics, UPID,
> +    MaintenanceType, Operation, S3Statistics, SnapshotAttributes, UPID,
>   };
>   use pbs_config::s3::S3_CFG_TYPE_ID;
>   use pbs_config::{BackupLockGuard, ConfigVersionCache};
> @@ -3458,6 +3458,19 @@ impl DataStore {
>           Ok(())
>       }
>   
> +    /// Set the attributes for the given snapshot, replacing the existing ones.
> +    pub fn set_attributes(
> +        self: &Arc<Self>,
> +        attributes: SnapshotAttributes,
> +        snapshot: BackupDir,
> +    ) -> Result<(), Error> {
> +        let backend = self.backend().context("failed to get backend")?;
> +        snapshot
> +            .update_manifest(&backend, |manifest| manifest.set_attributes(&attributes))
> +            .map_err(|err| format_err!("unable to update manifest blob - {err}"))?;
> +        Ok(())
> +    }
> +
>       /// Renames a corrupt chunk, returning the new path if the chunk was renamed successfully.
>       /// Returns with `Ok(None)` if the chunk source was not found.
>       pub fn rename_corrupt_chunk(&self, digest: &[u8; 32]) -> Result<Option<PathBuf>, Error> {
> diff --git a/pbs-datastore/src/manifest.rs b/pbs-datastore/src/manifest.rs
> index 16f9668f2..61982d64a 100644
> --- a/pbs-datastore/src/manifest.rs
> +++ b/pbs-datastore/src/manifest.rs
> @@ -9,7 +9,7 @@ use proxmox_sys::fs::{CreateOptions, replace_file};
>   
>   use pbs_api_types::{
>       BackupArchiveName, BackupType, CLIENT_LOG_BLOB_NAME, CryptMode, Fingerprint,
> -    MANIFEST_BLOB_NAME, SnapshotVerifyState,
> +    MANIFEST_BLOB_NAME, SNAPSHOT_ATTRIBUTE_REGEX, SnapshotAttributes, SnapshotVerifyState,
>   };
>   use pbs_tools::crypt_config::CryptConfig;
>   
> @@ -273,6 +273,27 @@ impl BackupManifest {
>           Ok(Some(serde_json::from_value::<SnapshotVerifyState>(verify)?))
>       }
>   
> +    /// Get the attributes of the snapshot
> +    ///
> +    /// Returns `None` if no attributes were stored for the snapshot. The unprotected part of the
> +    /// manifest can be provided by the client, so values with an invalid format are dropped and
> +    /// attributes which are not a list of values for each key count as not stored.
> +    pub fn attributes(&self) -> Option<SnapshotAttributes> {
> +        let mut attributes =
> +            SnapshotAttributes::deserialize(&self.unprotected["attributes"]).ok()?;
> +
> +        for (key, values) in attributes.iter_mut() {
> +            values.retain(|value| SNAPSHOT_ATTRIBUTE_REGEX.is_match(&format!("{key}={value}")));
> +        }
> +
> +        Some(attributes)
> +    }
> +
> +    /// Set the attributes of the snapshot, replacing the existing ones
> +    pub fn set_attributes(&mut self, attributes: &SnapshotAttributes) {
> +        self.unprotected["attributes"] = json!(attributes);
> +    }
> +
>       /// Set the sync-source-signature used to detect snapshot changes across encrypted/decrypted
>       /// sync flows.
>       ///
> @@ -597,3 +618,67 @@ fn test_manifest_eq_ignoring_verify_state() -> Result<(), Error> {
>   
>       Ok(())
>   }
> +
> +#[test]
> +fn test_manifest_attributes() -> Result<(), Error> {
> +    let crypt_config = CryptConfig::new([7; 32])?;
> +    let mut manifest = BackupManifest::new("vm/100/2020-06-26T13:56:05Z".parse()?);
> +    manifest.add_file(
> +        &"drive-scsi0.img.fidx".try_into()?,
> +        200,
> +        [1; 32],
> +        CryptMode::Encrypt,
> +    )?;
> +
> +    // nothing stored yet
> +    assert!(manifest.attributes().is_none());
> +
> +    let signature = manifest.signature(&crypt_config)?;
> +
> +    let mut attributes = SnapshotAttributes::new();
> +    for (key, value) in [
> +        ("pve-tag", "web"),
> +        ("pve-tag", "prod"),
> +        ("pve-pool", "customers/a"),
> +    ] {
> +        attributes
> +            .entry(key.to_string())
> +            .or_default()
> +            .insert(value.to_string());
> +    }
> +    manifest.set_attributes(&attributes);
> +
> +    // not covered by the signature, so attributes can be set for encrypted backups
> +    assert_eq!(manifest.signature(&crypt_config)?, signature);
> +
> +    // each value is a list element of its own, in a stable order
> +    assert_eq!(
> +        manifest.unprotected["attributes"],
> +        json!({ "pve-pool": ["customers/a"], "pve-tag": ["prod", "web"] }),
> +    );
> +
> +    let data = manifest.to_string(Some(&crypt_config))?;
> +    let mut manifest = BackupManifest::from_data(data.as_bytes(), Some(&crypt_config), false)?;
> +    assert_eq!(manifest.attributes(), Some(attributes));
> +
> +    // stored, but empty
> +    manifest.set_attributes(&SnapshotAttributes::new());
> +    assert_eq!(manifest.attributes(), Some(SnapshotAttributes::new()));
> +
> +    // data which does not come from the server
> +    manifest.unprotected["attributes"] = json!("pve-tag=prod");
> +    assert!(manifest.attributes().is_none());
> +
> +    manifest.unprotected["attributes"] = json!({ "pve-tag": "prod;web" });
> +    assert!(manifest.attributes().is_none());
> +
> +    manifest.unprotected["attributes"] = json!({
> +        "pve-tag": ["prod", "with space", "web;db"],
> +        "Invalid Key": ["prod"],
> +    });
> +    let attributes = manifest.attributes().unwrap();
> +    assert_eq!(attributes["pve-tag"].iter().collect::<Vec<_>>(), ["prod"]);
> +    assert!(attributes["Invalid Key"].is_empty());
> +
> +    Ok(())
> +}
> 
> base-commit: a7eb4e8a1477df960570fa72e9e7e2b8ac58bdab





  reply	other threads:[~2026-11-16 14:58 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13  8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-18 12:54   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner [this message]
2026-11-18 12:55   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06   ` Samuel Rufinatscha
2026-11-18 15:19     ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07   ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha
2026-11-19 11:58   ` Jakob-Tobias Winter

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=b460a85f-858f-4072-8850-b9c2b2897694@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    --cc=tobias@stuttgart.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal