public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: Jakob-Tobias Winter <tobias@stuttgart.de>, pbs-devel@lists.proxmox.com
Subject: Re: [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs
Date: Mon, 16 Nov 2026 15:57:50 +0100	[thread overview]
Message-ID: <7802620f-4d59-41d7-8807-a8cfc34fe007@proxmox.com> (raw)
In-Reply-To: <20261113085637.57360-5-tobias@stuttgart.de>

On 11/13/26 10:06 AM, Jakob-Tobias Winter wrote:
> Tape backup jobs get a list of attribute filters to select backup
> groups by the attributes of their snapshots, for example to write
> the guests tagged as production and as test to different media
> pools. A filter only matches a whole value of its key. Several
> filters are combined like group filters: one of the include filters
> has to match, if there are any, and none of the exclude filters.
> 
> The job option is optional. Existing job configurations stay valid
> and select the same groups as before.
> 
> Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
> ---
>   pbs-api-types/src/jobs.rs                     |  98 ++++++++++-
>   pbs-api-types/tests/attribute_filter_tests.rs | 155 ++++++++++++++++++
>   2 files changed, 252 insertions(+), 1 deletion(-)
>   create mode 100644 pbs-api-types/tests/attribute_filter_tests.rs
> 
> diff --git a/pbs-api-types/src/jobs.rs b/pbs-api-types/src/jobs.rs
> index 9a5d2b77..8e558e4a 100644
> --- a/pbs-api-types/src/jobs.rs
> +++ b/pbs-api-types/src/jobs.rs
> @@ -15,7 +15,7 @@ use crate::{
>       BackupType, CRYPT_KEY_ID_SCHEMA, DATASTORE_SCHEMA, DRIVE_NAME_SCHEMA, MEDIA_POOL_NAME_SCHEMA,
>       NS_MAX_DEPTH_REDUCED_SCHEMA, NotificationMode, PROXMOX_SAFE_ID_FORMAT,
>       PROXMOX_SAFE_ID_REGEX_STR, REMOTE_ID_SCHEMA, RateLimitConfig, SINGLE_LINE_COMMENT_SCHEMA,
> -    Userid,
> +    SNAPSHOT_ATTRIBUTE_REGEX, SnapshotAttributes, Userid,
>   };
>   
>   const_regex! {
> @@ -345,6 +345,10 @@ pub struct VerificationJobStatus {
>               schema: GROUP_FILTER_LIST_SCHEMA,
>               optional: true,
>           },
> +        "attribute-filter": {
> +            schema: ATTRIBUTE_FILTER_LIST_SCHEMA,
> +            optional: true,
> +        },
>           ns: {
>               type: BackupNamespace,
>               optional: true,
> @@ -379,6 +383,8 @@ pub struct TapeBackupJobSetup {
>       pub notification_mode: Option<NotificationMode>,
>       #[serde(skip_serializing_if = "Option::is_none")]
>       pub group_filter: Option<Vec<GroupFilter>>,
> +    #[serde(skip_serializing_if = "Option::is_none")]
> +    pub attribute_filter: Option<Vec<AttributeFilter>>,
>       #[serde(skip_serializing_if = "Option::is_none", default)]
>       pub ns: Option<BackupNamespace>,
>       #[serde(skip_serializing_if = "Option::is_none", default)]
> @@ -550,6 +556,96 @@ pub const GROUP_FILTER_SCHEMA: Schema = StringSchema::new(
>   pub const GROUP_FILTER_LIST_SCHEMA: Schema =
>       ArraySchema::new("List of group filters.", &GROUP_FILTER_SCHEMA).schema();
>   
> +#[derive(Clone, Debug, PartialEq, Eq)]
> +/// Filter for matching the attributes of a snapshot.
> +pub struct AttributeFilter {
> +    pub is_exclude: bool,
> +    pub key: String,

as mentioned on patch 0001, key ...

> +    pub value: String,

... and value should preferably have dedicated types ...

> +}
> +
> +impl AttributeFilter {
> +    /// Returns whether the value of the filter is one of the values the attributes have for its
> +    /// key.
> +    pub fn matches(&self, attributes: &SnapshotAttributes) -> bool {
> +        attributes
> +            .get(&self.key)
> +            .is_some_and(|values| values.contains(&self.value))
> +    }
> +
> +    /// Returns whether the attributes pass the filters.
> +    ///
> +    /// If there are include filters, at least one of them has to match. No exclude filter may
> +    /// match.
> +    pub fn apply_filters(filters: &[AttributeFilter], attributes: &SnapshotAttributes) -> bool {
> +        let is_included = if filters.iter().filter(|f| !f.is_exclude).count() == 0 {
> +            true
> +        } else {
> +            filters
> +                .iter()
> +                .filter(|f| !f.is_exclude)
> +                .any(|filter| filter.matches(attributes))
> +        };
> +
> +        is_included
> +            && !filters
> +                .iter()
> +                .filter(|f| f.is_exclude)
> +                .any(|filter| filter.matches(attributes))
> +    }
> +}
> +
> +impl std::str::FromStr for AttributeFilter {
> +    type Err = anyhow::Error;
> +
> +    fn from_str(s: &str) -> Result<Self, Self::Err> {
> +        let (is_exclude, attribute) = match s.split_once(':') {
> +            Some(("include", value)) => (false, value),
> +            Some(("exclude", value)) => (true, value),
> +            _ => (false, s),
> +        };
> +
> +        match attribute.split_once('=') {
> +            Some((key, value)) if SNAPSHOT_ATTRIBUTE_REGEX.is_match(attribute) => {

... this can then use the dedicate types for parsing

> +                Ok(AttributeFilter {
> +                    is_exclude,
> +                    key: key.to_string(),
> +                    value: value.to_string(),
> +                })
> +            }
> +            _ => bail!("input doesn't match expected format '[<exclude:|include:>]KEY=VALUE'"),
> +        }
> +    }
> +}
> +
> +// used for serializing below, caution!
> +impl std::fmt::Display for AttributeFilter {
> +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
> +        if self.is_exclude {
> +            f.write_str("exclude:")?;
> +        }
> +        write!(f, "{}={}", self.key, self.value)
> +    }
> +}
> +
> +proxmox_serde::forward_deserialize_to_from_str!(AttributeFilter);
> +proxmox_serde::forward_serialize_to_display!(AttributeFilter);
> +
> +fn verify_attribute_filter(input: &str) -> Result<(), anyhow::Error> {
> +    AttributeFilter::from_str(input).map(|_| ())
> +}
> +
> +pub const ATTRIBUTE_FILTER_SCHEMA: Schema = StringSchema::new(
> +    "Attribute filter, matches if the given value is one of the values of the key. Can be \
> +    inverted by prepending 'exclude:'.",
> +)
> +.format(&ApiStringFormat::VerifyFn(verify_attribute_filter))
> +.type_text("[<exclude:|include:>]KEY=VALUE")
> +.schema();
> +
> +pub const ATTRIBUTE_FILTER_LIST_SCHEMA: Schema =
> +    ArraySchema::new("List of attribute filters.", &ATTRIBUTE_FILTER_SCHEMA).schema();
> +
>   pub const TRANSFER_LAST_SCHEMA: Schema =
>       IntegerSchema::new("Limit transfer to last N snapshots (per group), skipping others")
>           .minimum(1)
> diff --git a/pbs-api-types/tests/attribute_filter_tests.rs b/pbs-api-types/tests/attribute_filter_tests.rs
> new file mode 100644
> index 00000000..c355e6b4
> --- /dev/null
> +++ b/pbs-api-types/tests/attribute_filter_tests.rs
> @@ -0,0 +1,155 @@
> +use pbs_api_types::{AttributeFilter, SnapshotAttributes};
> +use std::str::FromStr;
> +
> +fn attributes(list: &[(&str, &[&str])]) -> SnapshotAttributes {
> +    list.iter()
> +        .map(|(key, values)| {
> +            (
> +                key.to_string(),
> +                values.iter().map(|value| value.to_string()).collect(),
> +            )
> +        })
> +        .collect()
> +}
> +
> +fn filters(list: &[&str]) -> Vec<AttributeFilter> {
> +    list.iter()
> +        .map(|filter| AttributeFilter::from_str(filter).unwrap())
> +        .collect()
> +}
> +
> +#[test]
> +fn test_parse_filter() {
> +    let filter = AttributeFilter::from_str("pve-tag=prod").unwrap();
> +    assert!(!filter.is_exclude);
> +    assert_eq!(filter.key, "pve-tag");
> +    assert_eq!(filter.value, "prod");
> +    assert_eq!(filter.to_string(), "pve-tag=prod");
> +
> +    let filter = AttributeFilter::from_str("include:pve-tag=prod").unwrap();
> +    assert!(!filter.is_exclude);
> +    assert_eq!(filter.to_string(), "pve-tag=prod");
> +
> +    let filter = AttributeFilter::from_str("exclude:pve-pool=customers/a_1.test").unwrap();
> +    assert!(filter.is_exclude);
> +    assert_eq!(filter.key, "pve-pool");
> +    assert_eq!(filter.value, "customers/a_1.test");
> +    assert_eq!(filter.to_string(), "exclude:pve-pool=customers/a_1.test");
> +
> +    let invalid = [
> +        "",
> +        "pve-tag",
> +        "pve-tag=",
> +        "=prod",
> +        "Pve-Tag=prod",
> +        "1tag=prod",
> +        "pve-tag=prod;web",
> +        "pve-tag=prod,web",
> +        "pve-tag=prod web",
> +        "pve-tag=prod=web",
> +        "pve-tag=prod:web",
> +        "other:pve-tag=prod",
> +        "exclude:exclude:pve-tag=prod",
> +    ];
> +
> +    for filter in invalid {
> +        assert!(
> +            AttributeFilter::from_str(filter).is_err(),
> +            "'{filter}' must not parse"
> +        );
> +    }
> +}
> +
> +#[test]
> +fn test_no_filters() {
> +    let filters = [];
> +
> +    assert!(AttributeFilter::apply_filters(&filters, &attributes(&[])));
> +    assert!(AttributeFilter::apply_filters(
> +        &filters,
> +        &attributes(&[("pve-tag", &["prod", "web"])])
> +    ));
> +}
> +
> +#[test]
> +fn test_include_filters() {
> +    let filters = filters(&["pve-tag=prod", "pve-pool=customer1"]);
> +
> +    let do_backup = [
> +        attributes(&[("pve-tag", &["prod"])]),
> +        attributes(&[("pve-tag", &["db", "prod", "web"])]),
> +        attributes(&[("pve-pool", &["customer1"])]),
> +        attributes(&[("pve-pool", &["customer1"]), ("pve-tag", &["test"])]),
> +    ];
> +
> +    let dont_backup = [
> +        attributes(&[]),
> +        attributes(&[("pve-tag", &[])]),
> +        attributes(&[("pve-tag", &["nonprod", "prod-eu", "pro", "Prod"])]),
> +        attributes(&[("pve-pool", &["prod"])]),
> +        attributes(&[("pve-pool", &["customer1/sub", "customer10"])]),
> +        attributes(&[("other", &["prod", "customer1"])]),
> +    ];
> +
> +    for attributes in do_backup {
> +        assert!(AttributeFilter::apply_filters(&filters, &attributes));
> +    }
> +
> +    for attributes in dont_backup {
> +        assert!(!AttributeFilter::apply_filters(&filters, &attributes));
> +    }
> +}
> +
> +#[test]
> +fn test_exclude_filters() {
> +    let filters = filters(&["exclude:pve-tag=prod", "exclude:pve-tag=test"]);
> +
> +    let do_backup = [
> +        attributes(&[]),
> +        attributes(&[("pve-tag", &["web"])]),
> +        attributes(&[("pve-pool", &["prod"])]),
> +    ];
> +
> +    let dont_backup = [
> +        attributes(&[("pve-tag", &["prod"])]),
> +        attributes(&[("pve-tag", &["test", "web"])]),
> +        attributes(&[("pve-tag", &["prod", "test"])]),
> +    ];
> +
> +    for attributes in do_backup {
> +        assert!(AttributeFilter::apply_filters(&filters, &attributes));
> +    }
> +
> +    for attributes in dont_backup {
> +        assert!(!AttributeFilter::apply_filters(&filters, &attributes));
> +    }
> +}
> +
> +#[test]
> +fn test_include_and_exclude_filters() {
> +    let filters = filters(&[
> +        "exclude:pve-tag=test",
> +        "pve-pool=customer1",
> +        "pve-pool=customer2",
> +    ]);
> +
> +    let do_backup = [
> +        attributes(&[("pve-pool", &["customer1"])]),
> +        attributes(&[("pve-pool", &["customer2"]), ("pve-tag", &["prod"])]),
> +    ];
> +
> +    let dont_backup = [
> +        attributes(&[]),
> +        attributes(&[("pve-tag", &["prod"])]),
> +        attributes(&[("pve-pool", &["customer3"])]),
> +        attributes(&[("pve-pool", &["customer1"]), ("pve-tag", &["test"])]),
> +    ];
> +
> +    for attributes in do_backup {
> +        assert!(AttributeFilter::apply_filters(&filters, &attributes));
> +    }
> +
> +    for attributes in dont_backup {
> +        assert!(!AttributeFilter::apply_filters(&filters, &attributes));
> +    }
> +}





  reply	other threads:[~2026-11-16 14:57 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13  8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-18 12:54   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner [this message]
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-18 12:55   ` Samuel Rufinatscha
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57   ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06   ` Samuel Rufinatscha
2026-11-18 15:19     ` Christian Ebner
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13  8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07   ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha
2026-11-19 11:58   ` Jakob-Tobias Winter

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7802620f-4d59-41d7-8807-a8cfc34fe007@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    --cc=tobias@stuttgart.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal