From: Christian Ebner <c.ebner@proxmox.com>
To: Jakob-Tobias Winter <tobias@stuttgart.de>, pbs-devel@lists.proxmox.com
Subject: Re: [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs
Date: Mon, 16 Nov 2026 15:57:50 +0100 [thread overview]
Message-ID: <7802620f-4d59-41d7-8807-a8cfc34fe007@proxmox.com> (raw)
In-Reply-To: <20261113085637.57360-5-tobias@stuttgart.de>
On 11/13/26 10:06 AM, Jakob-Tobias Winter wrote:
> Tape backup jobs get a list of attribute filters to select backup
> groups by the attributes of their snapshots, for example to write
> the guests tagged as production and as test to different media
> pools. A filter only matches a whole value of its key. Several
> filters are combined like group filters: one of the include filters
> has to match, if there are any, and none of the exclude filters.
>
> The job option is optional. Existing job configurations stay valid
> and select the same groups as before.
>
> Signed-off-by: Jakob-Tobias Winter <tobias@stuttgart.de>
> ---
> pbs-api-types/src/jobs.rs | 98 ++++++++++-
> pbs-api-types/tests/attribute_filter_tests.rs | 155 ++++++++++++++++++
> 2 files changed, 252 insertions(+), 1 deletion(-)
> create mode 100644 pbs-api-types/tests/attribute_filter_tests.rs
>
> diff --git a/pbs-api-types/src/jobs.rs b/pbs-api-types/src/jobs.rs
> index 9a5d2b77..8e558e4a 100644
> --- a/pbs-api-types/src/jobs.rs
> +++ b/pbs-api-types/src/jobs.rs
> @@ -15,7 +15,7 @@ use crate::{
> BackupType, CRYPT_KEY_ID_SCHEMA, DATASTORE_SCHEMA, DRIVE_NAME_SCHEMA, MEDIA_POOL_NAME_SCHEMA,
> NS_MAX_DEPTH_REDUCED_SCHEMA, NotificationMode, PROXMOX_SAFE_ID_FORMAT,
> PROXMOX_SAFE_ID_REGEX_STR, REMOTE_ID_SCHEMA, RateLimitConfig, SINGLE_LINE_COMMENT_SCHEMA,
> - Userid,
> + SNAPSHOT_ATTRIBUTE_REGEX, SnapshotAttributes, Userid,
> };
>
> const_regex! {
> @@ -345,6 +345,10 @@ pub struct VerificationJobStatus {
> schema: GROUP_FILTER_LIST_SCHEMA,
> optional: true,
> },
> + "attribute-filter": {
> + schema: ATTRIBUTE_FILTER_LIST_SCHEMA,
> + optional: true,
> + },
> ns: {
> type: BackupNamespace,
> optional: true,
> @@ -379,6 +383,8 @@ pub struct TapeBackupJobSetup {
> pub notification_mode: Option<NotificationMode>,
> #[serde(skip_serializing_if = "Option::is_none")]
> pub group_filter: Option<Vec<GroupFilter>>,
> + #[serde(skip_serializing_if = "Option::is_none")]
> + pub attribute_filter: Option<Vec<AttributeFilter>>,
> #[serde(skip_serializing_if = "Option::is_none", default)]
> pub ns: Option<BackupNamespace>,
> #[serde(skip_serializing_if = "Option::is_none", default)]
> @@ -550,6 +556,96 @@ pub const GROUP_FILTER_SCHEMA: Schema = StringSchema::new(
> pub const GROUP_FILTER_LIST_SCHEMA: Schema =
> ArraySchema::new("List of group filters.", &GROUP_FILTER_SCHEMA).schema();
>
> +#[derive(Clone, Debug, PartialEq, Eq)]
> +/// Filter for matching the attributes of a snapshot.
> +pub struct AttributeFilter {
> + pub is_exclude: bool,
> + pub key: String,
as mentioned on patch 0001, key ...
> + pub value: String,
... and value should preferably have dedicated types ...
> +}
> +
> +impl AttributeFilter {
> + /// Returns whether the value of the filter is one of the values the attributes have for its
> + /// key.
> + pub fn matches(&self, attributes: &SnapshotAttributes) -> bool {
> + attributes
> + .get(&self.key)
> + .is_some_and(|values| values.contains(&self.value))
> + }
> +
> + /// Returns whether the attributes pass the filters.
> + ///
> + /// If there are include filters, at least one of them has to match. No exclude filter may
> + /// match.
> + pub fn apply_filters(filters: &[AttributeFilter], attributes: &SnapshotAttributes) -> bool {
> + let is_included = if filters.iter().filter(|f| !f.is_exclude).count() == 0 {
> + true
> + } else {
> + filters
> + .iter()
> + .filter(|f| !f.is_exclude)
> + .any(|filter| filter.matches(attributes))
> + };
> +
> + is_included
> + && !filters
> + .iter()
> + .filter(|f| f.is_exclude)
> + .any(|filter| filter.matches(attributes))
> + }
> +}
> +
> +impl std::str::FromStr for AttributeFilter {
> + type Err = anyhow::Error;
> +
> + fn from_str(s: &str) -> Result<Self, Self::Err> {
> + let (is_exclude, attribute) = match s.split_once(':') {
> + Some(("include", value)) => (false, value),
> + Some(("exclude", value)) => (true, value),
> + _ => (false, s),
> + };
> +
> + match attribute.split_once('=') {
> + Some((key, value)) if SNAPSHOT_ATTRIBUTE_REGEX.is_match(attribute) => {
... this can then use the dedicate types for parsing
> + Ok(AttributeFilter {
> + is_exclude,
> + key: key.to_string(),
> + value: value.to_string(),
> + })
> + }
> + _ => bail!("input doesn't match expected format '[<exclude:|include:>]KEY=VALUE'"),
> + }
> + }
> +}
> +
> +// used for serializing below, caution!
> +impl std::fmt::Display for AttributeFilter {
> + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
> + if self.is_exclude {
> + f.write_str("exclude:")?;
> + }
> + write!(f, "{}={}", self.key, self.value)
> + }
> +}
> +
> +proxmox_serde::forward_deserialize_to_from_str!(AttributeFilter);
> +proxmox_serde::forward_serialize_to_display!(AttributeFilter);
> +
> +fn verify_attribute_filter(input: &str) -> Result<(), anyhow::Error> {
> + AttributeFilter::from_str(input).map(|_| ())
> +}
> +
> +pub const ATTRIBUTE_FILTER_SCHEMA: Schema = StringSchema::new(
> + "Attribute filter, matches if the given value is one of the values of the key. Can be \
> + inverted by prepending 'exclude:'.",
> +)
> +.format(&ApiStringFormat::VerifyFn(verify_attribute_filter))
> +.type_text("[<exclude:|include:>]KEY=VALUE")
> +.schema();
> +
> +pub const ATTRIBUTE_FILTER_LIST_SCHEMA: Schema =
> + ArraySchema::new("List of attribute filters.", &ATTRIBUTE_FILTER_SCHEMA).schema();
> +
> pub const TRANSFER_LAST_SCHEMA: Schema =
> IntegerSchema::new("Limit transfer to last N snapshots (per group), skipping others")
> .minimum(1)
> diff --git a/pbs-api-types/tests/attribute_filter_tests.rs b/pbs-api-types/tests/attribute_filter_tests.rs
> new file mode 100644
> index 00000000..c355e6b4
> --- /dev/null
> +++ b/pbs-api-types/tests/attribute_filter_tests.rs
> @@ -0,0 +1,155 @@
> +use pbs_api_types::{AttributeFilter, SnapshotAttributes};
> +use std::str::FromStr;
> +
> +fn attributes(list: &[(&str, &[&str])]) -> SnapshotAttributes {
> + list.iter()
> + .map(|(key, values)| {
> + (
> + key.to_string(),
> + values.iter().map(|value| value.to_string()).collect(),
> + )
> + })
> + .collect()
> +}
> +
> +fn filters(list: &[&str]) -> Vec<AttributeFilter> {
> + list.iter()
> + .map(|filter| AttributeFilter::from_str(filter).unwrap())
> + .collect()
> +}
> +
> +#[test]
> +fn test_parse_filter() {
> + let filter = AttributeFilter::from_str("pve-tag=prod").unwrap();
> + assert!(!filter.is_exclude);
> + assert_eq!(filter.key, "pve-tag");
> + assert_eq!(filter.value, "prod");
> + assert_eq!(filter.to_string(), "pve-tag=prod");
> +
> + let filter = AttributeFilter::from_str("include:pve-tag=prod").unwrap();
> + assert!(!filter.is_exclude);
> + assert_eq!(filter.to_string(), "pve-tag=prod");
> +
> + let filter = AttributeFilter::from_str("exclude:pve-pool=customers/a_1.test").unwrap();
> + assert!(filter.is_exclude);
> + assert_eq!(filter.key, "pve-pool");
> + assert_eq!(filter.value, "customers/a_1.test");
> + assert_eq!(filter.to_string(), "exclude:pve-pool=customers/a_1.test");
> +
> + let invalid = [
> + "",
> + "pve-tag",
> + "pve-tag=",
> + "=prod",
> + "Pve-Tag=prod",
> + "1tag=prod",
> + "pve-tag=prod;web",
> + "pve-tag=prod,web",
> + "pve-tag=prod web",
> + "pve-tag=prod=web",
> + "pve-tag=prod:web",
> + "other:pve-tag=prod",
> + "exclude:exclude:pve-tag=prod",
> + ];
> +
> + for filter in invalid {
> + assert!(
> + AttributeFilter::from_str(filter).is_err(),
> + "'{filter}' must not parse"
> + );
> + }
> +}
> +
> +#[test]
> +fn test_no_filters() {
> + let filters = [];
> +
> + assert!(AttributeFilter::apply_filters(&filters, &attributes(&[])));
> + assert!(AttributeFilter::apply_filters(
> + &filters,
> + &attributes(&[("pve-tag", &["prod", "web"])])
> + ));
> +}
> +
> +#[test]
> +fn test_include_filters() {
> + let filters = filters(&["pve-tag=prod", "pve-pool=customer1"]);
> +
> + let do_backup = [
> + attributes(&[("pve-tag", &["prod"])]),
> + attributes(&[("pve-tag", &["db", "prod", "web"])]),
> + attributes(&[("pve-pool", &["customer1"])]),
> + attributes(&[("pve-pool", &["customer1"]), ("pve-tag", &["test"])]),
> + ];
> +
> + let dont_backup = [
> + attributes(&[]),
> + attributes(&[("pve-tag", &[])]),
> + attributes(&[("pve-tag", &["nonprod", "prod-eu", "pro", "Prod"])]),
> + attributes(&[("pve-pool", &["prod"])]),
> + attributes(&[("pve-pool", &["customer1/sub", "customer10"])]),
> + attributes(&[("other", &["prod", "customer1"])]),
> + ];
> +
> + for attributes in do_backup {
> + assert!(AttributeFilter::apply_filters(&filters, &attributes));
> + }
> +
> + for attributes in dont_backup {
> + assert!(!AttributeFilter::apply_filters(&filters, &attributes));
> + }
> +}
> +
> +#[test]
> +fn test_exclude_filters() {
> + let filters = filters(&["exclude:pve-tag=prod", "exclude:pve-tag=test"]);
> +
> + let do_backup = [
> + attributes(&[]),
> + attributes(&[("pve-tag", &["web"])]),
> + attributes(&[("pve-pool", &["prod"])]),
> + ];
> +
> + let dont_backup = [
> + attributes(&[("pve-tag", &["prod"])]),
> + attributes(&[("pve-tag", &["test", "web"])]),
> + attributes(&[("pve-tag", &["prod", "test"])]),
> + ];
> +
> + for attributes in do_backup {
> + assert!(AttributeFilter::apply_filters(&filters, &attributes));
> + }
> +
> + for attributes in dont_backup {
> + assert!(!AttributeFilter::apply_filters(&filters, &attributes));
> + }
> +}
> +
> +#[test]
> +fn test_include_and_exclude_filters() {
> + let filters = filters(&[
> + "exclude:pve-tag=test",
> + "pve-pool=customer1",
> + "pve-pool=customer2",
> + ]);
> +
> + let do_backup = [
> + attributes(&[("pve-pool", &["customer1"])]),
> + attributes(&[("pve-pool", &["customer2"]), ("pve-tag", &["prod"])]),
> + ];
> +
> + let dont_backup = [
> + attributes(&[]),
> + attributes(&[("pve-tag", &["prod"])]),
> + attributes(&[("pve-pool", &["customer3"])]),
> + attributes(&[("pve-pool", &["customer1"]), ("pve-tag", &["test"])]),
> + ];
> +
> + for attributes in do_backup {
> + assert!(AttributeFilter::apply_filters(&filters, &attributes));
> + }
> +
> + for attributes in dont_backup {
> + assert!(!AttributeFilter::apply_filters(&filters, &attributes));
> + }
> +}
next prev parent reply other threads:[~2026-11-16 14:57 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 8:56 [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH proxmox 01/12] pbs-api-types: add types and schemas for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:54 ` Samuel Rufinatscha
2026-11-13 8:56 ` [RFC PATCH proxmox 02/12] pbs-api-types: tape: add attribute filter to backup jobs Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner [this message]
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 03/12] datastore: add attributes to backup snapshots Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-18 12:55 ` Samuel Rufinatscha
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 04/12] fix #7452: api: datastore: add endpoints for snapshot attributes Jakob-Tobias Winter
2026-11-16 14:57 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 05/12] tape: backup job: allow selecting groups by " Jakob-Tobias Winter
2026-11-18 13:06 ` Samuel Rufinatscha
2026-11-18 15:19 ` Christian Ebner
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 06/12] ui: tape backup job: add attribute filter Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH proxmox-backup 07/12] docs: describe snapshot attributes and the tape job " Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH storage 08/12] pbs: add helper to set the attributes of a backup snapshot Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH guest-common 09/12] vzdump: add pbs-snapshot-attributes option Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 10/12] vzdump: store guest tags and pool as snapshot attributes on PBS Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH manager 11/12] ui: backup job: add selector for PBS snapshot attributes Jakob-Tobias Winter
2026-11-13 8:56 ` [RFC PATCH docs 12/12] vzdump: describe snapshot attributes on Proxmox Backup Server Jakob-Tobias Winter
2026-11-16 14:57 ` [RFC PATCH many 00/12] snapshot attributes for tags and pools of PVE guests, tape job filter Christian Ebner
2026-11-18 13:07 ` Samuel Rufinatscha
2026-11-18 14:22 ` Samuel Rufinatscha
2026-11-19 11:58 ` Jakob-Tobias Winter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7802620f-4d59-41d7-8807-a8cfc34fe007@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
--cc=tobias@stuttgart.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox