From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 608381FF0DF for ; Thu, 05 Nov 2026 12:13:55 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 498852166E; Thu, 05 Nov 2026 12:13:50 +0100 (CET) From: Hannes Laimer To: pve-devel@lists.proxmox.com Subject: [PATCH proxmox-ve-rs v4 05/45] ve-config: sdn: microseg: add identity allocation for group sets Date: Thu, 5 Nov 2026 12:12:44 +0100 Message-ID: <20261105111324.706922-6-h.laimer@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261105111324.706922-1-h.laimer@proxmox.com> References: <20261105111324.706922-1-h.laimer@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793877212267 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.282 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: ABBYOVNQGGS2JH33POG736Z2ZOYKVYMT X-Message-ID-Hash: ABBYOVNQGGS2JH33POG736Z2ZOYKVYMT X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A guest interface carries a set of groups, but the wire has room for a single 16-bit id. Group sets that no rule can tell apart form one class with one id. So the ids in use grow with what the rules distinguish, not with the number of distinct group sets. An id is stamped on one host and may be looked up on another after a config change. Reusing it for an unrelated class could give a packet still in flight the wrong verdict. So a class keeps its id while it exists. A new class takes an id never issued before. Once every id has been issued, an id unused for an hour can be issued again. Without one the allocation fails. Signed-off-by: Hannes Laimer --- .../src/sdn/microseg/identity.rs | 554 ++++++++++++++++++ proxmox-ve-config/src/sdn/microseg/mod.rs | 9 + proxmox-ve-config/src/sdn/mod.rs | 1 + 3 files changed, 564 insertions(+) create mode 100644 proxmox-ve-config/src/sdn/microseg/identity.rs create mode 100644 proxmox-ve-config/src/sdn/microseg/mod.rs diff --git a/proxmox-ve-config/src/sdn/microseg/identity.rs b/proxmox-ve-config/src/sdn/microseg/identity.rs new file mode 100644 index 000000000000..c1bdc1e8bebd --- /dev/null +++ b/proxmox-ve-config/src/sdn/microseg/identity.rs @@ -0,0 +1,554 @@ +//! Signature-based identity allocation for microseg. +//! +//! An *identity* is not one group set but a class of group sets that no policy can tell apart. Two +//! sets share an id when every policy's src and dst predicate fires the same way on both. The class +//! carries an *id* that is stamped on the wire, and a *representative*, the smallest member set in +//! sort order. A merge keeps the id of the smallest representative. +//! +//! Allocation is **fresh-first**. Nodes reload one after the other, so a packet stamped under the +//! old config can arrive at a node that already runs the new one. A class therefore keeps its id +//! only while it keeps every member set it had. Once a set leaves it, by a split or because the +//! set left the config, the id is retired and the parts get fresh numbers, so the old stamp +//! matches no pair. Ids come from a counter that only increases, and a retired id sits in a +//! quarantine (`RETIRE_QUARANTINE_SECS`) before it can be reused. Once the counter has issued every +//! number, allocation reclaims the lowest number whose quarantine has passed. It fails only with +//! all 65535 numbers live or quarantined at the same time. + +use std::collections::{BTreeMap, BTreeSet, HashMap}; + +use serde::{Deserialize, Deserializer, Serialize}; + +use proxmox_schema::api; + +/// The group set assigned to an interface. `BTreeSet`'s `Ord` compares the sorted elements in +/// order, which is exactly the representative order. +pub(crate) type GroupSet = BTreeSet; + +/// The reserved name of the built-in *untagged* group, the no-group identity. A source predicate +/// names it to match unstamped traffic, such as gateway, DHCP and ARP replies or cross-node frames +/// without a GBP tag. It can stand next to real groups, as in `any{untagged, web}`. It is not a +/// stored group and cannot be used in an assignment. +pub(crate) const UNTAGGED_GROUP: &str = "untagged"; + +/// An unstamped frame reads as this id. It is never issued to a class, so a real group set +/// (even one no policy matches) never collides with it and never inherits its `untagged` +/// allows. Its representative in the rule fold is the built-in `untagged` group. +pub(crate) const UNTAGGED_ID: u16 = 0; + +/// Reusing a number is only safe once no packet stamped under its old meaning can still exist. The +/// generous margin also covers nodes that apply a render late. A node cut off from pmxcfs for +/// longer than this while still passing traffic keeps stamping ids that may by then name another +/// class. A cluster in that state is broken in worse ways. +const RETIRE_QUARANTINE_SECS: u64 = 60 * 60; + +#[api()] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "lowercase")] +/// How a match compares the listed items with the present ones. +pub enum PredicateMatch { + /// Fires when at least one listed item is present. + #[default] + Any, + /// Fires when every listed item is present. + All, + /// Fires only when the present items are exactly the listed ones. + Exact, + /// Fires on every set holding a group, nothing is listed. + Every, +} + +/// A predicate over group names, the src or dst half of a policy. +/// Validation refuses an empty `groups` except for `Every`, which lists nothing and never fires +/// on the untagged identity. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Predicate { + pub(crate) kind: PredicateMatch, + pub(crate) groups: GroupSet, +} + +impl Predicate { + pub(crate) fn fires(&self, set: &GroupSet) -> bool { + match self.kind { + PredicateMatch::Any => self.groups.intersection(set).next().is_some(), + PredicateMatch::All => self.groups.is_subset(set), + PredicateMatch::Exact => &self.groups == set, + PredicateMatch::Every => set.iter().any(|group| group != UNTAGGED_GROUP), + } + } +} + +/// The verdict is not part of identity, it lives in the fold. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Policy { + pub(crate) src: Predicate, + pub(crate) dst: Predicate, +} + +/// How a set fires against each policy in order, as `[p0.src, p0.dst, p1.src, ...]`. Only equality +/// matters, sets with equal signatures get the same verdicts. Signatures compare only under one +/// policy order. +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub(crate) struct Signature(Vec); + +pub(crate) fn signature(set: &GroupSet, policies: &[Policy]) -> Signature { + let mut bits = Vec::with_capacity(policies.len() * 2); + for policy in policies { + bits.push(policy.src.fires(set)); + bits.push(policy.dst.fires(set)); + } + Signature(bits) +} + +/// How much of the id space a registry uses. A retired id whose quarantine has passed is counted +/// until the next allocation drops it. +#[derive(Debug, Serialize)] +pub struct IdSpace { + pub used: usize, + pub retired: usize, + /// Every id but the one untagged traffic carries. + pub total: usize, +} + +/// Each live id with its representative set, plus the id counter. Only which id each class +/// carries is kept, the classes themselves come from the policies. Serialized into the running +/// config so every host reads the same id-to-class naming. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct Registry { + next: u16, + #[serde(deserialize_with = "int_keys")] + classes: BTreeMap, + /// Retired id to unix time of retirement. Entries whose quarantine has passed are dropped by + /// the next [`allocate`]. + #[serde( + default, + deserialize_with = "int_keys", + skip_serializing_if = "BTreeMap::is_empty" + )] + retired: BTreeMap, +} + +/// A round trip through perlmod hands integer map keys back as strings, which serde does not +/// accept for a u16 key. +fn int_keys<'de, D: Deserializer<'de>, V: Deserialize<'de>>( + deserializer: D, +) -> Result, D::Error> { + BTreeMap::::deserialize(deserializer)? + .into_iter() + .map(|(key, value)| { + key.parse::() + .map(|key| (key, value)) + .map_err(serde::de::Error::custom) + }) + .collect() +} + +impl Default for Registry { + fn default() -> Self { + Self::new() + } +} + +impl Registry { + /// Ids start at 1, since 0 is the untagged id. + pub(crate) fn new() -> Self { + Self { + next: 1, + classes: BTreeMap::new(), + retired: BTreeMap::new(), + } + } + + pub fn classes(&self) -> &BTreeMap { + &self.classes + } + + pub fn id_space(&self) -> IdSpace { + IdSpace { + used: self.classes.len(), + retired: self.retired.len(), + total: u16::MAX as usize, + } + } +} + +/// The member sets of each class of `registry`, the `sets` grouped by the id their signature under +/// `policies` resolves to. +pub fn class_members( + registry: &Registry, + sets: &BTreeSet, + policies: &[Policy], +) -> BTreeMap> { + let ids: HashMap = registry + .classes + .iter() + .map(|(&id, rep)| (signature(rep, policies), id)) + .collect(); + let mut members: BTreeMap> = BTreeMap::new(); + for set in sets { + if let Some(&id) = ids.get(&signature(set, policies)) { + members.entry(id).or_default().insert(set.clone()); + } + } + members +} + +/// Recompute the registry from the sets the assignments name and the policies, fresh-first against +/// `prev`. +/// +/// Sets with the same signature form a class. A class keeps the id of a previous class whose +/// member sets all still land in it, so a class that lost a set gets a fresh id. A merge keeps the +/// id of the smallest representative and retires the rest. `prev_members` are the previous +/// classes' member sets as `class_members` gives them, and a class missing there is judged by its +/// representative alone. The sets no policy matches share the all-false signature and form one +/// class with its own id, distinct from `UNTAGGED_ID`. `now` is the render time (unix seconds) the +/// quarantine runs on. +pub fn allocate( + prev: &Registry, + prev_members: &BTreeMap>, + sets: &BTreeSet, + policies: &[Policy], + now: u64, +) -> anyhow::Result { + // The sets arrive in ascending order, so the first set seen with a signature is the class's + // representative and the classes come out in representative order. That order decides which + // class gets which fresh number, so the same input issues the same numbers. + let mut class_of: HashMap = HashMap::new(); + let mut reps: Vec<&GroupSet> = Vec::new(); + for set in sets { + class_of.entry(signature(set, policies)).or_insert_with(|| { + reps.push(set); + reps.len() - 1 + }); + } + + let mut claims: Vec> = vec![None; reps.len()]; + for (&id, rep) in &prev.classes { + let Some(&class) = class_of.get(&signature(rep, policies)) else { + continue; + }; + let intact = prev_members.get(&id).is_none_or(|members| { + members.iter().all(|member| { + sets.contains(member) && class_of.get(&signature(member, policies)) == Some(&class) + }) + }); + if !intact { + continue; + } + if claims[class].is_none_or(|(_, best)| rep < best) { + claims[class] = Some((id, rep)); + } + } + + let mut retired: BTreeMap = prev + .retired + .iter() + .filter(|&(_, &at)| now.saturating_sub(at) < RETIRE_QUARANTINE_SECS) + .map(|(&id, &at)| (id, at)) + .collect(); + + let mut next = prev.next; + let mut classes = BTreeMap::new(); + for (rep, claim) in reps.into_iter().zip(claims) { + let id = match claim { + Some((id, _)) => id, + None => { + // the counter wrapped to 0, so the fresh space is used up + if next == UNTAGGED_ID { + reclaim(prev, &retired, &classes)? + } else { + let id = next; + next = next.wrapping_add(1); + id + } + } + }; + classes.insert(id, rep.clone()); + } + + for &id in prev.classes.keys() { + if !classes.contains_key(&id) { + retired.insert(id, now); + } + } + + Ok(Registry { + next, + classes, + retired, + }) +} + +/// Called with the fresh space used up, so a number free here was retired and has left its +/// quarantine. An id that was live until this pass may still be on the wire, so it is kept like one +/// in quarantine. +fn reclaim( + prev: &Registry, + quarantined: &BTreeMap, + taken: &BTreeMap, +) -> anyhow::Result { + (1..=u16::MAX) + .find(|id| { + !prev.classes.contains_key(id) + && !quarantined.contains_key(id) + && !taken.contains_key(id) + }) + .ok_or_else(|| { + anyhow::anyhow!( + "microseg identity id space exhausted \ + (all 65535 ids are live or recently retired)" + ) + }) +} + +#[cfg(test)] +mod tests { + use super::PredicateMatch::*; + use super::*; + + fn set(xs: &[&str]) -> GroupSet { + xs.iter().map(|x| x.to_string()).collect() + } + + fn pred(kind: PredicateMatch, xs: &[&str]) -> Predicate { + Predicate { + kind, + groups: set(xs), + } + } + + fn id_of(reg: &Registry, set: &GroupSet, policies: &[Policy]) -> Option { + let sig = signature(set, policies); + reg.classes + .iter() + .find(|(_, rep)| signature(rep, policies) == sig) + .map(|(&id, _)| id) + } + + fn sets(xs: &[&[&str]]) -> BTreeSet { + xs.iter().map(|s| set(s)).collect() + } + + #[test] + fn unmatched_set_gets_its_own_id_not_the_untagged_id() { + let policies = [Policy { + src: pred(Any, &["untagged"]), + dst: pred(Any, &["b"]), + }]; + let reg = allocate( + &Registry::new(), + &BTreeMap::new(), + &sets(&[&["e"]]), + &policies, + 0, + ) + .unwrap(); + + let id = id_of(®, &set(&["e"]), &policies).expect("an unmatched set still has a class"); + assert_ne!( + id, UNTAGGED_ID, + "an unmatched set must not alias onto the untagged id" + ); + + let reg = allocate( + &Registry::new(), + &BTreeMap::new(), + &sets(&[&["e"], &["f"]]), + &policies, + 0, + ) + .unwrap(); + assert_eq!( + reg.classes().len(), + 1, + "all unmatched sets collapse to one class" + ); + assert_eq!( + id_of(®, &set(&["e"]), &policies), + id_of(®, &set(&["f"]), &policies) + ); + } + + #[test] + fn a_split_retires_the_id_and_issues_fresh_ones() { + let p1 = [Policy { + src: pred(Any, &["b"]), + dst: pred(Any, &["i"]), + }]; + let both = sets(&[&["b", "c"], &["b", "d"]]); + let r1 = allocate(&Registry::new(), &BTreeMap::new(), &both, &p1, 0).unwrap(); + assert_eq!(id_of(&r1, &set(&["b", "c"]), &p1), Some(1)); + assert_eq!(id_of(&r1, &set(&["b", "d"]), &p1), Some(1)); + + // then EXACT{b,c} is added, which fires only on {b,c}, so the class splits + let p2 = [ + Policy { + src: pred(Any, &["b"]), + dst: pred(Any, &["i"]), + }, + Policy { + src: pred(Exact, &["b", "c"]), + dst: pred(Any, &["i"]), + }, + ]; + let r2 = allocate(&r1, &class_members(&r1, &both, &p1), &both, &p2, 0).unwrap(); + + // a stale stamp of 1 from a node not yet reloaded must not pass as {b,c} alone + assert_eq!( + id_of(&r2, &set(&["b", "c"]), &p2), + Some(2), + "fresh id for one part" + ); + assert_eq!( + id_of(&r2, &set(&["b", "d"]), &p2), + Some(3), + "fresh id for the other" + ); + assert_eq!(r2.retired.get(&1), Some(&0), "the shared id is quarantined"); + assert_eq!(r2.next, 4); + } + + #[test] + fn a_departed_set_retires_the_id() { + let p = [Policy { + src: pred(Any, &["b"]), + dst: pred(Any, &["i"]), + }]; + let both = sets(&[&["b", "c"], &["b", "d"]]); + let r1 = allocate(&Registry::new(), &BTreeMap::new(), &both, &p, 0).unwrap(); + assert_eq!(id_of(&r1, &set(&["b", "d"]), &p), Some(1)); + + // {b,d} leaves the config, its guests may still stamp 1 on a node not yet reloaded + let one = sets(&[&["b", "c"]]); + let r2 = allocate(&r1, &class_members(&r1, &both, &p), &one, &p, 0).unwrap(); + assert_eq!( + id_of(&r2, &set(&["b", "c"]), &p), + Some(2), + "the remaining set gets a fresh id" + ); + assert_eq!(r2.retired.get(&1), Some(&0)); + } + + #[test] + fn merge_retires_the_fused_id_and_prefers_fresh() { + let p_split = [ + Policy { + src: pred(Any, &["b"]), + dst: pred(Any, &["i"]), + }, + Policy { + src: pred(Exact, &["b", "c"]), + dst: pred(Any, &["i"]), + }, + ]; + let both = sets(&[&["b", "c"], &["b", "d"]]); + let r_split = allocate(&Registry::new(), &BTreeMap::new(), &both, &p_split, 0).unwrap(); + assert_eq!(id_of(&r_split, &set(&["b", "c"]), &p_split), Some(1)); + assert_eq!(id_of(&r_split, &set(&["b", "d"]), &p_split), Some(2)); + assert_eq!(r_split.next, 3); + + // Drop the distinguisher, so {b,c} and {b,d} merge. + let p_merge = [Policy { + src: pred(Any, &["b"]), + dst: pred(Any, &["i"]), + }]; + let r_merge = allocate( + &r_split, + &class_members(&r_split, &both, &p_split), + &both, + &p_merge, + 0, + ) + .unwrap(); + assert_eq!(id_of(&r_merge, &set(&["b", "c"]), &p_merge), Some(1)); + assert_eq!( + id_of(&r_merge, &set(&["b", "d"]), &p_merge), + Some(1), + "merged into the survivor" + ); + assert!(!r_merge.classes().contains_key(&2), "id 2 is retired"); + assert_eq!(r_merge.retired.get(&2), Some(&0), "id 2 is quarantined"); + assert_eq!(r_merge.next, 3, "the counter never rewinds"); + + let r_next = allocate( + &r_merge, + &class_members(&r_merge, &both, &p_merge), + &sets(&[&["b", "c"], &["b", "d"], &["i"]]), + &p_merge, + 0, + ) + .unwrap(); + assert_eq!( + id_of(&r_next, &set(&["i"]), &p_merge), + Some(3), + "new class skips the retired id" + ); + } + + fn exhausted_registry(retired_id: u16, retired_at: u64, free_id: u16) -> Registry { + let classes: BTreeMap = (1..=u16::MAX) + .filter(|&id| id != retired_id && id != free_id) + .map(|id| (id, set(&[&id.to_string()]))) + .collect(); + Registry { + next: 0, + classes, + retired: BTreeMap::from([(retired_id, retired_at)]), + } + } + + #[test] + fn reclaims_a_retired_id_only_after_its_quarantine() { + // {b,e} fires both src predicates, no singleton does, so no previous class claims it and it + // needs an id with no fresh space left. + let policies = [ + Policy { + src: pred(Any, &["b"]), + dst: pred(Any, &["i"]), + }, + Policy { + src: pred(Any, &["e"]), + dst: pred(Any, &["i"]), + }, + ]; + let retired_at = 1_000; + let prev = exhausted_registry(7, retired_at, 11); + + // one number is free outright, so the first issued id takes it regardless of the quarantine + let reg = allocate( + &prev, + &BTreeMap::new(), + &sets(&[&["b", "e"]]), + &policies, + retired_at + 1, + ) + .unwrap(); + assert_eq!(id_of(®, &set(&["b", "e"]), &policies), Some(11)); + + // with 11 also live, the only candidate is the quarantined 7. Inside the quarantine + // issuing must fail rather than rebind a number packets may still carry + let mut prev = exhausted_registry(7, retired_at, 11); + prev.classes.insert(11, set(&["11"])); + let before = retired_at + RETIRE_QUARANTINE_SECS - 1; + let err = allocate( + &prev, + &BTreeMap::new(), + &sets(&[&["b", "e"]]), + &policies, + before, + ) + .unwrap_err(); + assert!(err.to_string().contains("exhausted"), "got: {err:#}"); + + let after = retired_at + RETIRE_QUARANTINE_SECS; + let reg = allocate( + &prev, + &BTreeMap::new(), + &sets(&[&["b", "e"]]), + &policies, + after, + ) + .unwrap(); + assert_eq!(id_of(®, &set(&["b", "e"]), &policies), Some(7)); + assert!(!reg.retired.contains_key(&7)); + } +} diff --git a/proxmox-ve-config/src/sdn/microseg/mod.rs b/proxmox-ve-config/src/sdn/microseg/mod.rs new file mode 100644 index 000000000000..37c4f92ff0c8 --- /dev/null +++ b/proxmox-ve-config/src/sdn/microseg/mod.rs @@ -0,0 +1,9 @@ +//! Microseg subsystem of the SDN config. +//! +//! Microseg is an identity-based stateless firewall. Each managed guest NIC is assigned a *set* of +//! *groups*. The render allocates a wire identity for each set (see [`identity`]), and every host +//! gets the same ids. Each host looks the identities up and compiles the policy *rules* into a +//! `(src_id, dst_id) -> allow|deny` map. A rule matches traffic by a `src` and a `dst` +//! *predicate*, each a match kind over a list of groups. + +pub mod identity; diff --git a/proxmox-ve-config/src/sdn/mod.rs b/proxmox-ve-config/src/sdn/mod.rs index 2133396145bd..f647e872c636 100644 --- a/proxmox-ve-config/src/sdn/mod.rs +++ b/proxmox-ve-config/src/sdn/mod.rs @@ -1,6 +1,7 @@ pub mod config; pub mod fabric; pub mod ipam; +pub mod microseg; pub mod prefix_list; pub mod route_map; pub mod wireguard; -- 2.47.3