From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id A3D091FF0E3 for ; Tue, 27 Oct 2026 10:03:47 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 32BAB216B4; Tue, 27 Oct 2026 10:03:16 +0100 (CET) Message-ID: Date: Tue, 27 Oct 2026 09:57:40 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: superseded: [RFC cluster/docs/ifupdown2/manager/network/proxmox{-ve-rs,-ebpf,-perl-rs} v2 00/27] sdn: add microsegmentation support To: pve-devel@lists.proxmox.com References: <20260709091852.538885-1-h.laimer@proxmox.com> From: Hannes Laimer Content-Language: en-US In-Reply-To: <20260709091852.538885-1-h.laimer@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793091461143 X-SPAM-LEVEL: Spam detection results: 0 AWL -2.150 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_DBL_SPAM 5 Contains a spam URL listed in the Spamhaus DBL blocklist [sdn.pm] Message-ID-Hash: JJRONGOUSGLCCJQ5WA5F6Y526GUACEXN X-Message-ID-Hash: JJRONGOUSGLCCJQ5WA5F6Y526GUACEXN X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: superseded-by: https://lore.proxmox.com/pve-devel/20261027085527.702738-1-h.laimer@proxmox.com/ On 2026-07-09 11:19, Hannes Laimer wrote: > This adds support for microsegmentation using eBPF programs attached to > interfaces. Mostly the tap/veth interfaces on the guests directly. > > # Overview > Each guest interface is assigned a *set* of groups. Rather than a hierarchy, > group membership is treated as a signature. The agent compiles each interface's > group set into a single per-NIC *identity*, and policies are rules between > groups, matched against those sets with predicates. > > A rule is a `src` predicate and a `dst` predicate plus a verdict and a > priority, where a predicate is a match kind over a list of groups: > - any{...} the interface shares at least one of the listed groups > - all{...} the interface has all of the listed groups > - exact{...} the interface's groups are exactly the listed set > > For a packet, the `src` predicate is evaluated against the sender's set and the > `dst` predicate against the receiver's. Among the rules that fire the highest > `prio` wins, a conflicting tie (with different verdicts) denies, and if no rule > fires the packet is denied (default-deny). For example, with the groups > web/app/db/prod, specific allows punched through a broad deny: > > - any{web} -> any{app} : allow prio 10 > - any{app} -> any{db} : allow prio 10 > - all{prod} -> all{prod} : allow prio 10 # these three carve out east-west... > - any{web,app,db} -> any{web,app,db} : deny prio 5 # ...through a broad tier-to-tier deny > (the last is technically not needed cause the default is deny, in a larger > context having this explicitly can be necesarry though) > > The allows above are one-directional. Rules are stateless and per-packet, there > is no connection tracking. The reply needs its own allow, so for TCP between web > and app both any{web} -> any{app} and any{app} -> any{web} have to be allowed. > > Groups land on an interface either directly (a guest matcher: a `vmid`, > optionally a single `iface`), by tag (a tag matcher: every guest carrying the > named tags, e.g. `prod`/`staging`/`web`), or by name (a regex matcher: every > guest whose name matches a regular expression). Assignments are additive, an > interface's set is the union of every assignment that resolves to it, so the > matchers compose. > > `untagged` is a reserved built-in group (mark 0) for traffic that arrives > unstamped (gateways, DHCP/ARP, cross-node frames without a tag). It can be > named in a predicate and composed with real groups (e.g. `any{untagged, web}`), > but it is not assignable. > > Enforcement happens on the receiving side, cause that's where we for sure know both > - where the packet is from > - and, its destination (us) > > # config and API > The config is a single section-config, `sdn/microseg.cfg`. It holds these > object types, keyed by section id: > - group, a numeric `mark` (auto-assigned if omitted) and an optional comment > - rule, a `src`/`dst` predicate pair (`*_match` is any/all/exact over a group list), > a `prio` and `allow` > - guestassignment, binds a guest's NIC(s) (`vmid`, optional `iface`) to a group set > - tagassignment, binds the NICs of every guest matching `tags` (by `tag_match`) to a > group set, expanded against the live guest inventory on render > - regexassignment, same but for every guest whose name matches a regular > expression > - bridge, marks a bridge-facing interface as an SRv6 carrier (no UI yet) > The API exposes endpoints under > `/cluster/sdn/microseg/{group,rule,assignment,bridge}`, guarded by SDN.Audit for reads > and SDN.Allocate for writes. The assignment endpoint discriminates the two matcher > kinds on a `type` property (like the fabric `protocol`). > Rule and tag/regex-assignment ids are derived from their contents (an FNV-1a hash), > guest assignments from `vm{vmid}i{iface}`, groups and bridges take a chosen id. On > commit the config is rendered into `.running-config`, which is what the agent reads. > Rendering resolves each interface's group set to a wire identity and expands the > dynamic matchers against the inventory. > > # identity > An identity is not one id per group set but one id per *signature*. Two > interfaces share an id when every rule predicate fires the same way on both, so > no rule can tell them apart. The identities are the classes of that partition, > each named by a 16-bit id, and that id is what rides on the wire. > > Editing rules reshapes the partition. A new predicate that separates two sets > which used to look alike *splits* their class in two, and dropping the > predicate that told two classes apart *merges* them back into one. On a split > the subclass that still carries the old representative keeps the id and the > other gets a fresh one, on a merge the surviving class keeps its id and the > fused-away ones retire. > > New ids come fresh-first from a counter. A retired id is not reused right away, > it sits in a time quarantine long enough that no packet stamped under its old > meaning can still be in flight. Only once the fresh space is exhausted does > allocation fall back to reusing a retired id whose quarantine has passed, so > config churn alone cannot exhaust the 16-bit id space. Actually running out > would take more than 65534 identities the policies can tell apart. So it > doesn't scale with count of group combinations in use, but with the distinct > semantic meaning they carry. > > # skb->mark > Every packet in the kernel lives in a `sk_buff` struct, which has a 32-bit > `mark` field you can read and write while the packet moves through the kernel. > Nothing else in our SDN stack touches `mark` today, so microseg has it to > itself. Worth keeping in mind if that ever changes. For now we only use the > lower 16 bits, to carry the interface's identity (mark 0 is reserved for > unstamped traffic), so up to 65535 identities. > > The catch is `mark` only lives as long as the packet stays in the kernel, and > sooner or later it leaves. To carry the identity between hosts we have to put it > on the wire itself. For that we support two carriers: > - VXLAN-GBP, the 16-bit GBP field on the VXLAN encapsulation > - SRv6, the 16-bit Tag field on the SRH > The kernel already moves `mark` in and out of the VXLAN-GBP field for us (found > that out way too late :P). For SRv6 we do it ourselves with a small eBPF program. > > # eBPF > Both tagging (setting `mark`) and enforcement happen in an eBPF program > directly attached to the guest's interface. The programs themselves read and > write to/from two maps: > - tap_to_group, the identity to stamp for a given interface > - rules, a map of (src_id, dst_id) -> action, where src is the `mark` the > packet carries and dst is our own interface's identity > > Specifically, on ingress we set the mark, and on egress we enforce. > > # Implementation > We have an `agent` that reads the running sdn config, and applies that state to > the kernel. The binary is stateless and one-shot, not a long running daemon. It > runs on SDN apply, on tap_plug of a guest interface, and on boot after > pve-sdn-commit but before guests start. Those triggers cover every situation > where we have to touch kernel state. > > It compiles the config into what the data plane needs. It folds each > interface's group set into a wire identity (a registry kept stable across > renders so identities agree across hosts, allocated as described above), and > folds the rules into the `(src_id, dst_id)` action map by priority (highest > wins, conflicting ties deny). > > It keeps track of what is currently loaded in the kernel with two files under > `/run/proxmox-ebpf`. One keeping track of the bpf program that was compiled > into the binary, and one for keeping track of changes to the data structures > the bpf program accesses. The distinction is useful cause for only a program > change we can swap the currently attached program with the updated one > atomically. In case the data structure (the maps) changed there is no other way > than to tear-down all the current state and repopulate the maps and re-attach > the programs. Swapping out the maps first would have old programs interact with > a new data schema, and swapping the programs first will lead to new programs > accessing an old data schema. Either way, not good, so we wipe the state in > that case and re-build it. So, in case we change the structure of the data our > bpf programs access, there will be a bunch of ms where the configured > segmentation is not enforced. But that is the only scenario where that can > happen. > > ## aya, aya-ebpf > Aya is a rust lib that helps with working with BPF more easily. It has both a > userspace part (`aya`) and a part (`aya-ebpf`) that compiles to bpf. I chose to > only use the userspace part, and use C for the BPF programs themselves. The > reasons were: > - we don't have `aya-ebpf` packaged > - `aya-ebpf` *requires* nightly toolchain to compile > - the bpf programs are very small, in case this should change at some point we > can reconsider. Also, it doesn't matter what produces the .o file, so this > could be swapped in really easily if we decided to > > So we compile the bpf program written in C to a .o targeting bpf using clang > and include it when compiling the agent binary. > > ## SRv6 > Besides VXLAN-GBP, the identity can also ride in the 16-bit Tag field of an SRv6 > SRH. The kernel already does this move for VXLAN-GBP but not for SRv6, so this > adds a small eBPF program that writes the identity into the SRH on egress and > reads it back on ingress. The `bridge` object marks the bridge-facing interface > it runs on. > > This is only the tag carrier, not an SRv6 transport. Our SDN stack has no SRv6 > zone yet, so there is nothing configured to route it over, and there is no UI. > If SDN grows an SRv6 transport later (a small L3, EVPN-lite style option), the > carrier is already in place. > > It is packaged as a droppable tail. The carrier bridge section (proxmox-ve-config), > the bridge API (pve-network) and the bridge subsystem (proxmox-ebpf) are each the > last patch of their repo's series, so SRv6 can be deferred without touching the rest. > > # enabling VXLAN-GBP > The kernel only moves the mark into the GBP field if the vxlan device was > created with the GBP flag set, and it's create-only, the kernel won't > toggle it on a running device. ifupdown2 had no attribute for this, so > this series includes a small ifupdown2 patch adding `vxlan-gbp`, which > threads the flag through to the netlink/iproute2 create path. > > # testing > I have put pre-built packages on sani, these include ifupdown2 with the patch > > # building > 1. proxmox-ve-rs (install) > 2. proxmox-ebpf > 3. pve-cluster (install) > 4. proxmox-perl-rs (install) > 5. pve-network > 6. pve-manager > > # notes > * everything eBPF related did not change since v1, this v2 only really changes identity semantics > * using GENEVE would also be an option, but with this current approach we gain > a lot of flexibility. Expanding predicate logic, or having not just > allow/deny but also match ports or alike is possible without touching the > underlying identity system. (limiting the per-packet overhead to just 16 > bits is also good I'd argue) > * the enforement mechanism is not set in stone, weather this is eBPF or maybe > nftables with a map we populate as shortly discussed with @Stefan. The > current approach plays nicely with both since it conveys identity through > `skb->mark` that nft can access as well > * I did not send the first commit for `proxmox-ebpf` since it contains a > `vmlinux.h` which is rather large but is needed for compiling. The commit is in > my staff repo. > > # changelog > v2: > - core model reworked: each interface now holds a *set* of groups compiled > into a per-NIC wire identity, replacing the single-group parent/child tree > - rules gained explicit `src`/`dst` predicates (any/all/exact) and a `prio`, > replacing the implicit tree-distance ordering > - `untagged` is now a nameable built-in group (mark 0) > - assignments are additive: a NIC's set is the union of all that resolve to it > - added tag-matcher and name-regex assignments > - rule and dynamic-assignment ids are now generated > - identity allocation is fresh-first with quarantined reclamation, so churn > cannot exhaust the id space > - split the SRv6 carrier support into droppable tail patches > - updated docs > - updated cover-letter > > > proxmox-ve-rs: > > Hannes Laimer (5): > ve-config: sdn: add microseg signature-identity engine > ve-config: sdn: add microseg config types > ve-config: sdn: microseg: add tag matcher > ve-config: sdn: microseg: add name regex matcher > ve-config: sdn: microseg: add carrier bridge section > > proxmox-ve-config/src/sdn/config.rs | 9 +- > .../src/sdn/microseg/identity.rs | 624 +++++ > proxmox-ve-config/src/sdn/microseg/mod.rs | 2349 +++++++++++++++++ > proxmox-ve-config/src/sdn/mod.rs | 1 + > 4 files changed, 2982 insertions(+), 1 deletion(-) > create mode 100644 proxmox-ve-config/src/sdn/microseg/identity.rs > create mode 100644 proxmox-ve-config/src/sdn/microseg/mod.rs > > > proxmox-ebpf: > > Hannes Laimer (3): > agent: add userspace coordinator and stateless policy subsystem > bpf: add bridge subsystem > debian: add packaging and boot-time oneshot unit > > Makefile | 66 +++++++ > debian/changelog | 5 + > debian/control | 35 ++++ > debian/copyright | 18 ++ > debian/proxmox-ebpf.install | 1 + > debian/proxmox-ebpf.postrm | 11 ++ > debian/proxmox-ebpf.prerm | 12 ++ > debian/proxmox-ebpf.service | 15 ++ > debian/rules | 33 ++++ > debian/source/format | 1 + > include/mark.h | 30 +++ > src/agent.rs | 99 ++++++++++ > src/bridge/bpf/srv6.bpf.c | 76 +++++++ > src/bridge/mod.rs | 76 +++++++ > src/main.rs | 69 +++++++ > src/policy/bpf/tap.bpf.c | 68 +++++++ > src/policy/bpf/types.h | 23 +++ > src/policy/mod.rs | 268 +++++++++++++++++++++++++ > src/policy/types.rs | 45 +++++ > src/running_config.rs | 38 ++++ > src/state.rs | 149 ++++++++++++++ > src/subsystem.rs | 385 ++++++++++++++++++++++++++++++++++++ > src/tc.rs | 152 ++++++++++++++ > 23 files changed, 1675 insertions(+) > create mode 100644 Makefile > create mode 100644 debian/changelog > create mode 100644 debian/control > create mode 100644 debian/copyright > create mode 100644 debian/proxmox-ebpf.install > create mode 100755 debian/proxmox-ebpf.postrm > create mode 100755 debian/proxmox-ebpf.prerm > create mode 100644 debian/proxmox-ebpf.service > create mode 100755 debian/rules > create mode 100644 debian/source/format > create mode 100644 include/mark.h > create mode 100644 src/agent.rs > create mode 100644 src/bridge/bpf/srv6.bpf.c > create mode 100644 src/bridge/mod.rs > create mode 100644 src/main.rs > create mode 100644 src/policy/bpf/tap.bpf.c > create mode 100644 src/policy/bpf/types.h > create mode 100644 src/policy/mod.rs > create mode 100644 src/policy/types.rs > create mode 100644 src/running_config.rs > create mode 100644 src/state.rs > create mode 100644 src/subsystem.rs > create mode 100644 src/tc.rs > > > pve-cluster: > > Hannes Laimer (1): > cfs: add 'sdn/microseg.cfg' to observed files > > src/PVE/Cluster.pm | 1 + > 1 file changed, 1 insertion(+) > > > proxmox-perl-rs: > > Hannes Laimer (1): > pve-rs: sdn: add microseg config binding > > pve-rs/Makefile | 1 + > pve-rs/src/bindings/sdn/microseg.rs | 235 ++++++++++++++++++++++++++++ > pve-rs/src/bindings/sdn/mod.rs | 1 + > 3 files changed, 237 insertions(+) > create mode 100644 pve-rs/src/bindings/sdn/microseg.rs > > > ifupdown2: > > Hannes Laimer (1): > d/patches: add support for VXLAN-GBP flag > > ...addons-vxlan-add-vxlan-gbp-attribute.patch | 228 ++++++++++++++++++ > debian/patches/series | 1 + > 2 files changed, 229 insertions(+) > create mode 100644 debian/patches/pve/0016-addons-vxlan-add-vxlan-gbp-attribute.patch > > > pve-network: > > Hannes Laimer (8): > sdn: microseg: add config, API and guest inventory > sdn: dry-run: surface pending microseg changes > sdn: zones: trigger microseg apply on tap_plug > sdn: zones: add vxlan-gbp option to vxlan and evpn zones > evpn: disable vxlan-learning on create if GBP is enabled > sdn: microseg: add tag matcher > sdn: microseg: add name regex matcher > sdn: microseg: add carrier bridge API > > src/PVE/API2/Network/SDN.pm | 31 ++ > src/PVE/API2/Network/SDN/Makefile | 2 + > src/PVE/API2/Network/SDN/Microseg.pm | 204 +++++++ > .../API2/Network/SDN/Microseg/Assignment.pm | 185 +++++++ > src/PVE/API2/Network/SDN/Microseg/Bridge.pm | 179 ++++++ > src/PVE/API2/Network/SDN/Microseg/Group.pm | 179 ++++++ > src/PVE/API2/Network/SDN/Microseg/Makefile | 8 + > src/PVE/API2/Network/SDN/Microseg/Rule.pm | 180 +++++++ > src/PVE/Network/SDN.pm | 37 ++ > src/PVE/Network/SDN/Makefile | 1 + > src/PVE/Network/SDN/Microseg.pm | 509 ++++++++++++++++++ > src/PVE/Network/SDN/Zones.pm | 6 + > src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 11 + > src/PVE/Network/SDN/Zones/VxlanPlugin.pm | 9 + > 14 files changed, 1541 insertions(+) > create mode 100644 src/PVE/API2/Network/SDN/Microseg.pm > create mode 100644 src/PVE/API2/Network/SDN/Microseg/Assignment.pm > create mode 100644 src/PVE/API2/Network/SDN/Microseg/Bridge.pm > create mode 100644 src/PVE/API2/Network/SDN/Microseg/Group.pm > create mode 100644 src/PVE/API2/Network/SDN/Microseg/Makefile > create mode 100644 src/PVE/API2/Network/SDN/Microseg/Rule.pm > create mode 100644 src/PVE/Network/SDN/Microseg.pm > > > pve-manager: > > Hannes Laimer (6): > ui: sdn: add microsegmentation panel > ui: sdn: dry-run: show pending microseg diff > network: apply microseg state on reload > ui: sdn: zones: add vxlan-gbp checkbox to vxlan and evpn > ui: sdn: microseg: add tag matcher > ui: sdn: microseg: add name regex matcher > > PVE/API2/Network.pm | 4 + > www/manager6/Makefile | 9 + > www/manager6/Utils.js | 34 ++ > www/manager6/dc/Config.js | 8 + > www/manager6/form/MicrosegGroupSelector.js | 77 +++++ > www/manager6/form/MicrosegGuestSelector.js | 83 +++++ > www/manager6/sdn/MicrosegView.js | 143 ++++++++ > www/manager6/sdn/SdnDiffView.js | 24 ++ > www/manager6/sdn/microseg/AssignmentEdit.js | 245 ++++++++++++++ > www/manager6/sdn/microseg/Base.js | 137 ++++++++ > www/manager6/sdn/microseg/GroupEdit.js | 45 +++ > www/manager6/sdn/microseg/PolicyView.js | 181 ++++++++++ > www/manager6/sdn/microseg/RuleEdit.js | 104 ++++++ > www/manager6/sdn/microseg/Tree.js | 356 ++++++++++++++++++++ > www/manager6/sdn/zones/EvpnEdit.js | 8 + > www/manager6/sdn/zones/VxlanEdit.js | 11 + > 16 files changed, 1469 insertions(+) > create mode 100644 www/manager6/form/MicrosegGroupSelector.js > create mode 100644 www/manager6/form/MicrosegGuestSelector.js > create mode 100644 www/manager6/sdn/MicrosegView.js > create mode 100644 www/manager6/sdn/microseg/AssignmentEdit.js > create mode 100644 www/manager6/sdn/microseg/Base.js > create mode 100644 www/manager6/sdn/microseg/GroupEdit.js > create mode 100644 www/manager6/sdn/microseg/PolicyView.js > create mode 100644 www/manager6/sdn/microseg/RuleEdit.js > create mode 100644 www/manager6/sdn/microseg/Tree.js > > > pve-docs: > > Hannes Laimer (2): > sdn: add microsegmentation section > sdn: add VXLAN-GBP flag to evpn/vxlan zone sections > > pvesdn.adoc | 134 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 134 insertions(+) > > > Summary over all repositories: > 64 files changed, 8268 insertions(+), 1 deletions(-) >