From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 19F8C1FF0E0 for ; Thu, 09 Jul 2026 11:19:07 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5C9042146D; Thu, 09 Jul 2026 11:19:06 +0200 (CEST) From: Hannes Laimer To: pve-devel@lists.proxmox.com subject: SPAM: [RFC cluster/docs/ifupdown2/manager/network/proxmox{-ve-rs,-ebpf,-perl-rs} v2 00/27] sdn: add microsegmentation support Date: Thu, 9 Jul 2026 11:18:25 +0200 Message-ID: <20260709091852.538885-1-h.laimer@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1783588726957 X-SPAM-LEVEL: Spam detection results: 3 DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLACK 3 Contains an URL listed in the URIBL blacklist [types.rs] Message-ID-Hash: BS537XN54GFBF75CNJNOFLAKCRIOUB2J X-Message-ID-Hash: BS537XN54GFBF75CNJNOFLAKCRIOUB2J X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This adds support for microsegmentation using eBPF programs attached to interfaces. Mostly the tap/veth interfaces on the guests directly. # Overview Each guest interface is assigned a *set* of groups. Rather than a hierarchy, group membership is treated as a signature. The agent compiles each interface's group set into a single per-NIC *identity*, and policies are rules between groups, matched against those sets with predicates. A rule is a `src` predicate and a `dst` predicate plus a verdict and a priority, where a predicate is a match kind over a list of groups: - any{...} the interface shares at least one of the listed groups - all{...} the interface has all of the listed groups - exact{...} the interface's groups are exactly the listed set For a packet, the `src` predicate is evaluated against the sender's set and the `dst` predicate against the receiver's. Among the rules that fire the highest `prio` wins, a conflicting tie (with different verdicts) denies, and if no rule fires the packet is denied (default-deny). For example, with the groups web/app/db/prod, specific allows punched through a broad deny: - any{web} -> any{app} : allow prio 10 - any{app} -> any{db} : allow prio 10 - all{prod} -> all{prod} : allow prio 10 # these three carve out east-west... - any{web,app,db} -> any{web,app,db} : deny prio 5 # ...through a broad tier-to-tier deny (the last is technically not needed cause the default is deny, in a larger context having this explicitly can be necesarry though) The allows above are one-directional. Rules are stateless and per-packet, there is no connection tracking. The reply needs its own allow, so for TCP between web and app both any{web} -> any{app} and any{app} -> any{web} have to be allowed. Groups land on an interface either directly (a guest matcher: a `vmid`, optionally a single `iface`), by tag (a tag matcher: every guest carrying the named tags, e.g. `prod`/`staging`/`web`), or by name (a regex matcher: every guest whose name matches a regular expression). Assignments are additive, an interface's set is the union of every assignment that resolves to it, so the matchers compose. `untagged` is a reserved built-in group (mark 0) for traffic that arrives unstamped (gateways, DHCP/ARP, cross-node frames without a tag). It can be named in a predicate and composed with real groups (e.g. `any{untagged, web}`), but it is not assignable. Enforcement happens on the receiving side, cause that's where we for sure know both - where the packet is from - and, its destination (us) # config and API The config is a single section-config, `sdn/microseg.cfg`. It holds these object types, keyed by section id: - group, a numeric `mark` (auto-assigned if omitted) and an optional comment - rule, a `src`/`dst` predicate pair (`*_match` is any/all/exact over a group list), a `prio` and `allow` - guestassignment, binds a guest's NIC(s) (`vmid`, optional `iface`) to a group set - tagassignment, binds the NICs of every guest matching `tags` (by `tag_match`) to a group set, expanded against the live guest inventory on render - regexassignment, same but for every guest whose name matches a regular expression - bridge, marks a bridge-facing interface as an SRv6 carrier (no UI yet) The API exposes endpoints under `/cluster/sdn/microseg/{group,rule,assignment,bridge}`, guarded by SDN.Audit for reads and SDN.Allocate for writes. The assignment endpoint discriminates the two matcher kinds on a `type` property (like the fabric `protocol`). Rule and tag/regex-assignment ids are derived from their contents (an FNV-1a hash), guest assignments from `vm{vmid}i{iface}`, groups and bridges take a chosen id. On commit the config is rendered into `.running-config`, which is what the agent reads. Rendering resolves each interface's group set to a wire identity and expands the dynamic matchers against the inventory. # identity An identity is not one id per group set but one id per *signature*. Two interfaces share an id when every rule predicate fires the same way on both, so no rule can tell them apart. The identities are the classes of that partition, each named by a 16-bit id, and that id is what rides on the wire. Editing rules reshapes the partition. A new predicate that separates two sets which used to look alike *splits* their class in two, and dropping the predicate that told two classes apart *merges* them back into one. On a split the subclass that still carries the old representative keeps the id and the other gets a fresh one, on a merge the surviving class keeps its id and the fused-away ones retire. New ids come fresh-first from a counter. A retired id is not reused right away, it sits in a time quarantine long enough that no packet stamped under its old meaning can still be in flight. Only once the fresh space is exhausted does allocation fall back to reusing a retired id whose quarantine has passed, so config churn alone cannot exhaust the 16-bit id space. Actually running out would take more than 65534 identities the policies can tell apart. So it doesn't scale with count of group combinations in use, but with the distinct semantic meaning they carry. # skb->mark Every packet in the kernel lives in a `sk_buff` struct, which has a 32-bit `mark` field you can read and write while the packet moves through the kernel. Nothing else in our SDN stack touches `mark` today, so microseg has it to itself. Worth keeping in mind if that ever changes. For now we only use the lower 16 bits, to carry the interface's identity (mark 0 is reserved for unstamped traffic), so up to 65535 identities. The catch is `mark` only lives as long as the packet stays in the kernel, and sooner or later it leaves. To carry the identity between hosts we have to put it on the wire itself. For that we support two carriers: - VXLAN-GBP, the 16-bit GBP field on the VXLAN encapsulation - SRv6, the 16-bit Tag field on the SRH The kernel already moves `mark` in and out of the VXLAN-GBP field for us (found that out way too late :P). For SRv6 we do it ourselves with a small eBPF program. # eBPF Both tagging (setting `mark`) and enforcement happen in an eBPF program directly attached to the guest's interface. The programs themselves read and write to/from two maps: - tap_to_group, the identity to stamp for a given interface - rules, a map of (src_id, dst_id) -> action, where src is the `mark` the packet carries and dst is our own interface's identity Specifically, on ingress we set the mark, and on egress we enforce. # Implementation We have an `agent` that reads the running sdn config, and applies that state to the kernel. The binary is stateless and one-shot, not a long running daemon. It runs on SDN apply, on tap_plug of a guest interface, and on boot after pve-sdn-commit but before guests start. Those triggers cover every situation where we have to touch kernel state. It compiles the config into what the data plane needs. It folds each interface's group set into a wire identity (a registry kept stable across renders so identities agree across hosts, allocated as described above), and folds the rules into the `(src_id, dst_id)` action map by priority (highest wins, conflicting ties deny). It keeps track of what is currently loaded in the kernel with two files under `/run/proxmox-ebpf`. One keeping track of the bpf program that was compiled into the binary, and one for keeping track of changes to the data structures the bpf program accesses. The distinction is useful cause for only a program change we can swap the currently attached program with the updated one atomically. In case the data structure (the maps) changed there is no other way than to tear-down all the current state and repopulate the maps and re-attach the programs. Swapping out the maps first would have old programs interact with a new data schema, and swapping the programs first will lead to new programs accessing an old data schema. Either way, not good, so we wipe the state in that case and re-build it. So, in case we change the structure of the data our bpf programs access, there will be a bunch of ms where the configured segmentation is not enforced. But that is the only scenario where that can happen. ## aya, aya-ebpf Aya is a rust lib that helps with working with BPF more easily. It has both a userspace part (`aya`) and a part (`aya-ebpf`) that compiles to bpf. I chose to only use the userspace part, and use C for the BPF programs themselves. The reasons were: - we don't have `aya-ebpf` packaged - `aya-ebpf` *requires* nightly toolchain to compile - the bpf programs are very small, in case this should change at some point we can reconsider. Also, it doesn't matter what produces the .o file, so this could be swapped in really easily if we decided to So we compile the bpf program written in C to a .o targeting bpf using clang and include it when compiling the agent binary. ## SRv6 Besides VXLAN-GBP, the identity can also ride in the 16-bit Tag field of an SRv6 SRH. The kernel already does this move for VXLAN-GBP but not for SRv6, so this adds a small eBPF program that writes the identity into the SRH on egress and reads it back on ingress. The `bridge` object marks the bridge-facing interface it runs on. This is only the tag carrier, not an SRv6 transport. Our SDN stack has no SRv6 zone yet, so there is nothing configured to route it over, and there is no UI. If SDN grows an SRv6 transport later (a small L3, EVPN-lite style option), the carrier is already in place. It is packaged as a droppable tail. The carrier bridge section (proxmox-ve-config), the bridge API (pve-network) and the bridge subsystem (proxmox-ebpf) are each the last patch of their repo's series, so SRv6 can be deferred without touching the rest. # enabling VXLAN-GBP The kernel only moves the mark into the GBP field if the vxlan device was created with the GBP flag set, and it's create-only, the kernel won't toggle it on a running device. ifupdown2 had no attribute for this, so this series includes a small ifupdown2 patch adding `vxlan-gbp`, which threads the flag through to the netlink/iproute2 create path. # testing I have put pre-built packages on sani, these include ifupdown2 with the patch # building 1. proxmox-ve-rs (install) 2. proxmox-ebpf 3. pve-cluster (install) 4. proxmox-perl-rs (install) 5. pve-network 6. pve-manager # notes * everything eBPF related did not change since v1, this v2 only really changes identity semantics * using GENEVE would also be an option, but with this current approach we gain a lot of flexibility. Expanding predicate logic, or having not just allow/deny but also match ports or alike is possible without touching the underlying identity system. (limiting the per-packet overhead to just 16 bits is also good I'd argue) * the enforement mechanism is not set in stone, weather this is eBPF or maybe nftables with a map we populate as shortly discussed with @Stefan. The current approach plays nicely with both since it conveys identity through `skb->mark` that nft can access as well * I did not send the first commit for `proxmox-ebpf` since it contains a `vmlinux.h` which is rather large but is needed for compiling. The commit is in my staff repo. # changelog v2: - core model reworked: each interface now holds a *set* of groups compiled into a per-NIC wire identity, replacing the single-group parent/child tree - rules gained explicit `src`/`dst` predicates (any/all/exact) and a `prio`, replacing the implicit tree-distance ordering - `untagged` is now a nameable built-in group (mark 0) - assignments are additive: a NIC's set is the union of all that resolve to it - added tag-matcher and name-regex assignments - rule and dynamic-assignment ids are now generated - identity allocation is fresh-first with quarantined reclamation, so churn cannot exhaust the id space - split the SRv6 carrier support into droppable tail patches - updated docs - updated cover-letter proxmox-ve-rs: Hannes Laimer (5): ve-config: sdn: add microseg signature-identity engine ve-config: sdn: add microseg config types ve-config: sdn: microseg: add tag matcher ve-config: sdn: microseg: add name regex matcher ve-config: sdn: microseg: add carrier bridge section proxmox-ve-config/src/sdn/config.rs | 9 +- .../src/sdn/microseg/identity.rs | 624 +++++ proxmox-ve-config/src/sdn/microseg/mod.rs | 2349 +++++++++++++++++ proxmox-ve-config/src/sdn/mod.rs | 1 + 4 files changed, 2982 insertions(+), 1 deletion(-) create mode 100644 proxmox-ve-config/src/sdn/microseg/identity.rs create mode 100644 proxmox-ve-config/src/sdn/microseg/mod.rs proxmox-ebpf: Hannes Laimer (3): agent: add userspace coordinator and stateless policy subsystem bpf: add bridge subsystem debian: add packaging and boot-time oneshot unit Makefile | 66 +++++++ debian/changelog | 5 + debian/control | 35 ++++ debian/copyright | 18 ++ debian/proxmox-ebpf.install | 1 + debian/proxmox-ebpf.postrm | 11 ++ debian/proxmox-ebpf.prerm | 12 ++ debian/proxmox-ebpf.service | 15 ++ debian/rules | 33 ++++ debian/source/format | 1 + include/mark.h | 30 +++ src/agent.rs | 99 ++++++++++ src/bridge/bpf/srv6.bpf.c | 76 +++++++ src/bridge/mod.rs | 76 +++++++ src/main.rs | 69 +++++++ src/policy/bpf/tap.bpf.c | 68 +++++++ src/policy/bpf/types.h | 23 +++ src/policy/mod.rs | 268 +++++++++++++++++++++++++ src/policy/types.rs | 45 +++++ src/running_config.rs | 38 ++++ src/state.rs | 149 ++++++++++++++ src/subsystem.rs | 385 ++++++++++++++++++++++++++++++++++++ src/tc.rs | 152 ++++++++++++++ 23 files changed, 1675 insertions(+) create mode 100644 Makefile create mode 100644 debian/changelog create mode 100644 debian/control create mode 100644 debian/copyright create mode 100644 debian/proxmox-ebpf.install create mode 100755 debian/proxmox-ebpf.postrm create mode 100755 debian/proxmox-ebpf.prerm create mode 100644 debian/proxmox-ebpf.service create mode 100755 debian/rules create mode 100644 debian/source/format create mode 100644 include/mark.h create mode 100644 src/agent.rs create mode 100644 src/bridge/bpf/srv6.bpf.c create mode 100644 src/bridge/mod.rs create mode 100644 src/main.rs create mode 100644 src/policy/bpf/tap.bpf.c create mode 100644 src/policy/bpf/types.h create mode 100644 src/policy/mod.rs create mode 100644 src/policy/types.rs create mode 100644 src/running_config.rs create mode 100644 src/state.rs create mode 100644 src/subsystem.rs create mode 100644 src/tc.rs pve-cluster: Hannes Laimer (1): cfs: add 'sdn/microseg.cfg' to observed files src/PVE/Cluster.pm | 1 + 1 file changed, 1 insertion(+) proxmox-perl-rs: Hannes Laimer (1): pve-rs: sdn: add microseg config binding pve-rs/Makefile | 1 + pve-rs/src/bindings/sdn/microseg.rs | 235 ++++++++++++++++++++++++++++ pve-rs/src/bindings/sdn/mod.rs | 1 + 3 files changed, 237 insertions(+) create mode 100644 pve-rs/src/bindings/sdn/microseg.rs ifupdown2: Hannes Laimer (1): d/patches: add support for VXLAN-GBP flag ...addons-vxlan-add-vxlan-gbp-attribute.patch | 228 ++++++++++++++++++ debian/patches/series | 1 + 2 files changed, 229 insertions(+) create mode 100644 debian/patches/pve/0016-addons-vxlan-add-vxlan-gbp-attribute.patch pve-network: Hannes Laimer (8): sdn: microseg: add config, API and guest inventory sdn: dry-run: surface pending microseg changes sdn: zones: trigger microseg apply on tap_plug sdn: zones: add vxlan-gbp option to vxlan and evpn zones evpn: disable vxlan-learning on create if GBP is enabled sdn: microseg: add tag matcher sdn: microseg: add name regex matcher sdn: microseg: add carrier bridge API src/PVE/API2/Network/SDN.pm | 31 ++ src/PVE/API2/Network/SDN/Makefile | 2 + src/PVE/API2/Network/SDN/Microseg.pm | 204 +++++++ .../API2/Network/SDN/Microseg/Assignment.pm | 185 +++++++ src/PVE/API2/Network/SDN/Microseg/Bridge.pm | 179 ++++++ src/PVE/API2/Network/SDN/Microseg/Group.pm | 179 ++++++ src/PVE/API2/Network/SDN/Microseg/Makefile | 8 + src/PVE/API2/Network/SDN/Microseg/Rule.pm | 180 +++++++ src/PVE/Network/SDN.pm | 37 ++ src/PVE/Network/SDN/Makefile | 1 + src/PVE/Network/SDN/Microseg.pm | 509 ++++++++++++++++++ src/PVE/Network/SDN/Zones.pm | 6 + src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 11 + src/PVE/Network/SDN/Zones/VxlanPlugin.pm | 9 + 14 files changed, 1541 insertions(+) create mode 100644 src/PVE/API2/Network/SDN/Microseg.pm create mode 100644 src/PVE/API2/Network/SDN/Microseg/Assignment.pm create mode 100644 src/PVE/API2/Network/SDN/Microseg/Bridge.pm create mode 100644 src/PVE/API2/Network/SDN/Microseg/Group.pm create mode 100644 src/PVE/API2/Network/SDN/Microseg/Makefile create mode 100644 src/PVE/API2/Network/SDN/Microseg/Rule.pm create mode 100644 src/PVE/Network/SDN/Microseg.pm pve-manager: Hannes Laimer (6): ui: sdn: add microsegmentation panel ui: sdn: dry-run: show pending microseg diff network: apply microseg state on reload ui: sdn: zones: add vxlan-gbp checkbox to vxlan and evpn ui: sdn: microseg: add tag matcher ui: sdn: microseg: add name regex matcher PVE/API2/Network.pm | 4 + www/manager6/Makefile | 9 + www/manager6/Utils.js | 34 ++ www/manager6/dc/Config.js | 8 + www/manager6/form/MicrosegGroupSelector.js | 77 +++++ www/manager6/form/MicrosegGuestSelector.js | 83 +++++ www/manager6/sdn/MicrosegView.js | 143 ++++++++ www/manager6/sdn/SdnDiffView.js | 24 ++ www/manager6/sdn/microseg/AssignmentEdit.js | 245 ++++++++++++++ www/manager6/sdn/microseg/Base.js | 137 ++++++++ www/manager6/sdn/microseg/GroupEdit.js | 45 +++ www/manager6/sdn/microseg/PolicyView.js | 181 ++++++++++ www/manager6/sdn/microseg/RuleEdit.js | 104 ++++++ www/manager6/sdn/microseg/Tree.js | 356 ++++++++++++++++++++ www/manager6/sdn/zones/EvpnEdit.js | 8 + www/manager6/sdn/zones/VxlanEdit.js | 11 + 16 files changed, 1469 insertions(+) create mode 100644 www/manager6/form/MicrosegGroupSelector.js create mode 100644 www/manager6/form/MicrosegGuestSelector.js create mode 100644 www/manager6/sdn/MicrosegView.js create mode 100644 www/manager6/sdn/microseg/AssignmentEdit.js create mode 100644 www/manager6/sdn/microseg/Base.js create mode 100644 www/manager6/sdn/microseg/GroupEdit.js create mode 100644 www/manager6/sdn/microseg/PolicyView.js create mode 100644 www/manager6/sdn/microseg/RuleEdit.js create mode 100644 www/manager6/sdn/microseg/Tree.js pve-docs: Hannes Laimer (2): sdn: add microsegmentation section sdn: add VXLAN-GBP flag to evpn/vxlan zone sections pvesdn.adoc | 134 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 134 insertions(+) Summary over all repositories: 64 files changed, 8268 insertions(+), 1 deletions(-) -- Generated by murpp 0.12.0