From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id D55081FF0AE for ; Tue, 27 Oct 2026 09:56:37 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id DDC2221669; Tue, 27 Oct 2026 09:56:15 +0100 (CET) From: Hannes Laimer To: pve-devel@lists.proxmox.com Subject: [PATCH access-control/cluster/container/docs/manager/network/proxmox{-ebpf,-ve-rs,-perl-rs}/qemu-server v3 00/45] sdn: add microsegmentation support Date: Tue, 27 Oct 2026 09:54:42 +0100 Message-ID: <20261027085527.702738-1-h.laimer@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793091333196 X-SPAM-LEVEL: Spam detection results: 0 AWL -2.218 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_DBL_SPAM 5 Contains a spam URL listed in the Spamhaus DBL blocklist [sdn.pm] Message-ID-Hash: 4I2B5XUHN5SP22ZKHDVAHKOLZWHBLTJW X-Message-ID-Hash: 4I2B5XUHN5SP22ZKHDVAHKOLZWHBLTJW X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This series adds group-based microsegmentation to SDN, enforced by eBPF programs on the guest interfaces. Assignments put guest interfaces into groups, and rules between groups decide which traffic passes. Sets that every rule treats the same share one identity, a 16-bit value that travels with the frame in the low bits of the packet mark. The host stamps it where a frame comes off the guest interface, so a guest cannot forge its own. A new zone option turns on VXLAN-GBP[2] for VXLAN and EVPN zones, which carries the identity across nodes. On a zone without it, traffic from another node arrives without an identity. An identity whose sets are gone after a config change is retired. Its value is reused only after a quarantine, since a frame stamped under the old config can still be in flight. The config in /etc/pve/sdn/microseg.cfg, with its API under /cluster/sdn/microseg and the ACL path /sdn/microseg, holds groups, rules and assignments and takes effect with an SDN apply. Groups can nest, and a member of a subgroup is in every group above it. A rule allows or denies traffic from a source predicate to a destination predicate at a priority. A predicate of the kind `any`, `all` or `exact` lists groups and matches an interface that has one of them, all of them, or those and no others. An `every` predicate lists none and matches each interface that has a group. Beyond the pair, a rule can be narrowed to a protocol and ports, an ICMP type, a firewall macro or non-first fragments. Which groups an interface has is determined by the assignments, an ordered list that matches interfaces by guest, tag or name regex. An interface takes the groups of the first one that covers it. Default is deny. Only interfaces that have at least one group assigned are enforced. Enforcement happens when a frame arrives at an enforced interface, so nothing is dropped on the way out of a guest's interface. Traffic from interfaces that are not enforced is untagged, which includes guests without groups on their NICs, the host and the outside. Such traffic has to be allowed explicitly with the built-in untagged group. Rules are stateless and directional, so a reply needs its own allow, or the rule is marked bidirectional. Stateless also means no reassembly, so fragments after the first carry no transport header. A narrowed rule matches them only when it is narrowed to fragments. The highest priority wins. At the same priority the rule with the more specific `all` or `exact` predicate decides, so a rule on a subgroup outranks the same rule on its parent. Equally specific rules deny if one of them denies. Narrowed rules are checked before the rules that decide the whole pair at the same priority. A guest interface gets its groups the moment it comes up, from the assignment that matches the guest at that moment. Changing the tags or the name of a running guest takes effect right after the config is written. The NICs get the groups the assignments now give them. Once no assignment matches the guest, their enforcement stops. A change that reaches the config another way, such as a restore or a migration, takes effect the next time the NIC comes up or at the next SDN apply. A tag or name the assignments match moves a guest between groups. So adding or removing such a tag, or renaming into or out of such a name, needs SDN.Use on /sdn/microseg besides the right to change the option. Putting a NIC on a vnet needs SDN.Use on the vnet the same way. As long as the policy has an assignment, every node keeps the programs and maps loaded, about 20 MiB plus what the pairs take. Without one, the nodes drop them. The BPF state does not survive a reboot, so a node applies the policy again at boot, before any guest starts. The programs are compiled into libpve-rs-perl, so an upgrade of that package applies the policy again as well. The SDN apply checks the policy against what a node can hold before it writes the running config, and refuses one that exceeds it. A node that cannot apply the policy fails its network reload task and keeps its previous enforcement. A node without loaded programs, as after a failed load at boot, starts no guest with an assigned interface until an apply or a reload succeeds there. A guest also does not start when its node cannot read or parse the running config, or cannot set up the enforcement of its assigned interface. ARP and neighbor solicitations/advertisements pass wherever either direction of a pair allows anything, so the two ends of a one-way allow can resolve each other's addresses. They also pass when the host itself sends them without an identity. That is how the suppression replies of an EVPN zone reach the guest. A frame the host turns around on the interface it came in on passes as a whole, since only the host can make one. Behind a firewall bridge the host's frames cannot be told from other unstamped ones, so an interface there takes these frames from every unstamped sender. The narrowed rules of a pair run first, so a narrowed rule that denies neighbor discovery still holds. In patch order, the series adds - proxmox-ebpf, the policy subsystem with the tc programs that stamp and check the identities - proxmox-ve-rs, the types, the identity engine and the desired state in proxmox-ve-config, and the new proxmox-microseg crate that hands the desired state to the subsystem - pve-cluster, microseg.cfg as an observed file - pve-access-control, the /sdn/microseg ACL path - proxmox-perl-rs, the config binding and the data plane binding in pve-rs - pve-network, the config, the API, the dry-run view, the zone option and the enforcement on interface plug and after a library upgrade - pve-manager, the panel, the dry-run view and the apply on network reload and at boot - qemu-server and pve-container, the immediate update of a running guest's NICs on a tag or name change, and SDN.Use for a tag or name the policy matches - pve-docs, documentation - pve-network needs pve-cluster, pve-access-control, pve-rs, pve-guest-common and ifupdown2 with the vxlan-gbp attribute - pve-manager needs pve-docs to build and pve-network and pve-access-control at runtime - pve-rs needs proxmox-microseg and proxmox-ve-config - qemu-server and pve-container need pve-network - proxmox-microseg needs proxmox-ebpf[1] and proxmox-ve-config The ifupdown2 vxlan-gbp attribute[2] is sent separately, and so is the pve-network patch that writes a VXLAN zone's local tunnel address[3]. This series needs both. A GBP device that ifupdown2 changes in place drops every GBP frame afterward[4], and without[3] that happens when a VXLAN zone with GBP exists next to an EVPN zone. ifupdown2 gives the local tunnel address of the EVPN zone's interfaces to every VXLAN interface without one of its own, and on a reload sets that address on the VXLAN zone's live device. [1] https://lore.proxmox.com/pve-devel/20261019142413.2255644-1-h.laimer@proxmox.com/ [2] https://lore.proxmox.com/pve-devel/20260828134708.692354-1-h.laimer@proxmox.com/ [3] https://lore.proxmox.com/pve-devel/20260925134050.754528-1-h.laimer@proxmox.com/ [4] https://lore.kernel.org/netdev/20261023-b4-vxlan-changelink-gbp-v1-0-f0d13951c09b@proxmox.com/ note: I did also have this implemented with nft instead of ebpf as a dataplane. Without narrowing, packets per second (small UDP packets) was about the same, with multiple checks nft's rate went down, nothing super dramatic but about a 1.5-2x compared to eBPF (~140k vs ~250k). eBPF is also more flexible, in case we want to expand possible actions in the future, but replacing the dataplane is pretty simple which also allowed me to do the side-by-side testing. changelog v3: - the data plane is a library crate in the proxmox-ve-rs workspace that pve-rs binds, on a policy subsystem in proxmox-ebpf, no agent binary and no boot unit. A node loads the programs while the policy has an assignment and takes new programs on a package upgrade - the model changed in five places: * groups have no numeric mark and can form a hierarchy * assignments are an ordered list and an interface takes the groups of the first that covers it instead of the union of all * rules can be narrowed similar to how firewall rules work, additionally non-first fragments can be matched and an explicit bidirectional flag exists * at a priority tie the more specific rule decides * new `every` predicate that matches everything that has at least one group assigned - interface group (re)assignment happens whenever an interface comes up, sdn apply runs or the tags or name of a running guest change. Adding or removing a tag covered by an assignment, or renaming into or out of a name a regex assignment matches, now also needs SDN.Use on the policy, similar to how we check if a user is allowed to assign a vnet - if a node cannot apply the policy, its network reload task fails. while a node has no loaded programs, as after a failed load at boot, guests with an assigned NIC do not start on it. a nic with groups only comes up enforced - ARP and NS/NA are automatically allowed if there exists a (src,dst) -> [checks ->] allow in either direction, this is so address resolution for allowed connections is possible. Also host-generated ARP and NS/NA frames are allowed, this is specifically needed for EVPN suppression to work, and a frame the host turns around on the interface it came in on passes - api: section types and property keys are kebab-case, the endpoints check /sdn/microseg (a pve-access-control patch adds the ACL path), and the overview reports members, evaluation order and specificity instead of a realized state, plus the id space usage - rule and assignment ids are derived from their identifying fields, so those cannot change on an existing object - ui: a panel with the rule grid by priority, the assignments in match order, the dry-run view, the rule editor on the firewall's selectors, and a guest tab with its groups and rules - the SRv6 carrier and its bridge object are out, the ifupdown2 vxlan-gbp attribute is a standalone patch, the zone option stays, docs rewritten v2: - core model reworked: each interface now holds a *set* of groups compiled into a per-NIC wire identity, replacing the single-group parent/child tree - rules gained explicit `src`/`dst` predicates (any/all/exact) and a `prio`, replacing the implicit tree-distance ordering - `untagged` is now a nameable built-in group (mark 0) - assignments are additive: a NIC's set is the union of all that resolve to it - added tag-matcher and name-regex assignments - rule and dynamic-assignment ids are now generated - identity allocation is fresh-first with quarantined reclamation, so churn cannot exhaust the id space - split the SRv6 carrier support into droppable tail patches - updated docs - updated cover-letter proxmox-ebpf: Hannes Laimer (3): build: keep the build directory out of the BPF objects policy: add the enforcement subsystem for guest interfaces policy: add a check for non-first fragments Cargo.toml | 1 + build.rs | 4 + debian/control | 6 +- src/bpf-shim/bpf/bpf_helpers.h | 1 + src/bpf-shim/vmlinux.h | 1 + src/lib.rs | 3 + src/policy/bpf/mark.h | 27 + src/policy/bpf/policy.bpf.c | 385 ++++++++++++ src/policy/bpf/types.h | 75 +++ src/policy/mod.rs | 1005 ++++++++++++++++++++++++++++++++ src/policy/types.rs | 159 +++++ src/subsystem.rs | 35 +- tests/common/mod.rs | 40 ++ tests/policy.rs | 837 ++++++++++++++++++++++++++ 14 files changed, 2573 insertions(+), 6 deletions(-) create mode 100644 src/policy/bpf/mark.h create mode 100644 src/policy/bpf/policy.bpf.c create mode 100644 src/policy/bpf/types.h create mode 100644 src/policy/mod.rs create mode 100644 src/policy/types.rs create mode 100644 tests/policy.rs proxmox-ve-rs: Hannes Laimer (11): ve-config: firewall: add a constructor for sctp matches ve-config: sdn: microseg: add identity allocation for group sets ve-config: sdn: microseg: add config types ve-config: sdn: microseg: add tag matcher ve-config: sdn: microseg: add name regex matcher ve-config: sdn: microseg: narrow rules by protocol and ports ve-config: sdn: microseg: add group hierarchy ve-config: firewall: expose protocol, icmp type and code numbers ve-config: sdn: microseg: add the desired state microseg: add a crate that applies the policy to a node's guest NICs microseg: add a rule option that matches non-first fragments Cargo.toml | 2 + proxmox-microseg/Cargo.toml | 20 + proxmox-microseg/debian/changelog | 5 + proxmox-microseg/debian/control | 46 + proxmox-microseg/debian/copyright | 18 + proxmox-microseg/debian/debcargo.toml | 7 + proxmox-microseg/debian/source/format | 1 + proxmox-microseg/src/ebpf.rs | 247 +++ proxmox-microseg/src/lib.rs | 66 + proxmox-microseg/src/nic.rs | 120 ++ proxmox-microseg/src/running_config.rs | 26 + proxmox-ve-config/src/firewall/fw_macros.rs | 7 + proxmox-ve-config/src/firewall/ports.rs | 65 + .../src/firewall/types/rule_match.rs | 63 + proxmox-ve-config/src/sdn/microseg/api.rs | 710 +++++++++ .../src/sdn/microseg/identity.rs | 475 ++++++ proxmox-ve-config/src/sdn/microseg/mod.rs | 1394 +++++++++++++++++ proxmox-ve-config/src/sdn/microseg/state.rs | 616 ++++++++ proxmox-ve-config/src/sdn/microseg/types.rs | 549 +++++++ proxmox-ve-config/src/sdn/mod.rs | 1 + 20 files changed, 4438 insertions(+) create mode 100644 proxmox-microseg/Cargo.toml create mode 100644 proxmox-microseg/debian/changelog create mode 100644 proxmox-microseg/debian/control create mode 100644 proxmox-microseg/debian/copyright create mode 100644 proxmox-microseg/debian/debcargo.toml create mode 100644 proxmox-microseg/debian/source/format create mode 100644 proxmox-microseg/src/ebpf.rs create mode 100644 proxmox-microseg/src/lib.rs create mode 100644 proxmox-microseg/src/nic.rs create mode 100644 proxmox-microseg/src/running_config.rs create mode 100644 proxmox-ve-config/src/sdn/microseg/api.rs create mode 100644 proxmox-ve-config/src/sdn/microseg/identity.rs create mode 100644 proxmox-ve-config/src/sdn/microseg/mod.rs create mode 100644 proxmox-ve-config/src/sdn/microseg/state.rs create mode 100644 proxmox-ve-config/src/sdn/microseg/types.rs pve-cluster: Hannes Laimer (1): cfs: add 'sdn/microseg.cfg' to observed files src/PVE/Cluster.pm | 1 + src/pmxcfs/status.c | 1 + 2 files changed, 2 insertions(+) pve-access-control: Hannes Laimer (1): permissions: add ACL path for SDN microsegmentation src/PVE/AccessControl.pm | 1 + 1 file changed, 1 insertion(+) proxmox-perl-rs: Hannes Laimer (4): pve-rs: sdn: add microseg config binding pve-rs: sdn: microseg: bind the data plane pve-rs: sdn: microseg: check the data plane limits on render pve-rs: debian: fire the pve-microseg-update trigger on upgrade pve-rs/Cargo.toml | 2 + pve-rs/Makefile | 3 + pve-rs/debian/control | 1 + pve-rs/debian/triggers | 1 + pve-rs/src/bindings/sdn/microseg.rs | 350 ++++++++++++++++++++++++++++ pve-rs/src/bindings/sdn/mod.rs | 1 + 6 files changed, 358 insertions(+) create mode 100644 pve-rs/src/bindings/sdn/microseg.rs pve-network: Hannes Laimer (10): sdn: zones: add vxlan-gbp option to vxlan and evpn zones sdn: microseg: add config, API and guest inventory sdn: dry-run: show pending microseg changes sdn: microseg: enforce the policy on guest NICs sdn: microseg: add tag matcher sdn: microseg: add name regex matcher sdn: microseg: add an update for a running NIC sdn: microseg: expose protocol, port and macro rule options sdn: microseg: allow nesting groups under a parent sdn: microseg: expose the fragments rule option debian/control | 1 + debian/libpve-network-perl.postinst | 21 + debian/triggers | 1 + src/PVE/API2/Network/SDN.pm | 47 +- src/PVE/API2/Network/SDN/Makefile | 2 + src/PVE/API2/Network/SDN/Microseg.pm | 236 +++++++++ .../API2/Network/SDN/Microseg/Assignment.pm | 23 + src/PVE/API2/Network/SDN/Microseg/Base.pm | 382 ++++++++++++++ src/PVE/API2/Network/SDN/Microseg/Group.pm | 17 + src/PVE/API2/Network/SDN/Microseg/Makefile | 11 + src/PVE/API2/Network/SDN/Microseg/Rule.pm | 22 + src/PVE/API2/Network/SDN/Zones.pm | 8 + src/PVE/Network/SDN.pm | 25 + src/PVE/Network/SDN/Makefile | 1 + src/PVE/Network/SDN/Microseg.pm | 494 ++++++++++++++++++ src/PVE/Network/SDN/Zones.pm | 32 ++ src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 8 + src/PVE/Network/SDN/Zones/VxlanPlugin.pm | 12 + .../zones/evpn/gbp/expected_controller_config | 77 +++ .../zones/evpn/gbp/expected_sdn_interfaces | 45 ++ src/test/zones/evpn/gbp/interfaces | 7 + src/test/zones/evpn/gbp/sdn_config | 26 + .../zones/vxlan/gbp/expected_sdn_interfaces | 17 + src/test/zones/vxlan/gbp/interfaces | 7 + src/test/zones/vxlan/gbp/sdn_config | 11 + 25 files changed, 1531 insertions(+), 2 deletions(-) create mode 100644 src/PVE/API2/Network/SDN/Microseg.pm create mode 100644 src/PVE/API2/Network/SDN/Microseg/Assignment.pm create mode 100644 src/PVE/API2/Network/SDN/Microseg/Base.pm create mode 100644 src/PVE/API2/Network/SDN/Microseg/Group.pm create mode 100644 src/PVE/API2/Network/SDN/Microseg/Makefile create mode 100644 src/PVE/API2/Network/SDN/Microseg/Rule.pm create mode 100644 src/PVE/Network/SDN/Microseg.pm create mode 100644 src/test/zones/evpn/gbp/expected_controller_config create mode 100644 src/test/zones/evpn/gbp/expected_sdn_interfaces create mode 100644 src/test/zones/evpn/gbp/interfaces create mode 100644 src/test/zones/evpn/gbp/sdn_config create mode 100644 src/test/zones/vxlan/gbp/expected_sdn_interfaces create mode 100644 src/test/zones/vxlan/gbp/interfaces create mode 100644 src/test/zones/vxlan/gbp/sdn_config pve-manager: Hannes Laimer (11): network: apply the microseg policy on reload pve-sdn-commit: apply the microseg policy at boot ui: sdn: zones: add vxlan-gbp checkbox to vxlan and evpn ui: sdn: add microsegmentation panel ui: sdn: dry-run: show pending microseg diff ui: sdn: microseg: add tag matcher ui: sdn: microseg: add name regex matcher ui: sdn: microseg: add narrowing to the rule form ui: sdn: microseg: show and edit the group hierarchy ui: sdn: microseg: add fragments option to the rule form ui: sdn: microseg: add a guest tab with its groups and rules PVE/API2/Network.pm | 31 +- bin/pve-sdn-commit | 47 ++- services/pve-sdn-commit.service | 1 + www/css/ext6-pve.css | 23 ++ www/manager6/Makefile | 10 + www/manager6/Utils.js | 98 +++++ www/manager6/data/PermPathStore.js | 1 + www/manager6/dc/Config.js | 8 + www/manager6/form/SDNMicrosegGroupSelector.js | 105 ++++++ www/manager6/grid/FirewallRules.js | 2 + www/manager6/lxc/Config.js | 10 + www/manager6/qemu/Config.js | 10 + www/manager6/sdn/MicrosegView.js | 98 +++++ www/manager6/sdn/SdnDiffView.js | 29 ++ www/manager6/sdn/microseg/AssignmentEdit.js | 148 ++++++++ www/manager6/sdn/microseg/AssignmentView.js | 195 ++++++++++ www/manager6/sdn/microseg/Base.js | 262 ++++++++++++++ www/manager6/sdn/microseg/GroupEdit.js | 40 +++ www/manager6/sdn/microseg/GuestPanel.js | 117 ++++++ www/manager6/sdn/microseg/RuleEdit.js | 285 +++++++++++++++ www/manager6/sdn/microseg/RuleView.js | 338 ++++++++++++++++++ www/manager6/sdn/microseg/Tree.js | 278 ++++++++++++++ www/manager6/sdn/zones/EvpnEdit.js | 8 + www/manager6/sdn/zones/VxlanEdit.js | 11 + 24 files changed, 2126 insertions(+), 29 deletions(-) create mode 100644 www/manager6/form/SDNMicrosegGroupSelector.js create mode 100644 www/manager6/sdn/MicrosegView.js create mode 100644 www/manager6/sdn/microseg/AssignmentEdit.js create mode 100644 www/manager6/sdn/microseg/AssignmentView.js create mode 100644 www/manager6/sdn/microseg/Base.js create mode 100644 www/manager6/sdn/microseg/GroupEdit.js create mode 100644 www/manager6/sdn/microseg/GuestPanel.js create mode 100644 www/manager6/sdn/microseg/RuleEdit.js create mode 100644 www/manager6/sdn/microseg/RuleView.js create mode 100644 www/manager6/sdn/microseg/Tree.js qemu-server: Hannes Laimer (1): network: update the microseg enforcement on a tag or name change src/PVE/API2/Qemu.pm | 26 ++++++++++++++++++++++++++ src/PVE/QemuServer.pm | 12 +++++++++++- src/PVE/QemuServer/Network.pm | 14 ++++++++++++++ 3 files changed, 51 insertions(+), 1 deletion(-) pve-container: Hannes Laimer (1): lxc: update the microseg enforcement on a tag or hostname change src/PVE/API2/LXC/Config.pm | 3 +++ src/PVE/LXC.pm | 28 ++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) pve-docs: Hannes Laimer (2): sdn: zones: document the VXLAN-GBP option sdn: add microsegmentation section pvesdn.adoc | 403 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 403 insertions(+) Summary over all repositories: 98 files changed, 11514 insertions(+), 38 deletions(-) -- Generated by murpp 0.12.0