superseded: [PATCH access-control/cluster/container/docs/manager/network/proxmox{-ebpf,-ve-rs,-perl-rs}/qemu-server v3 00/45] sdn: add microsegmentation support

[Hannes Laimer <h.laimer@proxmox.com>]

superseded-by:
https://lore.proxmox.com/pve-devel/20261105111324.706922-1-h.laimer@proxmox.com

On 2026-10-27 09:56, Hannes Laimer wrote:
> This series adds group-based microsegmentation to SDN, enforced by eBPF
> programs on the guest interfaces. Assignments put guest interfaces into
> groups, and rules between groups decide which traffic passes. Sets that
> every rule treats the same share one identity, a 16-bit value that
> travels with the frame in the low bits of the packet mark. The host
> stamps it where a frame comes off the guest interface, so a guest cannot
> forge its own. A new zone option turns on VXLAN-GBP[2] for VXLAN and
> EVPN zones, which carries the identity across nodes. On a zone without
> it, traffic from another node arrives without an identity. An identity
> whose sets are gone after a config change is retired. Its value is
> reused only after a quarantine, since a frame stamped under the old
> config can still be in flight.
> 
> The config in /etc/pve/sdn/microseg.cfg, with its API under
> /cluster/sdn/microseg and the ACL path /sdn/microseg, holds groups,
> rules and assignments and takes effect with an SDN apply. Groups can
> nest, and a member of a subgroup is in every group above it. A rule
> allows or denies traffic from a source predicate to a destination
> predicate at a priority. A predicate of the kind `any`, `all` or
> `exact` lists groups and matches an interface that has one of them, all
> of them, or those and no others. An `every` predicate lists none and
> matches each interface that has a group. Beyond the pair, a rule can be
> narrowed to a protocol and ports, an ICMP type, a firewall macro or
> non-first fragments. Which groups an interface has is determined by the
> assignments, an ordered list that matches interfaces by guest, tag or
> name regex. An interface takes the groups of the first one that covers
> it.
> 
> Default is deny. Only interfaces that have at least one group assigned
> are enforced. Enforcement happens when a frame arrives at an enforced
> interface, so nothing is dropped on the way out of a guest's interface.
> Traffic from interfaces that are not enforced is untagged, which
> includes guests without groups on their NICs, the host and the outside.
> Such traffic has to be allowed explicitly with the built-in untagged
> group. Rules are stateless and directional, so a reply needs its own
> allow, or the rule is marked bidirectional. Stateless also means no
> reassembly, so fragments after the first carry no transport header. A
> narrowed rule matches them only when it is narrowed to fragments.
> 
> The highest priority wins. At the same priority the rule with the more
> specific `all` or `exact` predicate decides, so a rule on a subgroup
> outranks the same rule on its parent. Equally specific rules deny if one
> of them denies. Narrowed rules are checked before the rules that decide
> the whole pair at the same priority.
> 
> A guest interface gets its groups the moment it comes up, from the
> assignment that matches the guest at that moment. Changing the tags or
> the name of a running guest takes effect right after the config is
> written. The NICs get the groups the assignments now give them. Once no
> assignment matches the guest, their enforcement stops. A change that
> reaches the config another way, such as a restore or a migration, takes
> effect the next time the NIC comes up or at the next SDN apply. A tag
> or name the assignments match moves a guest between groups. So adding
> or removing such a tag, or renaming into or out of such a name, needs
> SDN.Use on /sdn/microseg besides the right to change the option.
> Putting a NIC on a vnet needs SDN.Use on the vnet the same way.
> 
> As long as the policy has an assignment, every node keeps the programs
> and maps loaded, about 20 MiB plus what the pairs take. Without one, the
> nodes drop them. The BPF state does not survive a reboot, so a node
> applies the policy again at boot, before any guest starts. The programs
> are compiled into libpve-rs-perl, so an upgrade of that package applies
> the policy again as well.
> 
> The SDN apply checks the policy against what a node can hold before it
> writes the running config, and refuses one that exceeds it. A node that
> cannot apply the policy fails its network reload task and keeps its
> previous enforcement. A node without loaded programs, as after a failed
> load at boot, starts no guest with an assigned interface until an apply
> or a reload succeeds there. A guest also does not start when its node
> cannot read or parse the running config, or cannot set up the
> enforcement of its assigned interface.
> 
> ARP and neighbor solicitations/advertisements pass wherever either
> direction of a pair allows anything, so the two ends of a one-way allow
> can resolve each other's addresses. They also pass when the host itself
> sends them without an identity. That is how the suppression replies of
> an EVPN zone reach the guest. A frame the host turns around on the
> interface it came in on passes as a whole, since only the host can make
> one. Behind a firewall bridge the host's frames cannot be told from
> other unstamped ones, so an interface there takes these frames from
> every unstamped sender. The narrowed rules of a pair run first, so a
> narrowed rule that denies neighbor discovery still holds.
> 
> In patch order, the series adds
>  - proxmox-ebpf, the policy subsystem with the tc programs that stamp
>    and check the identities
>  - proxmox-ve-rs, the types, the identity engine and the desired state
>    in proxmox-ve-config, and the new proxmox-microseg crate that hands
>    the desired state to the subsystem
>  - pve-cluster, microseg.cfg as an observed file
>  - pve-access-control, the /sdn/microseg ACL path
>  - proxmox-perl-rs, the config binding and the data plane binding in
>    pve-rs
>  - pve-network, the config, the API, the dry-run view, the zone option
>    and the enforcement on interface plug and after a library upgrade
>  - pve-manager, the panel, the dry-run view and the apply on network
>    reload and at boot
>  - qemu-server and pve-container, the immediate update of a running
>    guest's NICs on a tag or name change, and SDN.Use for a tag or name
>    the policy matches
>  - pve-docs, documentation
> 
>  - pve-network needs pve-cluster, pve-access-control, pve-rs,
>    pve-guest-common and ifupdown2 with the vxlan-gbp attribute
>  - pve-manager needs pve-docs to build and pve-network and
>    pve-access-control at runtime
>  - pve-rs needs proxmox-microseg and proxmox-ve-config
>  - qemu-server and pve-container need pve-network
>  - proxmox-microseg needs proxmox-ebpf[1] and proxmox-ve-config
> 
> The ifupdown2 vxlan-gbp attribute[2] is sent separately, and so is the
> pve-network patch that writes a VXLAN zone's local tunnel address[3].
> This series needs both. A GBP device that ifupdown2 changes in place
> drops every GBP frame afterward[4], and without[3] that happens when a
> VXLAN zone with GBP exists next to an EVPN zone. ifupdown2 gives the
> local tunnel address of the EVPN zone's interfaces to every VXLAN
> interface without one of its own, and on a reload sets that address on
> the VXLAN zone's live device.
> 
> [1] https://lore.proxmox.com/pve-devel/20261019142413.2255644-1-h.laimer@proxmox.com/
> [2] https://lore.proxmox.com/pve-devel/20260828134708.692354-1-h.laimer@proxmox.com/
> [3] https://lore.proxmox.com/pve-devel/20260925134050.754528-1-h.laimer@proxmox.com/
> [4] https://lore.kernel.org/netdev/20261023-b4-vxlan-changelink-gbp-v1-0-f0d13951c09b@proxmox.com/
> 
> note: I did also have this implemented with nft instead of ebpf as a
> dataplane. Without narrowing, packets per second (small UDP packets) was
> about the same, with multiple checks nft's rate went down, nothing super
> dramatic but about a 1.5-2x compared to eBPF (~140k vs ~250k). eBPF is
> also more flexible, in case we want to expand possible actions in the
> future, but replacing the dataplane is pretty simple which also allowed
> me to do the side-by-side testing.
> 
> 
> changelog
> v3:
>  - the data plane is a library crate in the proxmox-ve-rs workspace that
>    pve-rs binds, on a policy subsystem in proxmox-ebpf, no agent binary
>    and no boot unit. A node loads the programs while the policy has an
>    assignment and takes new programs on a package upgrade
>  - the model changed in five places:
>     * groups have no numeric mark and can form a hierarchy
>     * assignments are an ordered list and an interface takes the groups
>       of the first that covers it instead of the union of all
>     * rules can be narrowed similar to how firewall rules work,
>       additionally non-first fragments can be matched and an explicit
>       bidirectional flag exists
>     * at a priority tie the more specific rule decides
>     * new `every` predicate that matches everything that has at least
>       one group assigned
>  - interface group (re)assignment happens whenever an interface comes
>    up, sdn apply runs or the tags or name of a running guest change.
>    Adding or removing a tag covered by an assignment, or renaming into
>    or out of a name a regex assignment matches, now also needs SDN.Use
>    on the policy, similar to how we check if a user is allowed to assign
>    a vnet
>  - if a node cannot apply the policy, its network reload task fails.
>    while a node has no loaded programs, as after a failed load at boot,
>    guests with an assigned NIC do not start on it. a nic with groups
>    only comes up enforced
>  - ARP and NS/NA are automatically allowed if there exists a (src,dst)
>    -> [checks ->] allow in either direction, this is so address
>    resolution for allowed connections is possible. Also host-generated
>    ARP and NS/NA frames are allowed, this is specifically needed for
>    EVPN suppression to work, and a frame the host turns around on the
>    interface it came in on passes
>  - api: section types and property keys are kebab-case, the endpoints
>    check /sdn/microseg (a pve-access-control patch adds the ACL path),
>    and the overview reports members, evaluation order and specificity
>    instead of a realized state, plus the id space usage
>  - rule and assignment ids are derived from their identifying fields,
>    so those cannot change on an existing object
>  - ui: a panel with the rule grid by priority, the assignments in match
>    order, the dry-run view, the rule editor on the firewall's selectors,
>    and a guest tab with its groups and rules
>  - the SRv6 carrier and its bridge object are out, the ifupdown2
>    vxlan-gbp attribute is a standalone patch, the zone option stays,
>    docs rewritten
> 
> v2:
>  - core model reworked: each interface now holds a *set* of groups
>    compiled into a per-NIC wire identity, replacing the single-group
>    parent/child tree
>  - rules gained explicit `src`/`dst` predicates (any/all/exact) and a
>    `prio`, replacing the implicit tree-distance ordering
>  - `untagged` is now a nameable built-in group (mark 0)
>  - assignments are additive: a NIC's set is the union of all that
>    resolve to it
>  - added tag-matcher and name-regex assignments
>  - rule and dynamic-assignment ids are now generated
>  - identity allocation is fresh-first with quarantined reclamation, so
>    churn cannot exhaust the id space
>  - split the SRv6 carrier support into droppable tail patches
>  - updated docs
>  - updated cover-letter
> 
> 
> proxmox-ebpf:
> 
> Hannes Laimer (3):
>   build: keep the build directory out of the BPF objects
>   policy: add the enforcement subsystem for guest interfaces
>   policy: add a check for non-first fragments
> 
>  Cargo.toml                     |    1 +
>  build.rs                       |    4 +
>  debian/control                 |    6 +-
>  src/bpf-shim/bpf/bpf_helpers.h |    1 +
>  src/bpf-shim/vmlinux.h         |    1 +
>  src/lib.rs                     |    3 +
>  src/policy/bpf/mark.h          |   27 +
>  src/policy/bpf/policy.bpf.c    |  385 ++++++++++++
>  src/policy/bpf/types.h         |   75 +++
>  src/policy/mod.rs              | 1005 ++++++++++++++++++++++++++++++++
>  src/policy/types.rs            |  159 +++++
>  src/subsystem.rs               |   35 +-
>  tests/common/mod.rs            |   40 ++
>  tests/policy.rs                |  837 ++++++++++++++++++++++++++
>  14 files changed, 2573 insertions(+), 6 deletions(-)
>  create mode 100644 src/policy/bpf/mark.h
>  create mode 100644 src/policy/bpf/policy.bpf.c
>  create mode 100644 src/policy/bpf/types.h
>  create mode 100644 src/policy/mod.rs
>  create mode 100644 src/policy/types.rs
>  create mode 100644 tests/policy.rs
> 
> 
> proxmox-ve-rs:
> 
> Hannes Laimer (11):
>   ve-config: firewall: add a constructor for sctp matches
>   ve-config: sdn: microseg: add identity allocation for group sets
>   ve-config: sdn: microseg: add config types
>   ve-config: sdn: microseg: add tag matcher
>   ve-config: sdn: microseg: add name regex matcher
>   ve-config: sdn: microseg: narrow rules by protocol and ports
>   ve-config: sdn: microseg: add group hierarchy
>   ve-config: firewall: expose protocol, icmp type and code numbers
>   ve-config: sdn: microseg: add the desired state
>   microseg: add a crate that applies the policy to a node's guest NICs
>   microseg: add a rule option that matches non-first fragments
> 
>  Cargo.toml                                    |    2 +
>  proxmox-microseg/Cargo.toml                   |   20 +
>  proxmox-microseg/debian/changelog             |    5 +
>  proxmox-microseg/debian/control               |   46 +
>  proxmox-microseg/debian/copyright             |   18 +
>  proxmox-microseg/debian/debcargo.toml         |    7 +
>  proxmox-microseg/debian/source/format         |    1 +
>  proxmox-microseg/src/ebpf.rs                  |  247 +++
>  proxmox-microseg/src/lib.rs                   |   66 +
>  proxmox-microseg/src/nic.rs                   |  120 ++
>  proxmox-microseg/src/running_config.rs        |   26 +
>  proxmox-ve-config/src/firewall/fw_macros.rs   |    7 +
>  proxmox-ve-config/src/firewall/ports.rs       |   65 +
>  .../src/firewall/types/rule_match.rs          |   63 +
>  proxmox-ve-config/src/sdn/microseg/api.rs     |  710 +++++++++
>  .../src/sdn/microseg/identity.rs              |  475 ++++++
>  proxmox-ve-config/src/sdn/microseg/mod.rs     | 1394 +++++++++++++++++
>  proxmox-ve-config/src/sdn/microseg/state.rs   |  616 ++++++++
>  proxmox-ve-config/src/sdn/microseg/types.rs   |  549 +++++++
>  proxmox-ve-config/src/sdn/mod.rs              |    1 +
>  20 files changed, 4438 insertions(+)
>  create mode 100644 proxmox-microseg/Cargo.toml
>  create mode 100644 proxmox-microseg/debian/changelog
>  create mode 100644 proxmox-microseg/debian/control
>  create mode 100644 proxmox-microseg/debian/copyright
>  create mode 100644 proxmox-microseg/debian/debcargo.toml
>  create mode 100644 proxmox-microseg/debian/source/format
>  create mode 100644 proxmox-microseg/src/ebpf.rs
>  create mode 100644 proxmox-microseg/src/lib.rs
>  create mode 100644 proxmox-microseg/src/nic.rs
>  create mode 100644 proxmox-microseg/src/running_config.rs
>  create mode 100644 proxmox-ve-config/src/sdn/microseg/api.rs
>  create mode 100644 proxmox-ve-config/src/sdn/microseg/identity.rs
>  create mode 100644 proxmox-ve-config/src/sdn/microseg/mod.rs
>  create mode 100644 proxmox-ve-config/src/sdn/microseg/state.rs
>  create mode 100644 proxmox-ve-config/src/sdn/microseg/types.rs
> 
> 
> pve-cluster:
> 
> Hannes Laimer (1):
>   cfs: add 'sdn/microseg.cfg' to observed files
> 
>  src/PVE/Cluster.pm  | 1 +
>  src/pmxcfs/status.c | 1 +
>  2 files changed, 2 insertions(+)
> 
> 
> pve-access-control:
> 
> Hannes Laimer (1):
>   permissions: add ACL path for SDN microsegmentation
> 
>  src/PVE/AccessControl.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> 
> proxmox-perl-rs:
> 
> Hannes Laimer (4):
>   pve-rs: sdn: add microseg config binding
>   pve-rs: sdn: microseg: bind the data plane
>   pve-rs: sdn: microseg: check the data plane limits on render
>   pve-rs: debian: fire the pve-microseg-update trigger on upgrade
> 
>  pve-rs/Cargo.toml                   |   2 +
>  pve-rs/Makefile                     |   3 +
>  pve-rs/debian/control               |   1 +
>  pve-rs/debian/triggers              |   1 +
>  pve-rs/src/bindings/sdn/microseg.rs | 350 ++++++++++++++++++++++++++++
>  pve-rs/src/bindings/sdn/mod.rs      |   1 +
>  6 files changed, 358 insertions(+)
>  create mode 100644 pve-rs/src/bindings/sdn/microseg.rs
> 
> 
> pve-network:
> 
> Hannes Laimer (10):
>   sdn: zones: add vxlan-gbp option to vxlan and evpn zones
>   sdn: microseg: add config, API and guest inventory
>   sdn: dry-run: show pending microseg changes
>   sdn: microseg: enforce the policy on guest NICs
>   sdn: microseg: add tag matcher
>   sdn: microseg: add name regex matcher
>   sdn: microseg: add an update for a running NIC
>   sdn: microseg: expose protocol, port and macro rule options
>   sdn: microseg: allow nesting groups under a parent
>   sdn: microseg: expose the fragments rule option
> 
>  debian/control                                |   1 +
>  debian/libpve-network-perl.postinst           |  21 +
>  debian/triggers                               |   1 +
>  src/PVE/API2/Network/SDN.pm                   |  47 +-
>  src/PVE/API2/Network/SDN/Makefile             |   2 +
>  src/PVE/API2/Network/SDN/Microseg.pm          | 236 +++++++++
>  .../API2/Network/SDN/Microseg/Assignment.pm   |  23 +
>  src/PVE/API2/Network/SDN/Microseg/Base.pm     | 382 ++++++++++++++
>  src/PVE/API2/Network/SDN/Microseg/Group.pm    |  17 +
>  src/PVE/API2/Network/SDN/Microseg/Makefile    |  11 +
>  src/PVE/API2/Network/SDN/Microseg/Rule.pm     |  22 +
>  src/PVE/API2/Network/SDN/Zones.pm             |   8 +
>  src/PVE/Network/SDN.pm                        |  25 +
>  src/PVE/Network/SDN/Makefile                  |   1 +
>  src/PVE/Network/SDN/Microseg.pm               | 494 ++++++++++++++++++
>  src/PVE/Network/SDN/Zones.pm                  |  32 ++
>  src/PVE/Network/SDN/Zones/EvpnPlugin.pm       |   8 +
>  src/PVE/Network/SDN/Zones/VxlanPlugin.pm      |  12 +
>  .../zones/evpn/gbp/expected_controller_config |  77 +++
>  .../zones/evpn/gbp/expected_sdn_interfaces    |  45 ++
>  src/test/zones/evpn/gbp/interfaces            |   7 +
>  src/test/zones/evpn/gbp/sdn_config            |  26 +
>  .../zones/vxlan/gbp/expected_sdn_interfaces   |  17 +
>  src/test/zones/vxlan/gbp/interfaces           |   7 +
>  src/test/zones/vxlan/gbp/sdn_config           |  11 +
>  25 files changed, 1531 insertions(+), 2 deletions(-)
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg.pm
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Assignment.pm
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Base.pm
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Group.pm
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Makefile
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Rule.pm
>  create mode 100644 src/PVE/Network/SDN/Microseg.pm
>  create mode 100644 src/test/zones/evpn/gbp/expected_controller_config
>  create mode 100644 src/test/zones/evpn/gbp/expected_sdn_interfaces
>  create mode 100644 src/test/zones/evpn/gbp/interfaces
>  create mode 100644 src/test/zones/evpn/gbp/sdn_config
>  create mode 100644 src/test/zones/vxlan/gbp/expected_sdn_interfaces
>  create mode 100644 src/test/zones/vxlan/gbp/interfaces
>  create mode 100644 src/test/zones/vxlan/gbp/sdn_config
> 
> 
> pve-manager:
> 
> Hannes Laimer (11):
>   network: apply the microseg policy on reload
>   pve-sdn-commit: apply the microseg policy at boot
>   ui: sdn: zones: add vxlan-gbp checkbox to vxlan and evpn
>   ui: sdn: add microsegmentation panel
>   ui: sdn: dry-run: show pending microseg diff
>   ui: sdn: microseg: add tag matcher
>   ui: sdn: microseg: add name regex matcher
>   ui: sdn: microseg: add narrowing to the rule form
>   ui: sdn: microseg: show and edit the group hierarchy
>   ui: sdn: microseg: add fragments option to the rule form
>   ui: sdn: microseg: add a guest tab with its groups and rules
> 
>  PVE/API2/Network.pm                           |  31 +-
>  bin/pve-sdn-commit                            |  47 ++-
>  services/pve-sdn-commit.service               |   1 +
>  www/css/ext6-pve.css                          |  23 ++
>  www/manager6/Makefile                         |  10 +
>  www/manager6/Utils.js                         |  98 +++++
>  www/manager6/data/PermPathStore.js            |   1 +
>  www/manager6/dc/Config.js                     |   8 +
>  www/manager6/form/SDNMicrosegGroupSelector.js | 105 ++++++
>  www/manager6/grid/FirewallRules.js            |   2 +
>  www/manager6/lxc/Config.js                    |  10 +
>  www/manager6/qemu/Config.js                   |  10 +
>  www/manager6/sdn/MicrosegView.js              |  98 +++++
>  www/manager6/sdn/SdnDiffView.js               |  29 ++
>  www/manager6/sdn/microseg/AssignmentEdit.js   | 148 ++++++++
>  www/manager6/sdn/microseg/AssignmentView.js   | 195 ++++++++++
>  www/manager6/sdn/microseg/Base.js             | 262 ++++++++++++++
>  www/manager6/sdn/microseg/GroupEdit.js        |  40 +++
>  www/manager6/sdn/microseg/GuestPanel.js       | 117 ++++++
>  www/manager6/sdn/microseg/RuleEdit.js         | 285 +++++++++++++++
>  www/manager6/sdn/microseg/RuleView.js         | 338 ++++++++++++++++++
>  www/manager6/sdn/microseg/Tree.js             | 278 ++++++++++++++
>  www/manager6/sdn/zones/EvpnEdit.js            |   8 +
>  www/manager6/sdn/zones/VxlanEdit.js           |  11 +
>  24 files changed, 2126 insertions(+), 29 deletions(-)
>  create mode 100644 www/manager6/form/SDNMicrosegGroupSelector.js
>  create mode 100644 www/manager6/sdn/MicrosegView.js
>  create mode 100644 www/manager6/sdn/microseg/AssignmentEdit.js
>  create mode 100644 www/manager6/sdn/microseg/AssignmentView.js
>  create mode 100644 www/manager6/sdn/microseg/Base.js
>  create mode 100644 www/manager6/sdn/microseg/GroupEdit.js
>  create mode 100644 www/manager6/sdn/microseg/GuestPanel.js
>  create mode 100644 www/manager6/sdn/microseg/RuleEdit.js
>  create mode 100644 www/manager6/sdn/microseg/RuleView.js
>  create mode 100644 www/manager6/sdn/microseg/Tree.js
> 
> 
> qemu-server:
> 
> Hannes Laimer (1):
>   network: update the microseg enforcement on a tag or name change
> 
>  src/PVE/API2/Qemu.pm          | 26 ++++++++++++++++++++++++++
>  src/PVE/QemuServer.pm         | 12 +++++++++++-
>  src/PVE/QemuServer/Network.pm | 14 ++++++++++++++
>  3 files changed, 51 insertions(+), 1 deletion(-)
> 
> 
> pve-container:
> 
> Hannes Laimer (1):
>   lxc: update the microseg enforcement on a tag or hostname change
> 
>  src/PVE/API2/LXC/Config.pm |  3 +++
>  src/PVE/LXC.pm             | 28 ++++++++++++++++++++++++++++
>  2 files changed, 31 insertions(+)
> 
> 
> pve-docs:
> 
> Hannes Laimer (2):
>   sdn: zones: document the VXLAN-GBP option
>   sdn: add microsegmentation section
> 
>  pvesdn.adoc | 403 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 403 insertions(+)
> 
> 
> Summary over all repositories:
>   98 files changed, 11514 insertions(+), 38 deletions(-)
>