superseded: [RFC cluster/docs/ifupdown2/manager/network/proxmox{-ve-rs,-ebpf,-perl-rs} v2 00/27] sdn: add microsegmentation support

[Hannes Laimer <h.laimer@proxmox.com>]

superseded-by:
https://lore.proxmox.com/pve-devel/20261027085527.702738-1-h.laimer@proxmox.com/

On 2026-07-09 11:19, Hannes Laimer wrote:
> This adds support for microsegmentation using eBPF programs attached to
> interfaces. Mostly the tap/veth interfaces on the guests directly.
> 
> # Overview
> Each guest interface is assigned a *set* of groups. Rather than a hierarchy,
> group membership is treated as a signature. The agent compiles each interface's
> group set into a single per-NIC *identity*, and policies are rules between
> groups, matched against those sets with predicates.
> 
> A rule is a `src` predicate and a `dst` predicate plus a verdict and a
> priority, where a predicate is a match kind over a list of groups:
>  - any{...}    the interface shares at least one of the listed groups
>  - all{...}    the interface has all of the listed groups
>  - exact{...}  the interface's groups are exactly the listed set
> 
> For a packet, the `src` predicate is evaluated against the sender's set and the
> `dst` predicate against the receiver's. Among the rules that fire the highest
> `prio` wins, a conflicting tie (with different verdicts) denies, and if no rule
> fires the packet is denied (default-deny). For example, with the groups
> web/app/db/prod, specific allows punched through a broad deny:
> 
>  - any{web} -> any{app}               : allow prio 10
>  - any{app} -> any{db}                : allow prio 10
>  - all{prod} -> all{prod}             : allow prio 10   # these three carve out east-west...
>  - any{web,app,db} -> any{web,app,db} : deny prio 5     # ...through a broad tier-to-tier deny
> (the last is technically not needed cause the default is deny, in a larger
> context having this explicitly can be necesarry though)
> 
> The allows above are one-directional. Rules are stateless and per-packet, there
> is no connection tracking. The reply needs its own allow, so for TCP between web
> and app both any{web} -> any{app} and any{app} -> any{web} have to be allowed.
> 
> Groups land on an interface either directly (a guest matcher: a `vmid`,
> optionally a single `iface`), by tag (a tag matcher: every guest carrying the
> named tags, e.g. `prod`/`staging`/`web`), or by name (a regex matcher: every
> guest whose name matches a regular expression). Assignments are additive, an
> interface's set is the union of every assignment that resolves to it, so the
> matchers compose.
> 
> `untagged` is a reserved built-in group (mark 0) for traffic that arrives
> unstamped (gateways, DHCP/ARP, cross-node frames without a tag). It can be
> named in a predicate and composed with real groups (e.g. `any{untagged, web}`),
> but it is not assignable.
> 
> Enforcement happens on the receiving side, cause that's where we for sure know both
>  - where the packet is from
>  - and, its destination (us)
> 
> # config and API
> The config is a single section-config, `sdn/microseg.cfg`. It holds these
> object types, keyed by section id:
>  - group, a numeric `mark` (auto-assigned if omitted) and an optional comment
>  - rule, a `src`/`dst` predicate pair (`*_match` is any/all/exact over a group list),
>    a `prio` and `allow`
>  - guestassignment, binds a guest's NIC(s) (`vmid`, optional `iface`) to a group set
>  - tagassignment, binds the NICs of every guest matching `tags` (by `tag_match`) to a
>    group set, expanded against the live guest inventory on render
>  - regexassignment, same but for every guest whose name matches a regular
>    expression
>  - bridge, marks a bridge-facing interface as an SRv6 carrier (no UI yet)
> The API exposes endpoints under
> `/cluster/sdn/microseg/{group,rule,assignment,bridge}`, guarded by SDN.Audit for reads
> and SDN.Allocate for writes. The assignment endpoint discriminates the two matcher
> kinds on a `type` property (like the fabric `protocol`).
> Rule and tag/regex-assignment ids are derived from their contents (an FNV-1a hash),
> guest assignments from `vm{vmid}i{iface}`, groups and bridges take a chosen id. On
> commit the config is rendered into `.running-config`, which is what the agent reads.
> Rendering resolves each interface's group set to a wire identity and expands the
> dynamic matchers against the inventory.
> 
> # identity
> An identity is not one id per group set but one id per *signature*. Two
> interfaces share an id when every rule predicate fires the same way on both, so
> no rule can tell them apart. The identities are the classes of that partition,
> each named by a 16-bit id, and that id is what rides on the wire.
> 
> Editing rules reshapes the partition. A new predicate that separates two sets
> which used to look alike *splits* their class in two, and dropping the
> predicate that told two classes apart *merges* them back into one. On a split
> the subclass that still carries the old representative keeps the id and the
> other gets a fresh one, on a merge the surviving class keeps its id and the
> fused-away ones retire.
> 
> New ids come fresh-first from a counter. A retired id is not reused right away,
> it sits in a time quarantine long enough that no packet stamped under its old
> meaning can still be in flight. Only once the fresh space is exhausted does
> allocation fall back to reusing a retired id whose quarantine has passed, so
> config churn alone cannot exhaust the 16-bit id space. Actually running out
> would take more than 65534 identities the policies can tell apart. So it
> doesn't scale with count of group combinations in use, but with the distinct
> semantic meaning they carry.
> 
> # skb->mark
> Every packet in the kernel lives in a `sk_buff` struct, which has a 32-bit
> `mark` field you can read and write while the packet moves through the kernel.
> Nothing else in our SDN stack touches `mark` today, so microseg has it to
> itself. Worth keeping in mind if that ever changes. For now we only use the
> lower 16 bits, to carry the interface's identity (mark 0 is reserved for
> unstamped traffic), so up to 65535 identities.
> 
> The catch is `mark` only lives as long as the packet stays in the kernel, and
> sooner or later it leaves. To carry the identity between hosts we have to put it
> on the wire itself. For that we support two carriers:
>  - VXLAN-GBP, the 16-bit GBP field on the VXLAN encapsulation
>  - SRv6, the 16-bit Tag field on the SRH
> The kernel already moves `mark` in and out of the VXLAN-GBP field for us (found
> that out way too late :P). For SRv6 we do it ourselves with a small eBPF program.
> 
> # eBPF
> Both tagging (setting `mark`) and enforcement happen in an eBPF program
> directly attached to the guest's interface. The programs themselves read and
> write to/from two maps:
>  - tap_to_group, the identity to stamp for a given interface
>  - rules, a map of (src_id, dst_id) -> action, where src is the `mark` the
>    packet carries and dst is our own interface's identity
> 
> Specifically, on ingress we set the mark, and on egress we enforce.
> 
> # Implementation
> We have an `agent` that reads the running sdn config, and applies that state to
> the kernel. The binary is stateless and one-shot, not a long running daemon. It
> runs on SDN apply, on tap_plug of a guest interface, and on boot after
> pve-sdn-commit but before guests start. Those triggers cover every situation
> where we have to touch kernel state.
> 
> It compiles the config into what the data plane needs. It folds each
> interface's group set into a wire identity (a registry kept stable across
> renders so identities agree across hosts, allocated as described above), and
> folds the rules into the `(src_id, dst_id)` action map by priority (highest
> wins, conflicting ties deny).
> 
> It keeps track of what is currently loaded in the kernel with two files under
> `/run/proxmox-ebpf`. One keeping track of the bpf program that was compiled
> into the binary, and one for keeping track of changes to the data structures
> the bpf program accesses. The distinction is useful cause for only a program
> change we can swap the currently attached program with the updated one
> atomically. In case the data structure (the maps) changed there is no other way
> than to tear-down all the current state and repopulate the maps and re-attach
> the programs. Swapping out the maps first would have old programs interact with
> a new data schema, and swapping the programs first will lead to new programs
> accessing an old data schema. Either way, not good, so we wipe the state in
> that case and re-build it. So, in case we change the structure of the data our
> bpf programs access, there will be a bunch of ms where the configured
> segmentation is not enforced. But that is the only scenario where that can
> happen.
> 
> ## aya, aya-ebpf
> Aya is a rust lib that helps with working with BPF more easily. It has both a
> userspace part (`aya`) and a part (`aya-ebpf`) that compiles to bpf. I chose to
> only use the userspace part, and use C for the BPF programs themselves. The
> reasons were:
>  - we don't have `aya-ebpf` packaged
>  - `aya-ebpf` *requires* nightly toolchain to compile
>  - the bpf programs are very small, in case this should change at some point we
>    can reconsider. Also, it doesn't matter what produces the .o file, so this
>    could be swapped in really easily if we decided to
> 
> So we compile the bpf program written in C to a .o targeting bpf using clang
> and include it when compiling the agent binary.
> 
> ## SRv6
> Besides VXLAN-GBP, the identity can also ride in the 16-bit Tag field of an SRv6
> SRH. The kernel already does this move for VXLAN-GBP but not for SRv6, so this
> adds a small eBPF program that writes the identity into the SRH on egress and
> reads it back on ingress. The `bridge` object marks the bridge-facing interface
> it runs on.
> 
> This is only the tag carrier, not an SRv6 transport. Our SDN stack has no SRv6
> zone yet, so there is nothing configured to route it over, and there is no UI.
> If SDN grows an SRv6 transport later (a small L3, EVPN-lite style option), the
> carrier is already in place.
> 
> It is packaged as a droppable tail. The carrier bridge section (proxmox-ve-config),
> the bridge API (pve-network) and the bridge subsystem (proxmox-ebpf) are each the
> last patch of their repo's series, so SRv6 can be deferred without touching the rest.
> 
> # enabling VXLAN-GBP
> The kernel only moves the mark into the GBP field if the vxlan device was
> created with the GBP flag set, and it's create-only, the kernel won't
> toggle it on a running device. ifupdown2 had no attribute for this, so
> this series includes a small ifupdown2 patch adding `vxlan-gbp`, which
> threads the flag through to the netlink/iproute2 create path.
> 
> # testing
> I have put pre-built packages on sani, these include ifupdown2 with the patch
> 
> # building
> 1. proxmox-ve-rs (install)
> 2. proxmox-ebpf
> 3. pve-cluster (install)
> 4. proxmox-perl-rs (install)
> 5. pve-network
> 6. pve-manager
> 
> # notes
> * everything eBPF related did not change since v1, this v2 only really changes identity semantics
> * using GENEVE would also be an option, but with this current approach we gain
>   a lot of flexibility. Expanding predicate logic, or having not just
>   allow/deny but also match ports or alike is possible without touching the
>   underlying identity system. (limiting the per-packet overhead to just 16
>   bits is also good I'd argue)
> * the enforement mechanism is not set in stone, weather this is eBPF or maybe
>   nftables with a map we populate as shortly discussed with @Stefan. The
>   current approach plays nicely with both since it conveys identity through
>   `skb->mark` that nft can access as well 
> * I did not send the first commit for `proxmox-ebpf` since it contains a
>   `vmlinux.h` which is rather large but is needed for compiling. The commit is in
>   my staff repo.
> 
> # changelog
> v2:
>  - core model reworked: each interface now holds a *set* of groups compiled
>    into a per-NIC wire identity, replacing the single-group parent/child tree
>  - rules gained explicit `src`/`dst` predicates (any/all/exact) and a `prio`,
>    replacing the implicit tree-distance ordering
>  - `untagged` is now a nameable built-in group (mark 0)
>  - assignments are additive: a NIC's set is the union of all that resolve to it
>  - added tag-matcher and name-regex assignments
>  - rule and dynamic-assignment ids are now generated
>  - identity allocation is fresh-first with quarantined reclamation, so churn
>    cannot exhaust the id space
>  - split the SRv6 carrier support into droppable tail patches
>  - updated docs
>  - updated cover-letter
> 
> 
> proxmox-ve-rs:
> 
> Hannes Laimer (5):
>   ve-config: sdn: add microseg signature-identity engine
>   ve-config: sdn: add microseg config types
>   ve-config: sdn: microseg: add tag matcher
>   ve-config: sdn: microseg: add name regex matcher
>   ve-config: sdn: microseg: add carrier bridge section
> 
>  proxmox-ve-config/src/sdn/config.rs           |    9 +-
>  .../src/sdn/microseg/identity.rs              |  624 +++++
>  proxmox-ve-config/src/sdn/microseg/mod.rs     | 2349 +++++++++++++++++
>  proxmox-ve-config/src/sdn/mod.rs              |    1 +
>  4 files changed, 2982 insertions(+), 1 deletion(-)
>  create mode 100644 proxmox-ve-config/src/sdn/microseg/identity.rs
>  create mode 100644 proxmox-ve-config/src/sdn/microseg/mod.rs
> 
> 
> proxmox-ebpf:
> 
> Hannes Laimer (3):
>   agent: add userspace coordinator and stateless policy subsystem
>   bpf: add bridge subsystem
>   debian: add packaging and boot-time oneshot unit
> 
>  Makefile                    |  66 +++++++
>  debian/changelog            |   5 +
>  debian/control              |  35 ++++
>  debian/copyright            |  18 ++
>  debian/proxmox-ebpf.install |   1 +
>  debian/proxmox-ebpf.postrm  |  11 ++
>  debian/proxmox-ebpf.prerm   |  12 ++
>  debian/proxmox-ebpf.service |  15 ++
>  debian/rules                |  33 ++++
>  debian/source/format        |   1 +
>  include/mark.h              |  30 +++
>  src/agent.rs                |  99 ++++++++++
>  src/bridge/bpf/srv6.bpf.c   |  76 +++++++
>  src/bridge/mod.rs           |  76 +++++++
>  src/main.rs                 |  69 +++++++
>  src/policy/bpf/tap.bpf.c    |  68 +++++++
>  src/policy/bpf/types.h      |  23 +++
>  src/policy/mod.rs           | 268 +++++++++++++++++++++++++
>  src/policy/types.rs         |  45 +++++
>  src/running_config.rs       |  38 ++++
>  src/state.rs                | 149 ++++++++++++++
>  src/subsystem.rs            | 385 ++++++++++++++++++++++++++++++++++++
>  src/tc.rs                   | 152 ++++++++++++++
>  23 files changed, 1675 insertions(+)
>  create mode 100644 Makefile
>  create mode 100644 debian/changelog
>  create mode 100644 debian/control
>  create mode 100644 debian/copyright
>  create mode 100644 debian/proxmox-ebpf.install
>  create mode 100755 debian/proxmox-ebpf.postrm
>  create mode 100755 debian/proxmox-ebpf.prerm
>  create mode 100644 debian/proxmox-ebpf.service
>  create mode 100755 debian/rules
>  create mode 100644 debian/source/format
>  create mode 100644 include/mark.h
>  create mode 100644 src/agent.rs
>  create mode 100644 src/bridge/bpf/srv6.bpf.c
>  create mode 100644 src/bridge/mod.rs
>  create mode 100644 src/main.rs
>  create mode 100644 src/policy/bpf/tap.bpf.c
>  create mode 100644 src/policy/bpf/types.h
>  create mode 100644 src/policy/mod.rs
>  create mode 100644 src/policy/types.rs
>  create mode 100644 src/running_config.rs
>  create mode 100644 src/state.rs
>  create mode 100644 src/subsystem.rs
>  create mode 100644 src/tc.rs
> 
> 
> pve-cluster:
> 
> Hannes Laimer (1):
>   cfs: add 'sdn/microseg.cfg' to observed files
> 
>  src/PVE/Cluster.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> 
> proxmox-perl-rs:
> 
> Hannes Laimer (1):
>   pve-rs: sdn: add microseg config binding
> 
>  pve-rs/Makefile                     |   1 +
>  pve-rs/src/bindings/sdn/microseg.rs | 235 ++++++++++++++++++++++++++++
>  pve-rs/src/bindings/sdn/mod.rs      |   1 +
>  3 files changed, 237 insertions(+)
>  create mode 100644 pve-rs/src/bindings/sdn/microseg.rs
> 
> 
> ifupdown2:
> 
> Hannes Laimer (1):
>   d/patches: add support for VXLAN-GBP flag
> 
>  ...addons-vxlan-add-vxlan-gbp-attribute.patch | 228 ++++++++++++++++++
>  debian/patches/series                         |   1 +
>  2 files changed, 229 insertions(+)
>  create mode 100644 debian/patches/pve/0016-addons-vxlan-add-vxlan-gbp-attribute.patch
> 
> 
> pve-network:
> 
> Hannes Laimer (8):
>   sdn: microseg: add config, API and guest inventory
>   sdn: dry-run: surface pending microseg changes
>   sdn: zones: trigger microseg apply on tap_plug
>   sdn: zones: add vxlan-gbp option to vxlan and evpn zones
>   evpn: disable vxlan-learning on create if GBP is enabled
>   sdn: microseg: add tag matcher
>   sdn: microseg: add name regex matcher
>   sdn: microseg: add carrier bridge API
> 
>  src/PVE/API2/Network/SDN.pm                   |  31 ++
>  src/PVE/API2/Network/SDN/Makefile             |   2 +
>  src/PVE/API2/Network/SDN/Microseg.pm          | 204 +++++++
>  .../API2/Network/SDN/Microseg/Assignment.pm   | 185 +++++++
>  src/PVE/API2/Network/SDN/Microseg/Bridge.pm   | 179 ++++++
>  src/PVE/API2/Network/SDN/Microseg/Group.pm    | 179 ++++++
>  src/PVE/API2/Network/SDN/Microseg/Makefile    |   8 +
>  src/PVE/API2/Network/SDN/Microseg/Rule.pm     | 180 +++++++
>  src/PVE/Network/SDN.pm                        |  37 ++
>  src/PVE/Network/SDN/Makefile                  |   1 +
>  src/PVE/Network/SDN/Microseg.pm               | 509 ++++++++++++++++++
>  src/PVE/Network/SDN/Zones.pm                  |   6 +
>  src/PVE/Network/SDN/Zones/EvpnPlugin.pm       |  11 +
>  src/PVE/Network/SDN/Zones/VxlanPlugin.pm      |   9 +
>  14 files changed, 1541 insertions(+)
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg.pm
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Assignment.pm
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Bridge.pm
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Group.pm
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Makefile
>  create mode 100644 src/PVE/API2/Network/SDN/Microseg/Rule.pm
>  create mode 100644 src/PVE/Network/SDN/Microseg.pm
> 
> 
> pve-manager:
> 
> Hannes Laimer (6):
>   ui: sdn: add microsegmentation panel
>   ui: sdn: dry-run: show pending microseg diff
>   network: apply microseg state on reload
>   ui: sdn: zones: add vxlan-gbp checkbox to vxlan and evpn
>   ui: sdn: microseg: add tag matcher
>   ui: sdn: microseg: add name regex matcher
> 
>  PVE/API2/Network.pm                         |   4 +
>  www/manager6/Makefile                       |   9 +
>  www/manager6/Utils.js                       |  34 ++
>  www/manager6/dc/Config.js                   |   8 +
>  www/manager6/form/MicrosegGroupSelector.js  |  77 +++++
>  www/manager6/form/MicrosegGuestSelector.js  |  83 +++++
>  www/manager6/sdn/MicrosegView.js            | 143 ++++++++
>  www/manager6/sdn/SdnDiffView.js             |  24 ++
>  www/manager6/sdn/microseg/AssignmentEdit.js | 245 ++++++++++++++
>  www/manager6/sdn/microseg/Base.js           | 137 ++++++++
>  www/manager6/sdn/microseg/GroupEdit.js      |  45 +++
>  www/manager6/sdn/microseg/PolicyView.js     | 181 ++++++++++
>  www/manager6/sdn/microseg/RuleEdit.js       | 104 ++++++
>  www/manager6/sdn/microseg/Tree.js           | 356 ++++++++++++++++++++
>  www/manager6/sdn/zones/EvpnEdit.js          |   8 +
>  www/manager6/sdn/zones/VxlanEdit.js         |  11 +
>  16 files changed, 1469 insertions(+)
>  create mode 100644 www/manager6/form/MicrosegGroupSelector.js
>  create mode 100644 www/manager6/form/MicrosegGuestSelector.js
>  create mode 100644 www/manager6/sdn/MicrosegView.js
>  create mode 100644 www/manager6/sdn/microseg/AssignmentEdit.js
>  create mode 100644 www/manager6/sdn/microseg/Base.js
>  create mode 100644 www/manager6/sdn/microseg/GroupEdit.js
>  create mode 100644 www/manager6/sdn/microseg/PolicyView.js
>  create mode 100644 www/manager6/sdn/microseg/RuleEdit.js
>  create mode 100644 www/manager6/sdn/microseg/Tree.js
> 
> 
> pve-docs:
> 
> Hannes Laimer (2):
>   sdn: add microsegmentation section
>   sdn: add VXLAN-GBP flag to evpn/vxlan zone sections
> 
>  pvesdn.adoc | 134 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 134 insertions(+)
> 
> 
> Summary over all repositories:
>   64 files changed, 8268 insertions(+), 1 deletions(-)
>