[PATCH access-control/cluster/container/docs/manager/network/proxmox{-ebpf,-ve-rs,-perl-rs}/qemu-server v3 00/45] sdn: add microsegmentation support

[Hannes Laimer <h.laimer@proxmox.com>]

This series adds group-based microsegmentation to SDN, enforced by eBPF
programs on the guest interfaces. Assignments put guest interfaces into
groups, and rules between groups decide which traffic passes. Sets that
every rule treats the same share one identity, a 16-bit value that
travels with the frame in the low bits of the packet mark. The host
stamps it where a frame comes off the guest interface, so a guest cannot
forge its own. A new zone option turns on VXLAN-GBP[2] for VXLAN and
EVPN zones, which carries the identity across nodes. On a zone without
it, traffic from another node arrives without an identity. An identity
whose sets are gone after a config change is retired. Its value is
reused only after a quarantine, since a frame stamped under the old
config can still be in flight.

The config in /etc/pve/sdn/microseg.cfg, with its API under
/cluster/sdn/microseg and the ACL path /sdn/microseg, holds groups,
rules and assignments and takes effect with an SDN apply. Groups can
nest, and a member of a subgroup is in every group above it. A rule
allows or denies traffic from a source predicate to a destination
predicate at a priority. A predicate of the kind `any`, `all` or
`exact` lists groups and matches an interface that has one of them, all
of them, or those and no others. An `every` predicate lists none and
matches each interface that has a group. Beyond the pair, a rule can be
narrowed to a protocol and ports, an ICMP type, a firewall macro or
non-first fragments. Which groups an interface has is determined by the
assignments, an ordered list that matches interfaces by guest, tag or
name regex. An interface takes the groups of the first one that covers
it.

Default is deny. Only interfaces that have at least one group assigned
are enforced. Enforcement happens when a frame arrives at an enforced
interface, so nothing is dropped on the way out of a guest's interface.
Traffic from interfaces that are not enforced is untagged, which
includes guests without groups on their NICs, the host and the outside.
Such traffic has to be allowed explicitly with the built-in untagged
group. Rules are stateless and directional, so a reply needs its own
allow, or the rule is marked bidirectional. Stateless also means no
reassembly, so fragments after the first carry no transport header. A
narrowed rule matches them only when it is narrowed to fragments.

The highest priority wins. At the same priority the rule with the more
specific `all` or `exact` predicate decides, so a rule on a subgroup
outranks the same rule on its parent. Equally specific rules deny if one
of them denies. Narrowed rules are checked before the rules that decide
the whole pair at the same priority.

A guest interface gets its groups the moment it comes up, from the
assignment that matches the guest at that moment. Changing the tags or
the name of a running guest takes effect right after the config is
written. The NICs get the groups the assignments now give them. Once no
assignment matches the guest, their enforcement stops. A change that
reaches the config another way, such as a restore or a migration, takes
effect the next time the NIC comes up or at the next SDN apply. A tag
or name the assignments match moves a guest between groups. So adding
or removing such a tag, or renaming into or out of such a name, needs
SDN.Use on /sdn/microseg besides the right to change the option.
Putting a NIC on a vnet needs SDN.Use on the vnet the same way.

As long as the policy has an assignment, every node keeps the programs
and maps loaded, about 20 MiB plus what the pairs take. Without one, the
nodes drop them. The BPF state does not survive a reboot, so a node
applies the policy again at boot, before any guest starts. The programs
are compiled into libpve-rs-perl, so an upgrade of that package applies
the policy again as well.

The SDN apply checks the policy against what a node can hold before it
writes the running config, and refuses one that exceeds it. A node that
cannot apply the policy fails its network reload task and keeps its
previous enforcement. A node without loaded programs, as after a failed
load at boot, starts no guest with an assigned interface until an apply
or a reload succeeds there. A guest also does not start when its node
cannot read or parse the running config, or cannot set up the
enforcement of its assigned interface.

ARP and neighbor solicitations/advertisements pass wherever either
direction of a pair allows anything, so the two ends of a one-way allow
can resolve each other's addresses. They also pass when the host itself
sends them without an identity. That is how the suppression replies of
an EVPN zone reach the guest. A frame the host turns around on the
interface it came in on passes as a whole, since only the host can make
one. Behind a firewall bridge the host's frames cannot be told from
other unstamped ones, so an interface there takes these frames from
every unstamped sender. The narrowed rules of a pair run first, so a
narrowed rule that denies neighbor discovery still holds.

In patch order, the series adds
 - proxmox-ebpf, the policy subsystem with the tc programs that stamp
   and check the identities
 - proxmox-ve-rs, the types, the identity engine and the desired state
   in proxmox-ve-config, and the new proxmox-microseg crate that hands
   the desired state to the subsystem
 - pve-cluster, microseg.cfg as an observed file
 - pve-access-control, the /sdn/microseg ACL path
 - proxmox-perl-rs, the config binding and the data plane binding in
   pve-rs
 - pve-network, the config, the API, the dry-run view, the zone option
   and the enforcement on interface plug and after a library upgrade
 - pve-manager, the panel, the dry-run view and the apply on network
   reload and at boot
 - qemu-server and pve-container, the immediate update of a running
   guest's NICs on a tag or name change, and SDN.Use for a tag or name
   the policy matches
 - pve-docs, documentation

 - pve-network needs pve-cluster, pve-access-control, pve-rs,
   pve-guest-common and ifupdown2 with the vxlan-gbp attribute
 - pve-manager needs pve-docs to build and pve-network and
   pve-access-control at runtime
 - pve-rs needs proxmox-microseg and proxmox-ve-config
 - qemu-server and pve-container need pve-network
 - proxmox-microseg needs proxmox-ebpf[1] and proxmox-ve-config

The ifupdown2 vxlan-gbp attribute[2] is sent separately, and so is the
pve-network patch that writes a VXLAN zone's local tunnel address[3].
This series needs both. A GBP device that ifupdown2 changes in place
drops every GBP frame afterward[4], and without[3] that happens when a
VXLAN zone with GBP exists next to an EVPN zone. ifupdown2 gives the
local tunnel address of the EVPN zone's interfaces to every VXLAN
interface without one of its own, and on a reload sets that address on
the VXLAN zone's live device.

[1] https://lore.proxmox.com/pve-devel/20261019142413.2255644-1-h.laimer@proxmox.com/
[2] https://lore.proxmox.com/pve-devel/20260828134708.692354-1-h.laimer@proxmox.com/
[3] https://lore.proxmox.com/pve-devel/20260925134050.754528-1-h.laimer@proxmox.com/
[4] https://lore.kernel.org/netdev/20261023-b4-vxlan-changelink-gbp-v1-0-f0d13951c09b@proxmox.com/

note: I did also have this implemented with nft instead of ebpf as a
dataplane. Without narrowing, packets per second (small UDP packets) was
about the same, with multiple checks nft's rate went down, nothing super
dramatic but about a 1.5-2x compared to eBPF (~140k vs ~250k). eBPF is
also more flexible, in case we want to expand possible actions in the
future, but replacing the dataplane is pretty simple which also allowed
me to do the side-by-side testing.


changelog
v3:
 - the data plane is a library crate in the proxmox-ve-rs workspace that
   pve-rs binds, on a policy subsystem in proxmox-ebpf, no agent binary
   and no boot unit. A node loads the programs while the policy has an
   assignment and takes new programs on a package upgrade
 - the model changed in five places:
    * groups have no numeric mark and can form a hierarchy
    * assignments are an ordered list and an interface takes the groups
      of the first that covers it instead of the union of all
    * rules can be narrowed similar to how firewall rules work,
      additionally non-first fragments can be matched and an explicit
      bidirectional flag exists
    * at a priority tie the more specific rule decides
    * new `every` predicate that matches everything that has at least
      one group assigned
 - interface group (re)assignment happens whenever an interface comes
   up, sdn apply runs or the tags or name of a running guest change.
   Adding or removing a tag covered by an assignment, or renaming into
   or out of a name a regex assignment matches, now also needs SDN.Use
   on the policy, similar to how we check if a user is allowed to assign
   a vnet
 - if a node cannot apply the policy, its network reload task fails.
   while a node has no loaded programs, as after a failed load at boot,
   guests with an assigned NIC do not start on it. a nic with groups
   only comes up enforced
 - ARP and NS/NA are automatically allowed if there exists a (src,dst)
   -> [checks ->] allow in either direction, this is so address
   resolution for allowed connections is possible. Also host-generated
   ARP and NS/NA frames are allowed, this is specifically needed for
   EVPN suppression to work, and a frame the host turns around on the
   interface it came in on passes
 - api: section types and property keys are kebab-case, the endpoints
   check /sdn/microseg (a pve-access-control patch adds the ACL path),
   and the overview reports members, evaluation order and specificity
   instead of a realized state, plus the id space usage
 - rule and assignment ids are derived from their identifying fields,
   so those cannot change on an existing object
 - ui: a panel with the rule grid by priority, the assignments in match
   order, the dry-run view, the rule editor on the firewall's selectors,
   and a guest tab with its groups and rules
 - the SRv6 carrier and its bridge object are out, the ifupdown2
   vxlan-gbp attribute is a standalone patch, the zone option stays,
   docs rewritten

v2:
 - core model reworked: each interface now holds a *set* of groups
   compiled into a per-NIC wire identity, replacing the single-group
   parent/child tree
 - rules gained explicit `src`/`dst` predicates (any/all/exact) and a
   `prio`, replacing the implicit tree-distance ordering
 - `untagged` is now a nameable built-in group (mark 0)
 - assignments are additive: a NIC's set is the union of all that
   resolve to it
 - added tag-matcher and name-regex assignments
 - rule and dynamic-assignment ids are now generated
 - identity allocation is fresh-first with quarantined reclamation, so
   churn cannot exhaust the id space
 - split the SRv6 carrier support into droppable tail patches
 - updated docs
 - updated cover-letter


proxmox-ebpf:

Hannes Laimer (3):
  build: keep the build directory out of the BPF objects
  policy: add the enforcement subsystem for guest interfaces
  policy: add a check for non-first fragments

 Cargo.toml                     |    1 +
 build.rs                       |    4 +
 debian/control                 |    6 +-
 src/bpf-shim/bpf/bpf_helpers.h |    1 +
 src/bpf-shim/vmlinux.h         |    1 +
 src/lib.rs                     |    3 +
 src/policy/bpf/mark.h          |   27 +
 src/policy/bpf/policy.bpf.c    |  385 ++++++++++++
 src/policy/bpf/types.h         |   75 +++
 src/policy/mod.rs              | 1005 ++++++++++++++++++++++++++++++++
 src/policy/types.rs            |  159 +++++
 src/subsystem.rs               |   35 +-
 tests/common/mod.rs            |   40 ++
 tests/policy.rs                |  837 ++++++++++++++++++++++++++
 14 files changed, 2573 insertions(+), 6 deletions(-)
 create mode 100644 src/policy/bpf/mark.h
 create mode 100644 src/policy/bpf/policy.bpf.c
 create mode 100644 src/policy/bpf/types.h
 create mode 100644 src/policy/mod.rs
 create mode 100644 src/policy/types.rs
 create mode 100644 tests/policy.rs


proxmox-ve-rs:

Hannes Laimer (11):
  ve-config: firewall: add a constructor for sctp matches
  ve-config: sdn: microseg: add identity allocation for group sets
  ve-config: sdn: microseg: add config types
  ve-config: sdn: microseg: add tag matcher
  ve-config: sdn: microseg: add name regex matcher
  ve-config: sdn: microseg: narrow rules by protocol and ports
  ve-config: sdn: microseg: add group hierarchy
  ve-config: firewall: expose protocol, icmp type and code numbers
  ve-config: sdn: microseg: add the desired state
  microseg: add a crate that applies the policy to a node's guest NICs
  microseg: add a rule option that matches non-first fragments

 Cargo.toml                                    |    2 +
 proxmox-microseg/Cargo.toml                   |   20 +
 proxmox-microseg/debian/changelog             |    5 +
 proxmox-microseg/debian/control               |   46 +
 proxmox-microseg/debian/copyright             |   18 +
 proxmox-microseg/debian/debcargo.toml         |    7 +
 proxmox-microseg/debian/source/format         |    1 +
 proxmox-microseg/src/ebpf.rs                  |  247 +++
 proxmox-microseg/src/lib.rs                   |   66 +
 proxmox-microseg/src/nic.rs                   |  120 ++
 proxmox-microseg/src/running_config.rs        |   26 +
 proxmox-ve-config/src/firewall/fw_macros.rs   |    7 +
 proxmox-ve-config/src/firewall/ports.rs       |   65 +
 .../src/firewall/types/rule_match.rs          |   63 +
 proxmox-ve-config/src/sdn/microseg/api.rs     |  710 +++++++++
 .../src/sdn/microseg/identity.rs              |  475 ++++++
 proxmox-ve-config/src/sdn/microseg/mod.rs     | 1394 +++++++++++++++++
 proxmox-ve-config/src/sdn/microseg/state.rs   |  616 ++++++++
 proxmox-ve-config/src/sdn/microseg/types.rs   |  549 +++++++
 proxmox-ve-config/src/sdn/mod.rs              |    1 +
 20 files changed, 4438 insertions(+)
 create mode 100644 proxmox-microseg/Cargo.toml
 create mode 100644 proxmox-microseg/debian/changelog
 create mode 100644 proxmox-microseg/debian/control
 create mode 100644 proxmox-microseg/debian/copyright
 create mode 100644 proxmox-microseg/debian/debcargo.toml
 create mode 100644 proxmox-microseg/debian/source/format
 create mode 100644 proxmox-microseg/src/ebpf.rs
 create mode 100644 proxmox-microseg/src/lib.rs
 create mode 100644 proxmox-microseg/src/nic.rs
 create mode 100644 proxmox-microseg/src/running_config.rs
 create mode 100644 proxmox-ve-config/src/sdn/microseg/api.rs
 create mode 100644 proxmox-ve-config/src/sdn/microseg/identity.rs
 create mode 100644 proxmox-ve-config/src/sdn/microseg/mod.rs
 create mode 100644 proxmox-ve-config/src/sdn/microseg/state.rs
 create mode 100644 proxmox-ve-config/src/sdn/microseg/types.rs


pve-cluster:

Hannes Laimer (1):
  cfs: add 'sdn/microseg.cfg' to observed files

 src/PVE/Cluster.pm  | 1 +
 src/pmxcfs/status.c | 1 +
 2 files changed, 2 insertions(+)


pve-access-control:

Hannes Laimer (1):
  permissions: add ACL path for SDN microsegmentation

 src/PVE/AccessControl.pm | 1 +
 1 file changed, 1 insertion(+)


proxmox-perl-rs:

Hannes Laimer (4):
  pve-rs: sdn: add microseg config binding
  pve-rs: sdn: microseg: bind the data plane
  pve-rs: sdn: microseg: check the data plane limits on render
  pve-rs: debian: fire the pve-microseg-update trigger on upgrade

 pve-rs/Cargo.toml                   |   2 +
 pve-rs/Makefile                     |   3 +
 pve-rs/debian/control               |   1 +
 pve-rs/debian/triggers              |   1 +
 pve-rs/src/bindings/sdn/microseg.rs | 350 ++++++++++++++++++++++++++++
 pve-rs/src/bindings/sdn/mod.rs      |   1 +
 6 files changed, 358 insertions(+)
 create mode 100644 pve-rs/src/bindings/sdn/microseg.rs


pve-network:

Hannes Laimer (10):
  sdn: zones: add vxlan-gbp option to vxlan and evpn zones
  sdn: microseg: add config, API and guest inventory
  sdn: dry-run: show pending microseg changes
  sdn: microseg: enforce the policy on guest NICs
  sdn: microseg: add tag matcher
  sdn: microseg: add name regex matcher
  sdn: microseg: add an update for a running NIC
  sdn: microseg: expose protocol, port and macro rule options
  sdn: microseg: allow nesting groups under a parent
  sdn: microseg: expose the fragments rule option

 debian/control                                |   1 +
 debian/libpve-network-perl.postinst           |  21 +
 debian/triggers                               |   1 +
 src/PVE/API2/Network/SDN.pm                   |  47 +-
 src/PVE/API2/Network/SDN/Makefile             |   2 +
 src/PVE/API2/Network/SDN/Microseg.pm          | 236 +++++++++
 .../API2/Network/SDN/Microseg/Assignment.pm   |  23 +
 src/PVE/API2/Network/SDN/Microseg/Base.pm     | 382 ++++++++++++++
 src/PVE/API2/Network/SDN/Microseg/Group.pm    |  17 +
 src/PVE/API2/Network/SDN/Microseg/Makefile    |  11 +
 src/PVE/API2/Network/SDN/Microseg/Rule.pm     |  22 +
 src/PVE/API2/Network/SDN/Zones.pm             |   8 +
 src/PVE/Network/SDN.pm                        |  25 +
 src/PVE/Network/SDN/Makefile                  |   1 +
 src/PVE/Network/SDN/Microseg.pm               | 494 ++++++++++++++++++
 src/PVE/Network/SDN/Zones.pm                  |  32 ++
 src/PVE/Network/SDN/Zones/EvpnPlugin.pm       |   8 +
 src/PVE/Network/SDN/Zones/VxlanPlugin.pm      |  12 +
 .../zones/evpn/gbp/expected_controller_config |  77 +++
 .../zones/evpn/gbp/expected_sdn_interfaces    |  45 ++
 src/test/zones/evpn/gbp/interfaces            |   7 +
 src/test/zones/evpn/gbp/sdn_config            |  26 +
 .../zones/vxlan/gbp/expected_sdn_interfaces   |  17 +
 src/test/zones/vxlan/gbp/interfaces           |   7 +
 src/test/zones/vxlan/gbp/sdn_config           |  11 +
 25 files changed, 1531 insertions(+), 2 deletions(-)
 create mode 100644 src/PVE/API2/Network/SDN/Microseg.pm
 create mode 100644 src/PVE/API2/Network/SDN/Microseg/Assignment.pm
 create mode 100644 src/PVE/API2/Network/SDN/Microseg/Base.pm
 create mode 100644 src/PVE/API2/Network/SDN/Microseg/Group.pm
 create mode 100644 src/PVE/API2/Network/SDN/Microseg/Makefile
 create mode 100644 src/PVE/API2/Network/SDN/Microseg/Rule.pm
 create mode 100644 src/PVE/Network/SDN/Microseg.pm
 create mode 100644 src/test/zones/evpn/gbp/expected_controller_config
 create mode 100644 src/test/zones/evpn/gbp/expected_sdn_interfaces
 create mode 100644 src/test/zones/evpn/gbp/interfaces
 create mode 100644 src/test/zones/evpn/gbp/sdn_config
 create mode 100644 src/test/zones/vxlan/gbp/expected_sdn_interfaces
 create mode 100644 src/test/zones/vxlan/gbp/interfaces
 create mode 100644 src/test/zones/vxlan/gbp/sdn_config


pve-manager:

Hannes Laimer (11):
  network: apply the microseg policy on reload
  pve-sdn-commit: apply the microseg policy at boot
  ui: sdn: zones: add vxlan-gbp checkbox to vxlan and evpn
  ui: sdn: add microsegmentation panel
  ui: sdn: dry-run: show pending microseg diff
  ui: sdn: microseg: add tag matcher
  ui: sdn: microseg: add name regex matcher
  ui: sdn: microseg: add narrowing to the rule form
  ui: sdn: microseg: show and edit the group hierarchy
  ui: sdn: microseg: add fragments option to the rule form
  ui: sdn: microseg: add a guest tab with its groups and rules

 PVE/API2/Network.pm                           |  31 +-
 bin/pve-sdn-commit                            |  47 ++-
 services/pve-sdn-commit.service               |   1 +
 www/css/ext6-pve.css                          |  23 ++
 www/manager6/Makefile                         |  10 +
 www/manager6/Utils.js                         |  98 +++++
 www/manager6/data/PermPathStore.js            |   1 +
 www/manager6/dc/Config.js                     |   8 +
 www/manager6/form/SDNMicrosegGroupSelector.js | 105 ++++++
 www/manager6/grid/FirewallRules.js            |   2 +
 www/manager6/lxc/Config.js                    |  10 +
 www/manager6/qemu/Config.js                   |  10 +
 www/manager6/sdn/MicrosegView.js              |  98 +++++
 www/manager6/sdn/SdnDiffView.js               |  29 ++
 www/manager6/sdn/microseg/AssignmentEdit.js   | 148 ++++++++
 www/manager6/sdn/microseg/AssignmentView.js   | 195 ++++++++++
 www/manager6/sdn/microseg/Base.js             | 262 ++++++++++++++
 www/manager6/sdn/microseg/GroupEdit.js        |  40 +++
 www/manager6/sdn/microseg/GuestPanel.js       | 117 ++++++
 www/manager6/sdn/microseg/RuleEdit.js         | 285 +++++++++++++++
 www/manager6/sdn/microseg/RuleView.js         | 338 ++++++++++++++++++
 www/manager6/sdn/microseg/Tree.js             | 278 ++++++++++++++
 www/manager6/sdn/zones/EvpnEdit.js            |   8 +
 www/manager6/sdn/zones/VxlanEdit.js           |  11 +
 24 files changed, 2126 insertions(+), 29 deletions(-)
 create mode 100644 www/manager6/form/SDNMicrosegGroupSelector.js
 create mode 100644 www/manager6/sdn/MicrosegView.js
 create mode 100644 www/manager6/sdn/microseg/AssignmentEdit.js
 create mode 100644 www/manager6/sdn/microseg/AssignmentView.js
 create mode 100644 www/manager6/sdn/microseg/Base.js
 create mode 100644 www/manager6/sdn/microseg/GroupEdit.js
 create mode 100644 www/manager6/sdn/microseg/GuestPanel.js
 create mode 100644 www/manager6/sdn/microseg/RuleEdit.js
 create mode 100644 www/manager6/sdn/microseg/RuleView.js
 create mode 100644 www/manager6/sdn/microseg/Tree.js


qemu-server:

Hannes Laimer (1):
  network: update the microseg enforcement on a tag or name change

 src/PVE/API2/Qemu.pm          | 26 ++++++++++++++++++++++++++
 src/PVE/QemuServer.pm         | 12 +++++++++++-
 src/PVE/QemuServer/Network.pm | 14 ++++++++++++++
 3 files changed, 51 insertions(+), 1 deletion(-)


pve-container:

Hannes Laimer (1):
  lxc: update the microseg enforcement on a tag or hostname change

 src/PVE/API2/LXC/Config.pm |  3 +++
 src/PVE/LXC.pm             | 28 ++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+)


pve-docs:

Hannes Laimer (2):
  sdn: zones: document the VXLAN-GBP option
  sdn: add microsegmentation section

 pvesdn.adoc | 403 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 403 insertions(+)


Summary over all repositories:
  98 files changed, 11514 insertions(+), 38 deletions(-)

-- 
Generated by murpp 0.12.0