From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id C0C3A1FF0E3 for ; Tue, 27 Oct 2026 10:03:33 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 05DA7217A2; Tue, 27 Oct 2026 10:00:13 +0100 (CET) From: Hannes Laimer To: pve-devel@lists.proxmox.com Subject: [PATCH pve-docs v3 45/45] sdn: add microsegmentation section Date: Tue, 27 Oct 2026 09:55:27 +0100 Message-ID: <20261027085527.702738-46-h.laimer@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261027085527.702738-1-h.laimer@proxmox.com> References: <20261027085527.702738-1-h.laimer@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793091382601 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.351 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: CZ4EF2BSCCZWN35NBUBVYRIUJMSCZGIU X-Message-ID-Hash: CZ4EF2BSCCZWN35NBUBVYRIUJMSCZGIU X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Signed-off-by: Hannes Laimer --- pvesdn.adoc | 403 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 391 insertions(+), 12 deletions(-) diff --git a/pvesdn.adoc b/pvesdn.adoc index 7a674597d470..84f44ab73592 100644 --- a/pvesdn.adoc +++ b/pvesdn.adoc @@ -147,6 +147,9 @@ following sections: * xref:pvesdn_config_vnet[VNets]: Create virtual network bridges and manage subnets +* xref:pvesdn_config_microseg[Microsegmentation]: Segment traffic between + guests with group-based policies + The Options category allows adding and managing additional services to be used in your SDN setup. @@ -189,6 +192,83 @@ files located in '/etc/pve/sdn'. // TODO: extend implementation and technology details. +[[pvesdn_microseg_background]] +Microsegmentation Enforcement +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The node enforces the xref:pvesdn_config_microseg[microsegmentation] policy in +the kernel, at the network interface of the receiving guest. It checks each +packet on its own, without connection tracking, and drops a denied packet +without an answer. Nothing runs in the guest, so a guest cannot turn it off or +get around it. + +Each set of groups that an assignment lists gets a 16-bit number, its group +identity, and sets that every rule treats alike share one. The node marks every +packet an assigned interface sends with the identity of that interface. When a +packet reaches an assigned interface, the node checks the identity of the +sender against the identity of the receiver. Addresses play no part in this. + +A guest cannot forge its identity, since the node sets it. The network between +the nodes is trusted to carry the identity unchanged, as it is trusted with +VLAN tags. + +On the node, the identity lives in the low 16 bits of the packet mark and stays +there when the node routes a packet. Other uses of the mark, such as policy +routing by fwmark or firewall mark rules, must leave these bits alone. A rule +that matches on the mark masks them out, for example +`fwmark 0x10000/0xffff0000`. A packet the node itself sends with a value in +these bits counts as that identity. + +Between nodes, only VXLAN and EVPN zones with the *VXLAN-GBP* option carry the +identity, in the VXLAN header (see xref:pvesdn_zone_plugin_vxlan[VXLAN Zones] +and xref:pvesdn_zone_plugin_evpn[EVPN Zones]). On every other zone, traffic +from another node arrives without an identity and counts as untagged. The +policy is bound to the guest interface, not to the zone. An interface on a +bridge outside the SDN, such as `vmbr0`, is enforced like one on a VNet. + +For each pair of a sending and a receiving identity that a rule allows, the +node keeps a verdict and the checks of the narrowed rules. A narrowed rule +needs one check per protocol and per combination of a source and a destination +port range, so a destination port list of `80,443` needs two checks. There is +room for 65535 identities, 1048576 pairs and 512 checks per pair. A pair with +more than 7 checks keeps the rest in a list shared with the pairs that have the +same ones, and there is room for 16384 such lists. + +An SDN apply refuses a policy over one of these limits and names the limit, so +such a policy never reaches a node. An identity that is no longer needed stays +reserved for an hour, and the overview counts it as retired. + +A guest starts on a node only if all of its assigned interfaces can be enforced +there, and a migration fails on the target node where a start would fail. If a +node cannot read the applied SDN configuration, no guest with a network +interface starts on it, since the node cannot tell which interfaces are +assigned. + +If a node cannot apply the policy during an SDN apply, its network reload task +fails and the node keeps its previous enforcement. On a node without loaded +programs, for example after a failed apply at boot, no guest with an assigned +interface starts until a reload succeeds. If the node cannot load a new +package's programs, the previous package's programs stay in place. + +NOTE: Nodes with and without microsegmentation support can run in the same +cluster during an upgrade, but the SDN apply should only be run on the nodes +that support it. An apply from a node without support leaves the assigned +guests that start after it unenforced. + +The guest firewall still applies, a packet has to pass both. A packet that +microsegmentation drops gets no answer, even if the firewall is set to reject. + +With the guest firewall enabled, an interface usually sits behind its own +firewall bridge, only the nftables firewall on a Linux bridge needs none. Behind +a firewall bridge, the node cannot tell its own ARP and neighbor discovery +replies from those of other senders. Such an interface therefore accepts both +from every sender without an identity. Other interfaces accept them without an +identity only from the node itself, or where a rule allows untagged traffic. + +WARNING: Behind a firewall bridge, an unassigned guest on the same VNet can +answer ARP and neighbor discovery for addresses that are not its own. Only that +guest's own firewall, filtering its IP and MAC addresses, prevents it. + [[pvesdn_config_zone]] Zones ----- @@ -311,12 +391,13 @@ MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes lower than the outgoing physical interface. VXLAN-GBP:: Enable VXLAN Group Based Policy (GBP) on the zone's VXLAN - interfaces. GBP carries a 16-bit group ID in the VXLAN header, taken from - the packet mark on the sending node and restored on the receiving one. - Every VTEP in the zone must have it enabled, since one without it drops - GBP-tagged traffic. All zones that use the same VXLAN port must agree on - this option, an SDN apply refuses a configuration where they do not. - Optional. + interfaces. xref:pvesdn_config_microseg[Microsegmentation] uses it to carry + an interface's group identity between nodes. GBP carries a 16-bit group ID + in the VXLAN header, taken from the packet mark on the sending node and + restored on the receiving one. Every VTEP in the zone must have it enabled, + since one without it drops GBP-tagged traffic. All zones that use the same + VXLAN port must agree on this option, an SDN apply refuses a configuration + where they do not. Optional. NOTE: The kernel sets GBP only when it creates a VXLAN interface. Changing the option on an existing zone therefore recreates the zone's VXLAN interfaces on @@ -384,12 +465,13 @@ MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes defaults to 1450. VXLAN-GBP:: Enable VXLAN Group Based Policy (GBP) on the zone's VXLAN - interfaces. GBP carries a 16-bit group ID in the VXLAN header, taken from - the packet mark on the sending node and restored on the receiving one. - Every VTEP in the zone must have it enabled, since one without it drops - GBP-tagged traffic. All zones that use the same VXLAN port must agree on - this option, an SDN apply refuses a configuration where they do not. - Optional. + interfaces. xref:pvesdn_config_microseg[Microsegmentation] uses it to carry + an interface's group identity between nodes. GBP carries a 16-bit group ID + in the VXLAN header, taken from the packet mark on the sending node and + restored on the receiving one. Every VTEP in the zone must have it enabled, + since one without it drops GBP-tagged traffic. All zones that use the same + VXLAN port must agree on this option, an SDN apply refuses a configuration + where they do not. Optional. NOTE: The kernel sets GBP only when it creates a VXLAN interface. Changing the option on an existing zone therefore recreates the zone's VXLAN interfaces on @@ -467,6 +549,303 @@ DNS Zone Prefix:: Add a prefix to the domain registration, like .prefix. Optional. +[[pvesdn_config_microseg]] +Microsegmentation +----------------- + +Microsegmentation allows or denies traffic between groups of guests at their +network interfaces, independent of their addresses. The node enforces it in the +kernel, so a guest cannot turn it off, and a guest keeps its rules when it +moves to another node. + +The policy consists of three kinds of objects. *Groups* are labels. +*Assignments* put guest network interfaces into groups, by guest, by tag or by +name. *Rules* allow or deny traffic from one set of groups to another. An +interface that an assignment covers receives only what a rule allows. + +Microsegmentation is managed under `Datacenter -> SDN -> Microsegmentation`. +The *Microsegmentation* tab of a guest shows the groups of its interfaces and +the rules that apply to them. Changes to the policy take effect with the next +SDN apply. + +Rules are stateless and apply to one direction, so a connection needs a rule +for each direction, or one rule with *Both Directions* set. Traffic that does +not come from a guest in a group, such as the replies of a gateway or a DHCP +server, is untagged. It needs a rule with the built-in `untagged` group as the +source. + +Between nodes, only VXLAN and EVPN zones with the *VXLAN-GBP* option carry the +groups of a guest. On every other zone, traffic from another node arrives +untagged. The guest firewall still applies, a packet has to pass both. See +xref:pvesdn_microseg_background[Microsegmentation Enforcement] for how the +enforcement works and for its limits. + +Viewing the configuration needs `SDN.Audit` on the `/sdn/microseg` ACL path, +changing it needs `SDN.Allocate` there. The guests an assignment covers are +shown only where you have `VM.Audit` on the guest, as in the resource tree. + +[[pvesdn_microseg_group]] +Groups +~~~~~~ + +A group is a label for guest interfaces. An interface gets its groups from the +one assignment that covers it (see +xref:pvesdn_microseg_assignment[Assignments]). Rules match the groups of an +interface, never the interface itself. + +Groups can be nested. An interface in a subgroup is also in every group above +it. With `database` a subgroup of `production`, an interface assigned +`database` has both groups. An `any` or `all` match over `production` covers +it, an `exact` match over `production` does not, and rules on `database` cover +databases only. + +The groups that an `all` or `exact` match lists include the groups above them, +so an `exact` match over `database` means `production` and `database`. A rule +with such a match on a subgroup is therefore more specific than the same rule +on the parent. Specificity decides a tie between rules of the same priority +(see xref:pvesdn_microseg_rule[Rules]). An `any` match counts no groups. + +Group configuration options: + +Name:: 2 to 32 letters, digits, `-` and `_`, starting and ending with a letter + or digit. The name `untagged` and names shaped like a generated ID, such as + `vm100`, are reserved. + +Parent:: The group this one is a subgroup of. Optional. + +Comment:: A descriptive comment. Optional. + +NOTE: A group cannot be deleted while a rule names it, an assignment lists it, +or it has subgroups. + +[[pvesdn_microseg_rule]] +Rules +~~~~~ + +A rule allows or denies traffic from a source to a destination. Each side lists +groups and a match kind, checked against the groups of the sending and of the +receiving interface. The default is deny, so rules allow traffic, or deny part +of what a broader rule allows. Traffic within one group needs a rule like any +other. + +A rule applies to one direction. A TCP connection from `web` to `app` needs an +allow from `web` to `app`, and one from `app` to `web` for the replies. A rule +with *Both Directions* set is exactly that pair, the rule and its mirror with +source and destination swapped. + +Two guests can only talk once they have resolved each other's addresses. ARP +and the neighbor solicitations and advertisements of IPv6 therefore pass +between a sender and a receiver in both directions as soon as a rule allows +any traffic between them in one direction. + +They also pass when the node sends them itself. That is how the ARP replies of +a gateway on the node and the suppression replies of an EVPN zone reach a +guest. Nothing else passes without a rule. + +A rule can be narrowed to a protocol, ports or an ICMP type, like a firewall +rule. A narrowed rule decides only the IP traffic it names. Anything that is +not IP, ARP, LLDP or spanning tree for example, gets the verdict of the rules +without narrowing. + +Narrowed rules are checked from the highest priority down to the priority of +the highest rule without narrowing. At the same priority the more specific +rule comes first, and of two equally specific rules the deny. The first +narrowed rule that matches a packet decides it. Traffic that no narrowed rule +matches gets the verdict of the rules without narrowing, or is denied if there +are none. + +To allow SSH from `web` to `app`, allow `tcp` with destination port `22` from +`web` to `app`, and `tcp` with source port `22` from `app` to `web` for the +replies. One rule with *Both Directions* set and destination port `22` does the +same, since the mirror swaps the ports. Either way the reply rule allows every +segment from port `22`, not only replies, so `app` can also open connections to +`web` from that port. + +NOTE: Neighbor discovery is ICMPv6, so a narrowed deny of ICMPv6 blocks it even +where the reverse direction allows traffic. + +IP fragments after the first one carry no transport header. A rule narrowed by +ports or an ICMP type never matches them, and neither does a rule narrowed to +just a protocol. If only narrowed allows let traffic pass between a sender and a +receiver, the later fragments are dropped and fragmented packets never arrive. + +An allow with *Fragments Only* set lets them pass. It also passes the later +fragments of other datagrams, but those cannot be reassembled without their +first fragment, which the other rules decide. + +An IPv6 fragment counts as the protocol its fragment header names. If that is +an extension header instead of the transport protocol, a rule narrowed to the +transport protocol does not match it, but a rule without a protocol does. + +Where narrowed rules apply to a sender and a receiver, a packet whose transport +header cannot be read is dropped, since the rules cannot be checked against it. +Examples are a first fragment that ends before the transport header, an IPv6 +packet with more than 12 extension headers, and a packet with more than two +VLAN tags. Where no narrowed rule applies, such a packet gets the same verdict +as any other. + +The built-in `untagged` group stands for traffic without a group. It comes from +an interface that no assignment covers, from a host outside the policy, or from +another node over a zone without the *VXLAN-GBP* option. To allow such traffic, +list `untagged` in the source of a rule, on its own or next to other groups +with the `any` match. This is how the replies of a gateway, of a DHCP server or +of any other host outside the policy reach a guest. + +NOTE: In a source with the `all` or `exact` match, `untagged` must be the only +listed group, and an `every` source never includes it. A destination never +matches `untagged`, since every enforced interface has a group. It can list +`untagged` only next to other groups with the `any` match, where it has no +effect. Assignments cannot use it. + +Rule configuration options: + +Source Groups/Dest. Groups:: The groups the sending or the receiving interface + is checked against. Not set with the `every` match. + +Source Match/Dest. Match:: How the listed groups compare with the groups of an + interface. `any` matches an interface that has at least one of them, `all` + one that has all of them, and `exact` one that has exactly the listed + groups. For `all` and `exact`, the listed groups include the groups above + them. `every` lists no groups and matches every assigned interface. + Optional, defaults to `any`. + +Priority:: A number from 0 to 65535. Among the rules without narrowing that + match a sender and a receiver, the highest priority decides, and among those + the most specific. The specificity of a rule is the number of groups its + `all` and `exact` matches list, the groups above them included, and an `any` + or `every` match counts none. Equally specific rules deny if one of them + denies. Optional, defaults to 0. + +Action:: Allow or deny. + +Protocol/Source port/Dest. port/ICMP type:: Narrow the rule, with the syntax of + firewall rules. Ports need a protocol that has ports, and an ICMP type needs + an ICMP protocol. Optional. + +Macro:: A firewall macro, for example `HTTPS` or `Ceph`, in place of the + protocol, ports and ICMP type. Not combined with them. Optional. + +Fragments Only:: Narrow the rule to the IP fragments after the first one, of + the given protocol or of every protocol. Not combined with ports, an ICMP + type or a macro. Optional. + +Both Directions:: Also apply the rule with source and destination swapped, + ports included. Not with the `untagged` group or an ICMP type, whether set + directly or through a macro such as `Ping`. Optional. + +Comment:: A descriptive comment. Optional. + +The source, the destination, the direction and the narrowing identify a rule, +so they cannot be changed on an existing rule. Delete the rule and create it +again instead. The priority, the action and the comment can be changed. + +[[pvesdn_microseg_assignment]] +Assignments +~~~~~~~~~~~ + +An assignment puts guest network interfaces into a set of groups. Its matcher +selects one guest, the guests with certain tags, or the guests whose name +matches a regular expression. You choose the matcher when you add the +assignment, with *Guest Assignment*, *Tag Assignment* or *Regex Assignment* in +the *Add* menu. + +Assignments are an ordered list, checked from first to last. An interface gets +exactly the groups of the first assignment that covers it, and later ones have +no effect on it. A new assignment goes to the top unless you give it a +position, and you can move an existing one. Place an assignment for a single +guest above the tag or regex assignments that also cover that guest. + +An interface that no assignment covers is not enforced. All traffic reaches +it, and its own traffic reaches enforced guests as untagged. + +A combination of groups needs one assignment that lists all of them, since an +interface gets the groups of a single assignment. For guests tagged both `prod` +and `web`, add a tag assignment with the `all` match over both tags that lists +both groups. Place it above the assignments that match only one of the tags. + +Tag and regex matchers use the current tags and name of a guest. They are +matched each time an interface of the guest comes up, so a new guest or a +hot-plugged interface is enforced from the start. A change of a running guest's +tags or name made with `qm set` or `pct set`, through the API or the web +interface, takes effect immediately. + +Changes to the assignments themselves, their order included, are pending until +the next SDN apply. + +NOTE: Moving an assignment usually changes only its own place, so only that +assignment shows up as a pending change. After many moves to the same spot, all +assignments are renumbered once and show up together. + +A tag or a name can put a guest into a group. Setting one that an assignment +matches, at creation or later, therefore needs `SDN.Use` on `/sdn/microseg` +besides the right to change the option. This applies to a tag that a tag +assignment matches, and to a rename into or out of the names a regex +assignment matches. + +Assignment configuration options: + +Guest:: (guest matcher) The VM or container. + +Tags:: (tag matcher) The guest tags to match. They are normalized like guest + tags, so their case matters only with a case-sensitive datacenter tag style. + +Tag Match:: (tag matcher) `any` matches a guest with at least one of the + listed tags, `all` one with all of them, and `exact` one with exactly these + tags. Optional, defaults to `any`. + +Name Regex:: (regex matcher) A regular expression in Rust syntax, ASCII only, + without backreferences or lookaround. It is matched anywhere in the VM name + or the container hostname, so anchor it with `^` and `$` for the whole name. + A VM without a name is never matched, not even by `.*`, since the `VM 100` + shown for it is not its name. A container created without a hostname gets + one like `CT100`, which is matched like any other name. + +Network Interface:: The number of a guest interface, from 0 to 31, `0` for + `net0`. Left empty, the assignment covers all interfaces of the guest. + Optional. + +Groups:: The groups that the covered interfaces get. + +Position:: The place in the list, 0 being the first. A position past the end + puts the assignment last. Left empty on an existing assignment, it keeps its + place. Optional, defaults to 0 for a new assignment. + +Comment:: A descriptive comment. Optional. + +The matcher fields and the network interface identify an assignment, so they +cannot be changed on an existing one. Delete the assignment and create it +again instead. The groups, the position and the comment can be changed. + +[[pvesdn_microseg_example]] +Example +~~~~~~~ + +Web, application and database guests share one VNet and are tagged `web`, +`app` and `db`. The web guests should reach the application guests, and those +the databases, and nothing else. + +. Create the groups `web`, `app` and `db`. +. For each tag, add a tag assignment that puts the guests with that tag into + the group of the same name. +. Add a rule that allows `web` to `app` with *Both Directions* set, and one + that allows `app` to `db` the same way. +. Add a rule that allows `untagged` to `web`, `app` and `db`, for the replies + of the gateway and the DHCP server. +. Apply the SDN configuration. + +Web guests now reach the application guests, and those reach the databases. +Traffic from web to database and traffic within a tier is dropped. + +NOTE: The rule for `untagged` also admits every guest on the VNet that no +assignment covers. Either assign every guest, or narrow that rule to the +protocols the outside traffic needs. + +To keep a staging environment apart from production, tag each guest with its +environment as well and add the groups `prod` and `staging`. Then add one tag +assignment per combination, such as `all` over `prod` and `web`, that lists +both groups. In the rules, use the `all` match over the tier and the +environment, for example `all` over `web` and `prod`. + [[pvesdn_config_controllers]] Controllers ----------- -- 2.47.3