[PATCH pve-docs v3 45/45] sdn: add microsegmentation section

[Hannes Laimer <h.laimer@proxmox.com>]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pvesdn.adoc | 403 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 391 insertions(+), 12 deletions(-)

diff --git a/pvesdn.adoc b/pvesdn.adoc
index 7a674597d470..84f44ab73592 100644
--- a/pvesdn.adoc
+++ b/pvesdn.adoc
@@ -147,6 +147,9 @@ following sections:
 * xref:pvesdn_config_vnet[VNets]: Create virtual network bridges and
   manage subnets
 
+* xref:pvesdn_config_microseg[Microsegmentation]: Segment traffic between
+  guests with group-based policies
+
 The Options category allows adding and managing additional services to be used
 in your SDN setup.
 
@@ -189,6 +192,83 @@ files located in '/etc/pve/sdn'.
 
 // TODO: extend implementation and technology details.
 
+[[pvesdn_microseg_background]]
+Microsegmentation Enforcement
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The node enforces the xref:pvesdn_config_microseg[microsegmentation] policy in
+the kernel, at the network interface of the receiving guest. It checks each
+packet on its own, without connection tracking, and drops a denied packet
+without an answer. Nothing runs in the guest, so a guest cannot turn it off or
+get around it.
+
+Each set of groups that an assignment lists gets a 16-bit number, its group
+identity, and sets that every rule treats alike share one. The node marks every
+packet an assigned interface sends with the identity of that interface. When a
+packet reaches an assigned interface, the node checks the identity of the
+sender against the identity of the receiver. Addresses play no part in this.
+
+A guest cannot forge its identity, since the node sets it. The network between
+the nodes is trusted to carry the identity unchanged, as it is trusted with
+VLAN tags.
+
+On the node, the identity lives in the low 16 bits of the packet mark and stays
+there when the node routes a packet. Other uses of the mark, such as policy
+routing by fwmark or firewall mark rules, must leave these bits alone. A rule
+that matches on the mark masks them out, for example
+`fwmark 0x10000/0xffff0000`. A packet the node itself sends with a value in
+these bits counts as that identity.
+
+Between nodes, only VXLAN and EVPN zones with the *VXLAN-GBP* option carry the
+identity, in the VXLAN header (see xref:pvesdn_zone_plugin_vxlan[VXLAN Zones]
+and xref:pvesdn_zone_plugin_evpn[EVPN Zones]). On every other zone, traffic
+from another node arrives without an identity and counts as untagged. The
+policy is bound to the guest interface, not to the zone. An interface on a
+bridge outside the SDN, such as `vmbr0`, is enforced like one on a VNet.
+
+For each pair of a sending and a receiving identity that a rule allows, the
+node keeps a verdict and the checks of the narrowed rules. A narrowed rule
+needs one check per protocol and per combination of a source and a destination
+port range, so a destination port list of `80,443` needs two checks. There is
+room for 65535 identities, 1048576 pairs and 512 checks per pair. A pair with
+more than 7 checks keeps the rest in a list shared with the pairs that have the
+same ones, and there is room for 16384 such lists.
+
+An SDN apply refuses a policy over one of these limits and names the limit, so
+such a policy never reaches a node. An identity that is no longer needed stays
+reserved for an hour, and the overview counts it as retired.
+
+A guest starts on a node only if all of its assigned interfaces can be enforced
+there, and a migration fails on the target node where a start would fail. If a
+node cannot read the applied SDN configuration, no guest with a network
+interface starts on it, since the node cannot tell which interfaces are
+assigned.
+
+If a node cannot apply the policy during an SDN apply, its network reload task
+fails and the node keeps its previous enforcement. On a node without loaded
+programs, for example after a failed apply at boot, no guest with an assigned
+interface starts until a reload succeeds. If the node cannot load a new
+package's programs, the previous package's programs stay in place.
+
+NOTE: Nodes with and without microsegmentation support can run in the same
+cluster during an upgrade, but the SDN apply should only be run on the nodes
+that support it. An apply from a node without support leaves the assigned
+guests that start after it unenforced.
+
+The guest firewall still applies, a packet has to pass both. A packet that
+microsegmentation drops gets no answer, even if the firewall is set to reject.
+
+With the guest firewall enabled, an interface usually sits behind its own
+firewall bridge, only the nftables firewall on a Linux bridge needs none. Behind
+a firewall bridge, the node cannot tell its own ARP and neighbor discovery
+replies from those of other senders. Such an interface therefore accepts both
+from every sender without an identity. Other interfaces accept them without an
+identity only from the node itself, or where a rule allows untagged traffic.
+
+WARNING: Behind a firewall bridge, an unassigned guest on the same VNet can
+answer ARP and neighbor discovery for addresses that are not its own. Only that
+guest's own firewall, filtering its IP and MAC addresses, prevents it.
+
 [[pvesdn_config_zone]]
 Zones
 -----
@@ -311,12 +391,13 @@ MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
   lower than the outgoing physical interface.
 
 VXLAN-GBP:: Enable VXLAN Group Based Policy (GBP) on the zone's VXLAN
-  interfaces. GBP carries a 16-bit group ID in the VXLAN header, taken from
-  the packet mark on the sending node and restored on the receiving one.
-  Every VTEP in the zone must have it enabled, since one without it drops
-  GBP-tagged traffic. All zones that use the same VXLAN port must agree on
-  this option, an SDN apply refuses a configuration where they do not.
-  Optional.
+  interfaces. xref:pvesdn_config_microseg[Microsegmentation] uses it to carry
+  an interface's group identity between nodes. GBP carries a 16-bit group ID
+  in the VXLAN header, taken from the packet mark on the sending node and
+  restored on the receiving one. Every VTEP in the zone must have it enabled,
+  since one without it drops GBP-tagged traffic. All zones that use the same
+  VXLAN port must agree on this option, an SDN apply refuses a configuration
+  where they do not.  Optional.
 
 NOTE: The kernel sets GBP only when it creates a VXLAN interface. Changing the
 option on an existing zone therefore recreates the zone's VXLAN interfaces on
@@ -384,12 +465,13 @@ MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
   defaults to 1450.
 
 VXLAN-GBP:: Enable VXLAN Group Based Policy (GBP) on the zone's VXLAN
-  interfaces. GBP carries a 16-bit group ID in the VXLAN header, taken from
-  the packet mark on the sending node and restored on the receiving one.
-  Every VTEP in the zone must have it enabled, since one without it drops
-  GBP-tagged traffic. All zones that use the same VXLAN port must agree on
-  this option, an SDN apply refuses a configuration where they do not.
-  Optional.
+  interfaces. xref:pvesdn_config_microseg[Microsegmentation] uses it to carry
+  an interface's group identity between nodes. GBP carries a 16-bit group ID
+  in the VXLAN header, taken from the packet mark on the sending node and
+  restored on the receiving one. Every VTEP in the zone must have it enabled,
+  since one without it drops GBP-tagged traffic. All zones that use the same
+  VXLAN port must agree on this option, an SDN apply refuses a configuration
+  where they do not.  Optional.
 
 NOTE: The kernel sets GBP only when it creates a VXLAN interface. Changing the
 option on an existing zone therefore recreates the zone's VXLAN interfaces on
@@ -467,6 +549,303 @@ DNS Zone Prefix:: Add a prefix to the domain registration, like
   <hostname>.prefix.<domain>  Optional.
 
 
+[[pvesdn_config_microseg]]
+Microsegmentation
+-----------------
+
+Microsegmentation allows or denies traffic between groups of guests at their
+network interfaces, independent of their addresses. The node enforces it in the
+kernel, so a guest cannot turn it off, and a guest keeps its rules when it
+moves to another node.
+
+The policy consists of three kinds of objects. *Groups* are labels.
+*Assignments* put guest network interfaces into groups, by guest, by tag or by
+name. *Rules* allow or deny traffic from one set of groups to another. An
+interface that an assignment covers receives only what a rule allows.
+
+Microsegmentation is managed under `Datacenter -> SDN -> Microsegmentation`.
+The *Microsegmentation* tab of a guest shows the groups of its interfaces and
+the rules that apply to them. Changes to the policy take effect with the next
+SDN apply.
+
+Rules are stateless and apply to one direction, so a connection needs a rule
+for each direction, or one rule with *Both Directions* set. Traffic that does
+not come from a guest in a group, such as the replies of a gateway or a DHCP
+server, is untagged. It needs a rule with the built-in `untagged` group as the
+source.
+
+Between nodes, only VXLAN and EVPN zones with the *VXLAN-GBP* option carry the
+groups of a guest. On every other zone, traffic from another node arrives
+untagged. The guest firewall still applies, a packet has to pass both. See
+xref:pvesdn_microseg_background[Microsegmentation Enforcement] for how the
+enforcement works and for its limits.
+
+Viewing the configuration needs `SDN.Audit` on the `/sdn/microseg` ACL path,
+changing it needs `SDN.Allocate` there. The guests an assignment covers are
+shown only where you have `VM.Audit` on the guest, as in the resource tree.
+
+[[pvesdn_microseg_group]]
+Groups
+~~~~~~
+
+A group is a label for guest interfaces. An interface gets its groups from the
+one assignment that covers it (see
+xref:pvesdn_microseg_assignment[Assignments]). Rules match the groups of an
+interface, never the interface itself.
+
+Groups can be nested. An interface in a subgroup is also in every group above
+it. With `database` a subgroup of `production`, an interface assigned
+`database` has both groups. An `any` or `all` match over `production` covers
+it, an `exact` match over `production` does not, and rules on `database` cover
+databases only.
+
+The groups that an `all` or `exact` match lists include the groups above them,
+so an `exact` match over `database` means `production` and `database`. A rule
+with such a match on a subgroup is therefore more specific than the same rule
+on the parent. Specificity decides a tie between rules of the same priority
+(see xref:pvesdn_microseg_rule[Rules]). An `any` match counts no groups.
+
+Group configuration options:
+
+Name:: 2 to 32 letters, digits, `-` and `_`, starting and ending with a letter
+  or digit. The name `untagged` and names shaped like a generated ID, such as
+  `vm100`, are reserved.
+
+Parent:: The group this one is a subgroup of.  Optional.
+
+Comment:: A descriptive comment.  Optional.
+
+NOTE: A group cannot be deleted while a rule names it, an assignment lists it,
+or it has subgroups.
+
+[[pvesdn_microseg_rule]]
+Rules
+~~~~~
+
+A rule allows or denies traffic from a source to a destination. Each side lists
+groups and a match kind, checked against the groups of the sending and of the
+receiving interface. The default is deny, so rules allow traffic, or deny part
+of what a broader rule allows. Traffic within one group needs a rule like any
+other.
+
+A rule applies to one direction. A TCP connection from `web` to `app` needs an
+allow from `web` to `app`, and one from `app` to `web` for the replies. A rule
+with *Both Directions* set is exactly that pair, the rule and its mirror with
+source and destination swapped.
+
+Two guests can only talk once they have resolved each other's addresses. ARP
+and the neighbor solicitations and advertisements of IPv6 therefore pass
+between a sender and a receiver in both directions as soon as a rule allows
+any traffic between them in one direction.
+
+They also pass when the node sends them itself. That is how the ARP replies of
+a gateway on the node and the suppression replies of an EVPN zone reach a
+guest. Nothing else passes without a rule.
+
+A rule can be narrowed to a protocol, ports or an ICMP type, like a firewall
+rule. A narrowed rule decides only the IP traffic it names. Anything that is
+not IP, ARP, LLDP or spanning tree for example, gets the verdict of the rules
+without narrowing.
+
+Narrowed rules are checked from the highest priority down to the priority of
+the highest rule without narrowing. At the same priority the more specific
+rule comes first, and of two equally specific rules the deny. The first
+narrowed rule that matches a packet decides it. Traffic that no narrowed rule
+matches gets the verdict of the rules without narrowing, or is denied if there
+are none.
+
+To allow SSH from `web` to `app`, allow `tcp` with destination port `22` from
+`web` to `app`, and `tcp` with source port `22` from `app` to `web` for the
+replies. One rule with *Both Directions* set and destination port `22` does the
+same, since the mirror swaps the ports. Either way the reply rule allows every
+segment from port `22`, not only replies, so `app` can also open connections to
+`web` from that port.
+
+NOTE: Neighbor discovery is ICMPv6, so a narrowed deny of ICMPv6 blocks it even
+where the reverse direction allows traffic.
+
+IP fragments after the first one carry no transport header. A rule narrowed by
+ports or an ICMP type never matches them, and neither does a rule narrowed to
+just a protocol. If only narrowed allows let traffic pass between a sender and a
+receiver, the later fragments are dropped and fragmented packets never arrive.
+
+An allow with *Fragments Only* set lets them pass. It also passes the later
+fragments of other datagrams, but those cannot be reassembled without their
+first fragment, which the other rules decide.
+
+An IPv6 fragment counts as the protocol its fragment header names. If that is
+an extension header instead of the transport protocol, a rule narrowed to the
+transport protocol does not match it, but a rule without a protocol does.
+
+Where narrowed rules apply to a sender and a receiver, a packet whose transport
+header cannot be read is dropped, since the rules cannot be checked against it.
+Examples are a first fragment that ends before the transport header, an IPv6
+packet with more than 12 extension headers, and a packet with more than two
+VLAN tags. Where no narrowed rule applies, such a packet gets the same verdict
+as any other.
+
+The built-in `untagged` group stands for traffic without a group. It comes from
+an interface that no assignment covers, from a host outside the policy, or from
+another node over a zone without the *VXLAN-GBP* option. To allow such traffic,
+list `untagged` in the source of a rule, on its own or next to other groups
+with the `any` match. This is how the replies of a gateway, of a DHCP server or
+of any other host outside the policy reach a guest.
+
+NOTE: In a source with the `all` or `exact` match, `untagged` must be the only
+listed group, and an `every` source never includes it. A destination never
+matches `untagged`, since every enforced interface has a group. It can list
+`untagged` only next to other groups with the `any` match, where it has no
+effect. Assignments cannot use it.
+
+Rule configuration options:
+
+Source Groups/Dest. Groups:: The groups the sending or the receiving interface
+  is checked against. Not set with the `every` match.
+
+Source Match/Dest. Match:: How the listed groups compare with the groups of an
+  interface. `any` matches an interface that has at least one of them, `all`
+  one that has all of them, and `exact` one that has exactly the listed
+  groups. For `all` and `exact`, the listed groups include the groups above
+  them. `every` lists no groups and matches every assigned interface.
+  Optional, defaults to `any`.
+
+Priority:: A number from 0 to 65535. Among the rules without narrowing that
+  match a sender and a receiver, the highest priority decides, and among those
+  the most specific. The specificity of a rule is the number of groups its
+  `all` and `exact` matches list, the groups above them included, and an `any`
+  or `every` match counts none. Equally specific rules deny if one of them
+  denies.  Optional, defaults to 0.
+
+Action:: Allow or deny.
+
+Protocol/Source port/Dest. port/ICMP type:: Narrow the rule, with the syntax of
+  firewall rules. Ports need a protocol that has ports, and an ICMP type needs
+  an ICMP protocol.  Optional.
+
+Macro:: A firewall macro, for example `HTTPS` or `Ceph`, in place of the
+  protocol, ports and ICMP type. Not combined with them.  Optional.
+
+Fragments Only:: Narrow the rule to the IP fragments after the first one, of
+  the given protocol or of every protocol. Not combined with ports, an ICMP
+  type or a macro.  Optional.
+
+Both Directions:: Also apply the rule with source and destination swapped,
+  ports included. Not with the `untagged` group or an ICMP type, whether set
+  directly or through a macro such as `Ping`.  Optional.
+
+Comment:: A descriptive comment.  Optional.
+
+The source, the destination, the direction and the narrowing identify a rule,
+so they cannot be changed on an existing rule. Delete the rule and create it
+again instead. The priority, the action and the comment can be changed.
+
+[[pvesdn_microseg_assignment]]
+Assignments
+~~~~~~~~~~~
+
+An assignment puts guest network interfaces into a set of groups. Its matcher
+selects one guest, the guests with certain tags, or the guests whose name
+matches a regular expression. You choose the matcher when you add the
+assignment, with *Guest Assignment*, *Tag Assignment* or *Regex Assignment* in
+the *Add* menu.
+
+Assignments are an ordered list, checked from first to last. An interface gets
+exactly the groups of the first assignment that covers it, and later ones have
+no effect on it. A new assignment goes to the top unless you give it a
+position, and you can move an existing one. Place an assignment for a single
+guest above the tag or regex assignments that also cover that guest.
+
+An interface that no assignment covers is not enforced. All traffic reaches
+it, and its own traffic reaches enforced guests as untagged.
+
+A combination of groups needs one assignment that lists all of them, since an
+interface gets the groups of a single assignment. For guests tagged both `prod`
+and `web`, add a tag assignment with the `all` match over both tags that lists
+both groups. Place it above the assignments that match only one of the tags.
+
+Tag and regex matchers use the current tags and name of a guest. They are
+matched each time an interface of the guest comes up, so a new guest or a
+hot-plugged interface is enforced from the start. A change of a running guest's
+tags or name made with `qm set` or `pct set`, through the API or the web
+interface, takes effect immediately.
+
+Changes to the assignments themselves, their order included, are pending until
+the next SDN apply.
+
+NOTE: Moving an assignment usually changes only its own place, so only that
+assignment shows up as a pending change. After many moves to the same spot, all
+assignments are renumbered once and show up together.
+
+A tag or a name can put a guest into a group. Setting one that an assignment
+matches, at creation or later, therefore needs `SDN.Use` on `/sdn/microseg`
+besides the right to change the option. This applies to a tag that a tag
+assignment matches, and to a rename into or out of the names a regex
+assignment matches.
+
+Assignment configuration options:
+
+Guest:: (guest matcher) The VM or container.
+
+Tags:: (tag matcher) The guest tags to match. They are normalized like guest
+  tags, so their case matters only with a case-sensitive datacenter tag style.
+
+Tag Match:: (tag matcher) `any` matches a guest with at least one of the
+  listed tags, `all` one with all of them, and `exact` one with exactly these
+  tags.  Optional, defaults to `any`.
+
+Name Regex:: (regex matcher) A regular expression in Rust syntax, ASCII only,
+  without backreferences or lookaround. It is matched anywhere in the VM name
+  or the container hostname, so anchor it with `^` and `$` for the whole name.
+  A VM without a name is never matched, not even by `.*`, since the `VM 100`
+  shown for it is not its name. A container created without a hostname gets
+  one like `CT100`, which is matched like any other name.
+
+Network Interface:: The number of a guest interface, from 0 to 31, `0` for
+  `net0`. Left empty, the assignment covers all interfaces of the guest.
+  Optional.
+
+Groups:: The groups that the covered interfaces get.
+
+Position:: The place in the list, 0 being the first. A position past the end
+  puts the assignment last. Left empty on an existing assignment, it keeps its
+  place.  Optional, defaults to 0 for a new assignment.
+
+Comment:: A descriptive comment.  Optional.
+
+The matcher fields and the network interface identify an assignment, so they
+cannot be changed on an existing one. Delete the assignment and create it
+again instead. The groups, the position and the comment can be changed.
+
+[[pvesdn_microseg_example]]
+Example
+~~~~~~~
+
+Web, application and database guests share one VNet and are tagged `web`,
+`app` and `db`. The web guests should reach the application guests, and those
+the databases, and nothing else.
+
+. Create the groups `web`, `app` and `db`.
+. For each tag, add a tag assignment that puts the guests with that tag into
+  the group of the same name.
+. Add a rule that allows `web` to `app` with *Both Directions* set, and one
+  that allows `app` to `db` the same way.
+. Add a rule that allows `untagged` to `web`, `app` and `db`, for the replies
+  of the gateway and the DHCP server.
+. Apply the SDN configuration.
+
+Web guests now reach the application guests, and those reach the databases.
+Traffic from web to database and traffic within a tier is dropped.
+
+NOTE: The rule for `untagged` also admits every guest on the VNet that no
+assignment covers. Either assign every guest, or narrow that rule to the
+protocols the outside traffic needs.
+
+To keep a staging environment apart from production, tag each guest with its
+environment as well and add the groups `prod` and `staging`. Then add one tag
+assignment per combination, such as `all` over `prod` and `web`, that lists
+both groups. In the rules, use the `all` match over the tier and the
+environment, for example `all` over `web` and `prod`.
+
 [[pvesdn_config_controllers]]
 Controllers
 -----------
-- 
2.47.3