public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Stefan Hanreich <s.hanreich@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH pve-docs 14/14] pvesdn: simple zones: document dedicated VRFs
Date: Thu, 19 Nov 2026 20:45:46 +0100	[thread overview]
Message-ID: <20261119194547.674512-15-s.hanreich@proxmox.com> (raw)
In-Reply-To: <20261119194547.674512-1-s.hanreich@proxmox.com>

Describe how traffic of a simple zone with a dedicated VRF can leave the
zone, which interfaces belong to the VRF, and what this means for DHCP,
SNAT and the firewall.

Document the conntrack zones reserved for Proxmox VE, so that admins
do not use them in their own rules, that proxmox-firewall needs to be
installed to set them, and how to apply them again after changing the
interfaces of a VRF outside of the network API.

Also document the limitations that admins need to be aware of: no
isolation of guests with pve-firewall, firewall rules and SDN IPSets
matching addresses only, and the need to block access to infrastructure
addresses in the VRF themselves. For SNAT, every host interface in the
VRF acts as uplink, guests appear with the address of the node, so
access lists of an upstream router cannot tell them apart, and packets
with a forged source address from the subnet are masqueraded unless
reverse path filtering or the firewall drops them.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 pvesdn.adoc | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 100 insertions(+)

diff --git a/pvesdn.adoc b/pvesdn.adoc
index b701636..0c7ebb4 100644
--- a/pvesdn.adoc
+++ b/pvesdn.adoc
@@ -239,6 +239,106 @@ bridge is not linked to a physical interface, and VM traffic is only local on
 each the node.
 It can be used in NAT or routed setups.
 
+Simple zone configuration options:
+
+Dedicated VRF:: Put the VNets of the zone into a VRF of their own, see
+  xref:pvesdn_zone_plugin_simple_vrf[Dedicated VRF]. Optional, disabled by
+  default.
+
+[[pvesdn_zone_plugin_simple_vrf]]
+Dedicated VRF
+^^^^^^^^^^^^^
+
+With a dedicated VRF, the VNets of a simple zone are put into the VRF
+`vrf_<zone>`, which has a routing table of its own. Traffic of the zone is
+routed separately from the routing table of the host and from other zones, so
+the subnets of different zones with a dedicated VRF may overlap.
+
+Traffic can only leave the zone through interfaces in its VRF:
+
+* Host interfaces that the network configuration of the node puts into the VRF,
+  for example a VLAN interface to an upstream router.
+* Interfaces of xref:pvesdn_config_fabrics[fabrics] in the zone.
+
+Without any such interface, the zone is isolated. Services of the host, such as
+SSH or the web interface, cannot be reached from the zone, as long as the
+`net.ipv4.tcp_l3mdev_accept` and `net.ipv4.udp_l3mdev_accept` sysctls are not
+enabled.
+
+DHCP:: The DHCP server of the zone runs in its VRF. Guests use it as DNS server
+  by default, which needs a route to the upstream DNS servers through an
+  interface in the VRF. Otherwise, set a DNS server reachable from the zone with
+  the `dhcp-dns-server` option of the subnet.
+
+SNAT:: Subnets with SNAT enabled are masqueraded on every host interface in the
+  VRF. Only interfaces configured in `/etc/network/interfaces` itself are taken
+  into account, not the ones in files included from it. Interfaces of fabrics
+  are not used for SNAT. If routing balances connections across several such
+  interfaces, connections of a guest leave with different source addresses.
+  Connections are dropped if the interface goes down, even briefly.
++
+Every such interface acts as an uplink: traffic of the subnet routed through it
+carries the address of the node, also towards networks that should see the
+addresses of the guests. Only put interfaces into the VRF that should be used
+as uplinks.
++
+SNAT does not protect guests from incoming connections, use the firewall for
+that. Packets that arrive on an uplink with a source address from the subnet
+are masqueraded as well if they are routed out through an uplink. To drop them,
+enable strict reverse path filtering for IPv4 on the uplinks, by setting the
+`net.ipv4.conf.<interface>.rp_filter` sysctl to `1`, or drop such packets with
+the firewall. Strict reverse path filtering also drops replies that arrive on
+another uplink than the one routing would use to reach their source.
+
+Connection tracking:: Each zone with a dedicated VRF gets a conntrack zone, so
+  that connections of different VRFs never share a conntrack entry. It is
+  assigned automatically from the range 16384 to 17407 and shown as `ct-zone`
+  of the zone. {pve} reserves the conntrack zones 16384 to 32767, do not use
+  them in your own rules.
++
+The conntrack zone is set for the VRF device and all interfaces in the VRF by
+the `proxmox-firewall` package, which needs to be installed, even if the
+nftables-based firewall is not used. The rules are loaded into the nftables
+table `inet proxmox-sdn-ct-zones` when the network configuration is applied
+and at boot. If the interfaces in a VRF are changed in another way, for example
+by running `ifreload -a` manually, or if the nftables ruleset is flushed, for
+example by enabling `nftables.service` with its default configuration, apply
+the network configuration again.
++
+After a zone is deleted, a new zone may reuse its conntrack zone. Remaining
+connection tracking entries of the deleted zone expire after their timeout.
+
+Firewall:: Only the xref:pve_firewall_nft[nftables-based firewall] isolates the
+  connections of guests between zone VRFs. With `pve-firewall`, guests with the
+  firewall enabled on a network device share their connection tracking state
+  across all VRFs, and applying the configuration warns about this. Conntrack
+  helpers of `pve-firewall` do not apply in zone VRFs.
++
+Firewall rules and the IPSets generated for VNets match addresses only. With
+overlapping subnets, a rule applies to every zone with a matching subnet. For
+traffic of a zone VRF, the input interface seen by the host firewall is the VRF
+device `vrf_<zone>`.
+
+Infrastructure addresses in the VRF, such as the addresses of fabric interfaces
+and nodes or the address of an upstream router, are reachable from guests in the
+zone. To block them with the nftables-based firewall, add a datacenter IPSet,
+for example `vrf-infra`, with these addresses and drop traffic from the subnets
+of the zone to them:
+
+----
+# /etc/pve/firewall/cluster.fw
+
+[RULES]
+
+FORWARD DROP -source +sdn/vnet0-all -dest +dc/vrf-infra
+IN DROP -source +sdn/vnet0-all -dest +dc/vrf-infra
+----
+
+An upstream router should additionally protect its own addresses with access
+lists on its interface to the zone. This does not work for subnets with SNAT, as
+the router only sees the address of the node for their traffic, so only the
+firewall of the node can block it.
+
 
 [[pvesdn_zone_plugin_vlan]]
 VLAN Zones
-- 
2.47.3





      parent reply	other threads:[~2026-11-19 19:47 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 19:45 [PATCH docs/manager/network/proxmox{,-ve-rs,-firewall} 00/14] sdn: conntrack zones and SNAT for simple zones with a dedicated VRF Stefan Hanreich
2026-11-19 19:45 ` [PATCH proxmox 01/14] network-api: expose the master device of a link Stefan Hanreich
2026-11-19 19:45 ` [PATCH proxmox-ve-rs 02/14] sdn: config: read the conntrack zone of zones with a dedicated VRF Stefan Hanreich
2026-11-19 19:45 ` [PATCH proxmox-ve-rs 03/14] sdn: conntrack: map the interfaces of zone VRFs to their conntrack zone Stefan Hanreich
2026-11-19 19:45 ` [PATCH pve-network 04/14] sdn: simple: SNAT via the uplinks of zone VRFs Stefan Hanreich
2026-11-19 19:45 ` [PATCH pve-network 05/14] sdn: zones: assign a conntrack zone to zones with a dedicated VRF Stefan Hanreich
2026-11-19 19:45 ` [PATCH proxmox-firewall 06/14] firewall: apply vnet rules to host traffic from vnets in a VRF Stefan Hanreich
2026-11-19 19:45 ` [PATCH proxmox-firewall 07/14] nftables: expression: add a constructor for map lookups Stefan Hanreich
2026-11-19 19:45 ` [PATCH proxmox-firewall 08/14] nftables: statement: add a helper for setting the conntrack zone Stefan Hanreich
2026-11-19 19:45 ` [PATCH proxmox-firewall 09/14] nftables: expression: convert the conntrack zones of zone VRFs Stefan Hanreich
2026-11-19 19:45 ` [PATCH proxmox-firewall 10/14] firewall: add a command applying " Stefan Hanreich
2026-11-19 19:45 ` [PATCH proxmox-firewall 11/14] firewall: put guest traffic of zone VRFs into their conntrack zone Stefan Hanreich
2026-11-19 19:45 ` [PATCH pve-manager 12/14] network: apply the conntrack zones of zone VRFs Stefan Hanreich
2026-11-19 19:45 ` [PATCH pve-manager 13/14] network: warn if pve-firewall cannot isolate guests in " Stefan Hanreich
2026-11-19 19:45 ` Stefan Hanreich [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261119194547.674512-15-s.hanreich@proxmox.com \
    --to=s.hanreich@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal