From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 5AA0D1FF0AA for ; Tue, 20 Oct 2026 15:54:35 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D9D82216DC; Tue, 20 Oct 2026 15:54:32 +0200 (CEST) Message-ID: Date: Tue, 20 Oct 2026 15:54:26 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH pve-docs v2 1/1] network/sdn: update and clean up VLAN information To: Thomas Ellmenreich , pve-devel@lists.proxmox.com References: <20261020064508.32479-1-t.ellmenreich@proxmox.com> Content-Language: en-US From: Stefan Hanreich In-Reply-To: <20261020064508.32479-1-t.ellmenreich@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.333 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: JQRQLBTNRVFNE7LSCD3J74WTN2JLGJCR X-Message-ID-Hash: JQRQLBTNRVFNE7LSCD3J74WTN2JLGJCR X-MailFrom: s.hanreich@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 10/20/26 8:45 AM, Thomas Ellmenreich wrote: > Adjusted some of the details related to VLAN setups, both with the traditional > methods as well as the SDN setup. > > - Removed an outdated example and added more information to the remaining > examples. First, by adding the VLAN interface on the bridge, and then > showcasing that VLAN tags can be set in the guest network configurations. > > - Added SDN as a possible way to configure VLANs for guests, as well as adding > an example for SDN. > > - Added a "Common Issues" section to collect common networking issues. > > - Added a "Troubleshooting" section, to collect information on how to fix VLAN > related issues. > > Suggested-by: Stefan Hanreich > Signed-off-by: Thomas Ellmenreich > --- > > Some changes for the VLAN configuration documentation, would apprechiate any > help in regards to the wording and details of said documentation. > > Changelog: > > - PATCH v1 -> v2 > * Added a "Common Issues" section, that recounts different problems that > might appear. > > * Added a "Troubleshooting" section to show different ways to troubleshoot > problems with the networking. > > * Added a SDN VLAN example. > > * Some rewording, and adjustment of different sections. > > pve-network.adoc | 161 +++++++++++++++++++++++++++++++++++------------ > pvesdn.adoc | 2 +- > 2 files changed, 121 insertions(+), 42 deletions(-) > > diff --git a/pve-network.adoc b/pve-network.adoc > index 2b3e664..b1a598c 100644 > --- a/pve-network.adoc > +++ b/pve-network.adoc > @@ -637,6 +637,53 @@ completely done inside the guest and can not be influenced from the > outside. The benefit is that you can use more than one VLAN on a > single virtual NIC. > > +* *SDN configured VLAN:* > +It is also possible to xref:pvesdn_config_vnet[configure VLANs in SDN] by > +creating a VLAN zone on the bridge and then creating VNets for every VLAN tag. > +In this case, it is also recommended to use a VLAN-aware bridge. > + > +.Example: SDN VLAN Zones with VLAN aware bridges > + > +This is the preferred solution if users have a small amount of VLANs inside > +their cluster. On every node, users should configure a VLAN-aware bridge with > +the respective carrier for their VLANs: > + > +---- > +auto vmbr0 > +iface vmbr0 inet manual > + bridge-ports nic0 > + bridge-stp off > + bridge-fd 0 > + bridge-vlan-aware yes > + bridge-vids 2-4094 > +---- > + > +Then create an SDN zone on top of that bridge: > + > +---- > +vlan: vlanz > + bridge vmbr0 > + ipam pve > +---- > + > +Then, for each VLAN they want to use, create a VNet: > + > +---- > +vnet: vnet1234 > + zone vlanz > + tag 1234 > +---- > + > +After applying the SDN configuration, users can then select vnet1234 in the > +network device configuration of their guests. This method does not allow to > +configure an IP for the host directly on a VNet - but it can be done manually > +by adding an entry for the VNet to /etc/network/interfaces: > + > +---- > +auto vnet1234 > +iface vnet1234 inet static > + address 192.0.2.1/24 > +---- > > VLAN on the Host > ^^^^^^^^^^^^^^^^ > @@ -644,63 +691,47 @@ VLAN on the Host > To allow host communication with an isolated network. It is possible > to apply VLAN tags to any network device (NIC, Bond, Bridge). In > general, you should configure the VLAN on the interface with the least > -abstraction layers between itself and the physical NIC. > +abstraction layers between itself and the physical NIC. But it would still > +be best to configure the VLAN on a bridge instead of a bond or the NIC > +directly to avoid some of the issues mentioned xref:vlan_common_issues[here]. > > For example, in a default configuration where you want to place > the host management address on a separate VLAN. > > - > -.Example: Use VLAN 5 for the {pve} management IP with traditional Linux bridge > +.Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge > ---- > auto lo > iface lo inet loopback > > iface eno1 inet manual > > -iface eno1.5 inet manual > - > -auto vmbr0v5 > -iface vmbr0v5 inet static > - address 10.10.10.2/24 > - gateway 10.10.10.1 > - bridge-ports eno1.5 > - bridge-stp off > - bridge-fd 0 > - > auto vmbr0 > iface vmbr0 inet manual > bridge-ports eno1 > bridge-stp off > bridge-fd 0 > - > ----- > - > -.Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge > ----- > -auto lo > -iface lo inet loopback > - > -iface eno1 inet manual > - > + bridge-vlan-aware yes > + bridge-vids 2-4094 > > auto vmbr0.5 > iface vmbr0.5 inet static > address 10.10.10.2/24 > gateway 10.10.10.1 > +---- > + > +Guests can then reference the VLAN tag directly in their network > +configuration. Compared to an SDN-configured solution, this has the > +disadvantage that specific VLANs cannot be named, but the advantage that > +each VLAN does not have to be configured individually. > > -auto vmbr0 > -iface vmbr0 inet manual > - bridge-ports eno1 > - bridge-stp off > - bridge-fd 0 > - bridge-vlan-aware yes > - bridge-vids 2-4094 > +---- > +net0: name=eth0,bridge=vmbr0,tag=5,... > ---- > > The next example is the same setup but a bond is used to > make this network fail-safe. > > -.Example: Use VLAN 5 with bond0 for the {pve} management IP with traditional Linux bridge > +.Example: Use VLAN 5 with bond0 for the {pve} management IP with VLAN aware Linux bridge > ---- > auto lo > iface lo inet loopback > @@ -716,24 +747,72 @@ iface bond0 inet manual > bond-mode 802.3ad > bond-xmit-hash-policy layer2+3 > > -iface bond0.5 inet manual > - > -auto vmbr0v5 > -iface vmbr0v5 inet static > - address 10.10.10.2/24 > - gateway 10.10.10.1 > - bridge-ports bond0.5 > - bridge-stp off > - bridge-fd 0 > - > auto vmbr0 > iface vmbr0 inet manual > bridge-ports bond0 > + bridge-vlan-aware yes > + bridge-vids 2-4094 > bridge-stp off > bridge-fd 0 > > +auto vmbr0.5 > +iface vmbr0.5 inet static > + address 10.10.10.2/24 > + gateway 10.10.10.1 > + > ---- > > +Troubleshooting > +^^^^^^^^^^^^^^^ > + > +There are some helpful commands that can be utilised when debugging such issues: > + > +---- > +$ bridge -compressvlans vlan show > +---- > + > +This prints a list of bridges and their ports, as well as the VLANs that are > +currently configured for them: > + > +---- > +port vlan-id > +nic0 1 PVID Egress Untagged > + 2-4094 > +vmbr0 1 PVID Egress Untagged > +veth100i0 1234 PVID Egress Untagged > +---- > + > +---- > +$ bridge fdb show > +---- > + > +This prints a list of the current forwarding database entries of all bridges. > +The bridge decides how to forward a given packet based on this table. An entry > +of the fdb generally looks like this: > + > +---- > +aa:aa:aa:d8:36:7c dev nic0 vlan 49 master vmbr0 permanent > +---- > + > +This means that packets arriving on vmbr0, that have VLAN Tag 49 set and > +destination MAC address aa:aa:aa:d8:36:7c will get forwarded to the port nic0. > + > +[[vlan_common_issues]] > +Common Issues > +^^^^^^^^^^^^^ > + > +.Issue: Configuring VLAN on interface that is member of bridge > + > +It's not recommended to configure a VLAN on an interface that is a member of a > +bridge. Doing so will "blackhole" all traffic of that VLAN on the bridge. Might need a more specific example so this becomes clearer? It's hard to understand otherwise imo. > + > +.Issue: Limited number of tags when VLAN offloading to NIC > + > +Certain NIC's, only support VLAN offloading only for a reduced number of tags. > +In these cases it is best to either reduce the VLAN tag range on the bridge or > +disable offloading altogether. The exact problems that can arise depend on > +the exact hardware used. There are two other pitfalls I can think of that can occur when configuring VLANs directly on top of physical interfaces: When using a VLAN on the host (often the management VLAN) and starting a guest with a network device that has the same VLAN tag set, connectivity to the host is lost on that VLAN after starting the guest. e.g.: # NIC that carries the VLANs iface nic1 inet manual # management VLAN auto nic1.1337 iface nic1.1337 inet static address 10.133.7.10/24 # non vlan-aware bridge auto vmbr1 iface vmbr1 inet manual bridge-ports nic1 bridge-stp off bridge-fd 0 and then a guest: net0: name=eth0,bridge=vmbr1,hwaddr=BC:24:11:86:69:F3,tag=1337,type=veth ------------------- The other one is an ifupdown2 issue, when configuring the gateway on the physical NIC, when it also is a member of a VLAN-aware bridge: # NIC that carries all VLANs iface nic0 inet manual # management VLAN auto nic0.1337 iface nic0.1337 inet static address 192.0.2.10/24 gateway 192.0.2.1 # bridge for VM traffic auto vmbr0 iface vmbr0 inet manual bridge-ports nic0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 > 2025-12-19 14:05:57,411: MainThread: ifupdown: scheduler.py:114:run_iface_op(): error: nic0.1337: cmd '/bin/ip route replace default via 192.0.2.1 proto kernel dev nic0.1337 onlink' failed: returned 2 (Error: Nexthop device is not up.) see here for more details https://bugzilla.proxmox.com/show_bug.cgi?id=5406 https://github.com/CumulusNetworks/ifupdown2/issues/124 Both of those should not occur when following our guide above and the resolution is generally to use a VLAN-aware bridge instead. It might still be good to include it here as it can be easily referenced then? > Disabling IPv6 on the Node > ~~~~~~~~~~~~~~~~~~~~~~~~~~ > > diff --git a/pvesdn.adoc b/pvesdn.adoc > index 3fd3533..8e40ab4 100644 > --- a/pvesdn.adoc > +++ b/pvesdn.adoc > @@ -251,7 +251,7 @@ the network segments. This allows connectivity of VMs between different nodes. > VLAN zone configuration options: > > Bridge:: The local bridge or OVS switch, already configured on *each* node that > - allows node-to-node connection. > + allows node-to-node connection. This bridge should be VLAN aware to work properly. > > > [[pvesdn_zone_plugin_qinq]]