Re: [PATCH pve-docs v2 1/1] network/sdn: update and clean up VLAN information

[Stefan Hanreich <s.hanreich@proxmox.com>]

On 10/20/26 8:45 AM, Thomas Ellmenreich wrote:
> Adjusted some of the details related to VLAN setups, both with the traditional
> methods as well as the SDN setup.
> 
> - Removed an outdated example and added more information to the remaining
>   examples. First, by adding the VLAN interface on the bridge, and then
>   showcasing that VLAN tags can be set in the guest network configurations.
> 
> - Added SDN as a possible way to configure VLANs for guests, as well as adding
>   an example for SDN.
> 
> - Added a "Common Issues" section to collect common networking issues.
> 
> - Added a "Troubleshooting" section, to collect information on how to fix VLAN
>   related issues.
> 
> Suggested-by: Stefan Hanreich <s.hanreich@proxmox.com>
> Signed-off-by: Thomas Ellmenreich <t.ellmenreich@proxmox.com>
> ---
> 
> Some changes for the VLAN configuration documentation, would apprechiate any
> help in regards to the wording and details of said documentation.
> 
> Changelog:
> 
> - PATCH v1 -> v2
>     * Added a "Common Issues" section, that recounts different problems that
>       might appear.
> 
>     * Added a "Troubleshooting" section to show different ways to troubleshoot
>       problems with the networking.
> 
>     * Added a SDN VLAN example.
> 
>     * Some rewording, and adjustment of different sections.
> 
>  pve-network.adoc | 161 +++++++++++++++++++++++++++++++++++------------
>  pvesdn.adoc      |   2 +-
>  2 files changed, 121 insertions(+), 42 deletions(-)
> 
> diff --git a/pve-network.adoc b/pve-network.adoc
> index 2b3e664..b1a598c 100644
> --- a/pve-network.adoc
> +++ b/pve-network.adoc
> @@ -637,6 +637,53 @@ completely done inside the guest and can not be influenced from the
>  outside. The benefit is that you can use more than one VLAN on a
>  single virtual NIC.
>  
> +* *SDN configured VLAN:*
> +It is also possible to xref:pvesdn_config_vnet[configure VLANs in SDN] by
> +creating a VLAN zone on the bridge and then creating VNets for every VLAN tag.
> +In this case, it is also recommended to use a VLAN-aware bridge.
> +
> +.Example: SDN VLAN Zones with VLAN aware bridges
> +
> +This is the preferred solution if users have a small amount of VLANs inside
> +their cluster. On every node, users should configure a VLAN-aware bridge with
> +the respective carrier for their VLANs:
> +
> +----
> +auto vmbr0
> +iface vmbr0 inet manual
> +        bridge-ports nic0
> +        bridge-stp off
> +        bridge-fd 0
> +        bridge-vlan-aware yes
> +        bridge-vids 2-4094
> +----
> +
> +Then create an SDN zone on top of that bridge:
> +
> +----
> +vlan: vlanz
> +        bridge vmbr0
> +        ipam pve
> +----
> +
> +Then, for each VLAN they want to use, create a VNet:
> +
> +----
> +vnet: vnet1234
> +        zone vlanz
> +        tag 1234
> +----
> +
> +After applying the SDN configuration, users can then select vnet1234 in the
> +network device configuration of their guests. This method does not allow to
> +configure an IP for the host directly on a VNet - but it can be done manually
> +by adding an entry for the VNet to /etc/network/interfaces:
> +
> +----
> +auto vnet1234
> +iface vnet1234 inet static
> +        address 192.0.2.1/24
> +----
>  
>  VLAN on the Host
>  ^^^^^^^^^^^^^^^^
> @@ -644,63 +691,47 @@ VLAN on the Host
>  To allow host communication with an isolated network. It is possible
>  to apply VLAN tags to any network device (NIC, Bond, Bridge). In
>  general, you should configure the VLAN on the interface with the least
> -abstraction layers between itself and the physical NIC.
> +abstraction layers between itself and the physical NIC. But it would still
> +be best to configure the VLAN on a bridge instead of a bond or the NIC
> +directly to avoid some of the issues mentioned xref:vlan_common_issues[here].
>  
>  For example, in a default configuration where you want to place
>  the host management address on a separate VLAN.
>  
> -
> -.Example: Use VLAN 5 for the {pve} management IP with traditional Linux bridge
> +.Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge
>  ----
>  auto lo
>  iface lo inet loopback
>  
>  iface eno1 inet manual
>  
> -iface eno1.5 inet manual
> -
> -auto vmbr0v5
> -iface vmbr0v5 inet static
> -        address  10.10.10.2/24
> -        gateway  10.10.10.1
> -        bridge-ports eno1.5
> -        bridge-stp off
> -        bridge-fd 0
> -
>  auto vmbr0
>  iface vmbr0 inet manual
>          bridge-ports eno1
>          bridge-stp off
>          bridge-fd 0
> -
> -----
> -
> -.Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge
> -----
> -auto lo
> -iface lo inet loopback
> -
> -iface eno1 inet manual
> -
> +        bridge-vlan-aware yes
> +        bridge-vids 2-4094
>  
>  auto vmbr0.5
>  iface vmbr0.5 inet static
>          address  10.10.10.2/24
>          gateway  10.10.10.1
> +----
> +
> +Guests can then reference the VLAN tag directly in their network
> +configuration. Compared to an SDN-configured solution, this has the
> +disadvantage that specific VLANs cannot be named, but the advantage that
> +each VLAN does not have to be configured individually.
>  
> -auto vmbr0
> -iface vmbr0 inet manual
> -        bridge-ports eno1
> -        bridge-stp off
> -        bridge-fd 0
> -        bridge-vlan-aware yes
> -        bridge-vids 2-4094
> +----
> +net0: name=eth0,bridge=vmbr0,tag=5,...
>  ----
>  
>  The next example is the same setup but a bond is used to
>  make this network fail-safe.
>  
> -.Example: Use VLAN 5 with bond0 for the {pve} management IP with traditional Linux bridge
> +.Example: Use VLAN 5 with bond0 for the {pve} management IP with VLAN aware Linux bridge
>  ----
>  auto lo
>  iface lo inet loopback
> @@ -716,24 +747,72 @@ iface bond0 inet manual
>        bond-mode 802.3ad
>        bond-xmit-hash-policy layer2+3
>  
> -iface bond0.5 inet manual
> -
> -auto vmbr0v5
> -iface vmbr0v5 inet static
> -        address  10.10.10.2/24
> -        gateway  10.10.10.1
> -        bridge-ports bond0.5
> -        bridge-stp off
> -        bridge-fd 0
> -
>  auto vmbr0
>  iface vmbr0 inet manual
>          bridge-ports bond0
> +        bridge-vlan-aware yes
> +        bridge-vids 2-4094
>          bridge-stp off
>          bridge-fd 0
>  
> +auto vmbr0.5
> +iface vmbr0.5 inet static
> +        address 10.10.10.2/24
> +        gateway 10.10.10.1
> +
>  ----
>  
> +Troubleshooting
> +^^^^^^^^^^^^^^^
> +
> +There are some helpful commands that can be utilised when debugging such issues:
> +
> +----
> +$ bridge -compressvlans vlan show
> +----
> +
> +This prints a list of bridges and their ports, as well as the VLANs that are
> +currently configured for them:
> +
> +----
> +port              vlan-id
> +nic0              1 PVID Egress Untagged
> +                  2-4094
> +vmbr0             1 PVID Egress Untagged
> +veth100i0         1234 PVID Egress Untagged
> +----
> +
> +----
> +$ bridge fdb show
> +----
> +
> +This prints a list of the current forwarding database entries of all bridges.
> +The bridge decides how to forward a given packet based on this table. An entry
> +of the fdb generally looks like this:
> +
> +----
> +aa:aa:aa:d8:36:7c dev nic0 vlan 49 master vmbr0 permanent
> +----
> +
> +This means that packets arriving on vmbr0, that have VLAN Tag 49 set and
> +destination MAC address aa:aa:aa:d8:36:7c will get forwarded to the port nic0.
> +
> +[[vlan_common_issues]]
> +Common Issues
> +^^^^^^^^^^^^^
> +
> +.Issue: Configuring VLAN on interface that is member of bridge
> +
> +It's not recommended to configure a VLAN on an interface that is a member of a
> +bridge. Doing so will "blackhole" all traffic of that VLAN on the bridge.

Might need a more specific example so this becomes clearer? It's hard to understand
otherwise imo.

> +
> +.Issue: Limited number of tags when VLAN offloading to NIC
> +
> +Certain NIC's, only support VLAN offloading only for a reduced number of tags.
> +In these cases it is best to either reduce the VLAN tag range on the bridge or
> +disable offloading altogether. The exact problems that can arise depend on
> +the exact hardware used.

There are two other pitfalls I can think of that can occur when configuring
VLANs directly on top of physical interfaces:


When using a VLAN on the host (often the management VLAN) and starting a guest
with a network device that has the same VLAN tag set, connectivity to the host is
lost on that VLAN after starting the guest.

e.g.:

 # NIC that carries the VLANs
 iface nic1 inet manual
 
 # management VLAN
 auto nic1.1337
 iface nic1.1337 inet static
         address 10.133.7.10/24
 
 # non vlan-aware bridge
 auto vmbr1
 iface vmbr1 inet manual
         bridge-ports nic1
         bridge-stp off
         bridge-fd 0

and then a guest:

 net0: name=eth0,bridge=vmbr1,hwaddr=BC:24:11:86:69:F3,tag=1337,type=veth

-------------------

The other one is an ifupdown2 issue, when configuring the gateway on the
physical NIC, when it also is a member of a VLAN-aware bridge:

 # NIC that carries all VLANs
 iface nic0 inet manual
 
 # management VLAN
 auto nic0.1337
 iface nic0.1337 inet static
         address 192.0.2.10/24
         gateway 192.0.2.1
 
 # bridge for VM traffic
 auto vmbr0
 iface vmbr0 inet manual
         bridge-ports nic0
         bridge-stp off
         bridge-fd 0
         bridge-vlan-aware yes
         bridge-vids 2-4094

>  2025-12-19 14:05:57,411: MainThread: ifupdown: scheduler.py:114:run_iface_op(): error: nic0.1337: cmd '/bin/ip route replace default via 192.0.2.1 proto kernel dev nic0.1337 onlink' failed: returned 2 (Error: Nexthop device is not up.)

see here for more details
https://bugzilla.proxmox.com/show_bug.cgi?id=5406
https://github.com/CumulusNetworks/ifupdown2/issues/124



Both of those should not occur when following our guide above and the resolution
is generally to use a VLAN-aware bridge instead. It might still be good to
include it here as it can be easily referenced then?

>  Disabling IPv6 on the Node
>  ~~~~~~~~~~~~~~~~~~~~~~~~~~
>  
> diff --git a/pvesdn.adoc b/pvesdn.adoc
> index 3fd3533..8e40ab4 100644
> --- a/pvesdn.adoc
> +++ b/pvesdn.adoc
> @@ -251,7 +251,7 @@ the network segments.  This allows connectivity of VMs between different nodes.
>  VLAN zone configuration options:
>  
>  Bridge:: The local bridge or OVS switch, already configured on *each* node that
> -  allows node-to-node connection.
> +  allows node-to-node connection. This bridge should be VLAN aware to work properly.
>  
>  
>  [[pvesdn_zone_plugin_qinq]]