[PATCH common/network 0/3] Performance improvements for applying VLAN / QinQ zones

[Stefan Hanreich <s.hanreich@proxmox.com>]

## Problem

When creating a VLAN / QinQ zone on a non VLAN-aware bridge, the SDN config
generation enters a code path that calls ip_link_details from PVE::IPRoute2
repeatedly. This causes a severe slowdown when applying configurations with a
large amount of VNets. What exacerbated this problem was that, aside from
calling iproute2 twice, every vnet recalculated the bridge ports for its bridge
by iterating over all existing interfaces on the host which leads to a quadratic
increase in runtime once the SDN configuration had been applied once.

## Solution

This patch series introduces a new helper to pve-common that pre-computes the
bridge port mappings used in the SDN config generation. The SDN stack then calls
that function once at the beginning of the 'Apply' step, then re-uses the result
for each vnet. This avoids calling iproute2 repeatedly and the nested loop
inside the generate_sdn_config function in the VLAN / QinQ plugins.

An initial approach only cached the ip_link_details output and reused that, but
this approach did not avoid the nested loops, so I opted for pre-calculating the
required information instead and passing only that to the zone plugins. The
first patch is a remnant of the initial approach, which was still included
because it is useful for the other callsites.

## Benchmarks

This slashes the runtime for generating the SDN configuration considerably on a
test setup with 1000 VLAN VNets present on the host / config:

Without these patches applied:

  root@pve-vlan-test:~# time perl -e 'use PVE::Network::SDN; PVE::Network::SDN::generate_raw_etc_network_config(PVE::Network::SDN::compile_running_cfg(1))'
  real    5m59.853s
  user    5m49.009s
  sys     0m34.887s

With these patches applied:

  root@pve-vlan-test:~# time perl -e 'use PVE::Network::SDN; PVE::Network::SDN::generate_raw_etc_network_config(PVE::Network::SDN::compile_running_cfg(1))'
  real    0m0.868s
  user    0m0.818s
  sys     0m0.109s

## Future Work

This patch series does not address any bottlenecks in ifupdown2, which is now
the limiting factor when applying large configurations. Applying a SDN
configuration via ifreload with 1000 VLANs takes 22 seconds. I'll be exploring
potential improvements there with Christoph, but currently we do not have high
hopes of improving the runtime there considerably.

Currently the SDN configuration is applied sequentially on each node, I'll be
looking into creating a patch series that applies the network configuration in
parallel.


pve-common:

Stefan Hanreich (2):
  iproute2: restructure get_physical_bridge_ports
  iproute2: add helper for getting bridge port mappings

 src/PVE/IPRoute2.pm | 30 +++++++++++++++++++++++++++---
 1 file changed, 27 insertions(+), 3 deletions(-)


pve-network:

Stefan Hanreich (1):
  fix #7738: zones: use pre-computed bridge port list

 src/PVE/Network/SDN/Zones.pm            |  3 +++
 src/PVE/Network/SDN/Zones/Plugin.pm     |  7 +------
 src/PVE/Network/SDN/Zones/QinQPlugin.pm |  5 +++--
 src/PVE/Network/SDN/Zones/VlanPlugin.pm |  5 +++--
 src/test/run_test_zones.pl              | 13 ++++++++++---
 5 files changed, 20 insertions(+), 13 deletions(-)


Summary over all repositories:
  6 files changed, 47 insertions(+), 16 deletions(-)

-- 
Generated by murpp 0.12.0