[PVE-User] VMs same Network Bridge Massive Difference in Speed

[PVE-User] VMs same Network Bridge Massive Difference in Speed

[John Thomas]

Hi (from a hobbiest),

I have several VMs (some Debian some CentOS).  The VM network speed is
either ~20.0 Gbits/sec or ~2.5 Gbits/sec.  I can't figure out why.  I'd
like them all to do ~20 if possible.  All VMs are connected to the same
bridge.  Any suggestions for helping me debug (I'm not an expert)?

*host info:*
*lspci | grep -i eth*
67:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network
Connection (rev 01)
67:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network
Connection (rev 01)
67:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network
Connection (rev 01)
67:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network
Connection (rev 01)
b7:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for
10GBASE-T (rev 04)
b7:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for
10GBASE-T (rev 04)
b7:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for
10GbE SFP+ (rev 04)
b7:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for
10GbE SFP+ (rev 04)
*cat /etc/network/interfaces*
source /etc/network/interfaces.d/*
auto lo
iface lo inet loopback
iface eno1 inet manual
#1G I350 67:00.0
iface eno2 inet manual
#1G I350 67:00.1
iface eno3 inet manual
#1G I350 67:00.2
iface eno4 inet manual
#1G I350 67:00.3
iface eno7 inet manual
#10G
iface eno8 inet manual
#10G
auto vmbr0
iface vmbr0 inet static
        address 192.168.2.26/24
        gateway 192.168.2.1
        bridge-ports eno1 eno2 eno3 eno4 eno7 eno8
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-20
        offload-rx-vlan-filter off

*example ~20 speed Vmachine:*
*info from host:*
*cat /etc/pve/nodes/pve/qemu-server/102.conf*
agent: 1
balloon: 0
boot: c
bootdisk: scsi0
cores: 4
cpu: kvm64
memory: 2048
name: utils02
net0: virtio=C2:A1:D1:49:88:CE,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0:
local-lvm:vm-102-disk-1,cache=writeback,discard=on,format=raw,iothread=1,size=32G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=f209262f-1a25-4f56-a07a-ba8e720292ee
sockets: 1
tablet: 0
vga: qxl
vmgenid: ebf779b6-502d-4962-9d8f-0f2158853af8

*info from ~20 VM:*
(utils) [root@utils02 ~]# *cat /etc/centos-release*
CentOS Linux release 7.9.2009 (Core)
(utils) [root@utils02 ~]# *ethtool -i eth0*
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:00:12.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
(utils) [root@utils02 ~]# *iperf3 -c 192.168.2.26*
Connecting to host 192.168.2.26, port 5201
[  4] local 192.168.2.12 port 54866 connected to 192.168.2.26 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  2.50 GBytes  21.5 Gbits/sec    0   1.20 MBytes

[  4]   1.00-2.00   sec  2.49 GBytes  21.4 Gbits/sec    0   1.20 MBytes

[  4]   2.00-3.00   sec  2.49 GBytes  21.3 Gbits/sec    0   1.20 MBytes

[  4]   3.00-4.00   sec  2.48 GBytes  21.3 Gbits/sec    0   1.20 MBytes

[  4]   4.00-5.00   sec  2.44 GBytes  21.0 Gbits/sec    0   1.31 MBytes

[  4]   5.00-6.00   sec  2.49 GBytes  21.4 Gbits/sec    0   1.72 MBytes

[  4]   6.00-7.00   sec  2.48 GBytes  21.3 Gbits/sec    0   1.72 MBytes

[  4]   7.00-8.00   sec  2.47 GBytes  21.2 Gbits/sec    0   1.72 MBytes

[  4]   8.00-9.00   sec  2.50 GBytes  21.5 Gbits/sec    0   1.72 MBytes

[  4]   9.00-10.00  sec  2.47 GBytes  21.2 Gbits/sec    0   1.88 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  24.8 GBytes  21.3 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  24.8 GBytes  21.3 Gbits/sec
 receiver

*example ~2.5 speed Vmachine:*
*info from host:*
*cat /etc/pve/nodes/pve05/qemu-server/110.conf*
agent: 1
balloon: 0
boot: c
bootdisk: scsi0
cores: 4
cpu: kvm64
memory: 1024
name: smo
net0: virtio=5A:9B:FB:E3:0D:55,bridge=vmbr0,firewall=1,tag=4
numa: 0
onboot: 1
ostype: l26
scsi0:
local-lvm:vm-110-disk-0,cache=writeback,discard=on,format=raw,iothread=1,size=32G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=c156ed77-499e-4a42-8160-2e2e616a5ec4
sockets: 1
tablet: 0
vga: qxl
vmgenid: 578c9839-ccb3-4b48-acb2-1865f5583037

*info from ~2.5 VM:*
(utils) [root@smo ~]# *cat /etc/centos-release*
CentOS Linux release 7.9.2009 (Core)
(utils) [root@smo ~]# *ethtool -i eth0*
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:00:12.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
(utils) [root@smo ~]# *iperf3 -c 192.168.2.26*
Connecting to host 192.168.2.26, port 5201
[  4] local 192.168.71.10 port 48524 connected to 192.168.2.26 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   348 MBytes  2.92 Gbits/sec   19   1.92 MBytes

[  4]   1.00-2.00   sec   342 MBytes  2.87 Gbits/sec    0   2.09 MBytes

[  4]   2.00-3.00   sec   342 MBytes  2.87 Gbits/sec    0   2.22 MBytes

[  4]   3.00-4.00   sec   321 MBytes  2.69 Gbits/sec   92   1.64 MBytes

[  4]   4.00-5.00   sec   332 MBytes  2.79 Gbits/sec    0   1.78 MBytes

[  4]   5.00-6.00   sec   340 MBytes  2.85 Gbits/sec    0   1.92 MBytes

[  4]   6.00-7.00   sec   354 MBytes  2.97 Gbits/sec    0   2.05 MBytes

[  4]   7.00-8.00   sec   336 MBytes  2.82 Gbits/sec  383   1.54 MBytes

[  4]   8.00-9.00   sec   352 MBytes  2.96 Gbits/sec    0   1.69 MBytes

[  4]   9.00-10.00  sec   341 MBytes  2.86 Gbits/sec    0   1.84 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  3.33 GBytes  2.86 Gbits/sec  494             sender
[  4]   0.00-10.00  sec  3.33 GBytes  2.86 Gbits/sec
 receiver


-- 
Kind regards,
JT