[PVE-User] The low network throughput obsered on Open vSwitch bridge.

[PVE-User] The low network throughput obsered on Open vSwitch bridge.

[Hongyi Zhao]

I install the pve on an I7 8886U machine with 6 full-duplex gigabit
network cards. The network configuration of the pve host is shown
below:

<quote>
root@pve:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr0

auto enp2s0
iface enp2s0 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr1

iface enp3s0 inet manual
iface enp4s0 inet manual
iface enp5s0 inet manual
iface enp6s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.10.254/24
        gateway 192.168.10.1
        ovs_type OVSBridge
        ovs_ports enp1s0

auto vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports enp2s0
</quote>

Now I use scp to transfer file from pve (192.168.10.254) to another
physical machine (192.168.10.100) or vice versa, but run the following
two commands in order:

root@pve:~# scp macOS-10.13.qcow2 werner@192.168.10.100:/dev/null
werner@192.168.10.100's password:
macOS-10.13.qcow2                              37% 6209MB  83.6MB/s   02:01 ETA

werner@X10DAi:~$ scp macOS-10.13.qcow2 root@192.168.10.254:/dev/null
root@192.168.10.254's password:
macOS-10.13.qcow2                                        30% 1408MB
53.5MB/s   01:00 ETA

As you can see, the show different network transfer speed. OTOH, if I
run the above two commands simultaneously, the results will look like
the following:

werner@X10DAi:~$ scp macOS-10.13.qcow2 root@192.168.10.254:/dev/null
root@192.168.10.254's password:
macOS-10.13.qcow2                                        17%  811MB
26.0MB/s   02:26 ETA

root@pve:~# scp macOS-10.13.qcow2 werner@192.168.10.100:/dev/null
werner@192.168.10.100's password:
macOS-10.13.qcow2                              13% 2205MB  51.7MB/s   04:34 ETA


Any hints for the above observations and results will be highly appreciated.

Regards
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

Re: [PVE-User] The low network throughput obsered on Open vSwitch bridge.

[Dmitry Petuhov]

Hello.

That's how TCP throughput mediation works. In basic case, you cannot 
canot fully saturate link by two opposite TCP connections regardless of 
using Open vSwitch.

There's one dirty hack that can override this behavior: you could use 
qdisc on interface that prioritize packets that have ACK bit set over 
those that don't in same connection. It will saturate link, but may lead 
to other issues on certain types of traffic (because of packet 
reordering), so [almost] nobody use it in real life.

Multiple parallel connections in same direction also saturate link more 
evenly.


15.03.2021 6:33, Hongyi Zhao пишет:
> I install the pve on an I7 8886U machine with 6 full-duplex gigabit
> network cards. The network configuration of the pve host is shown
> below:
>
> <quote>
> root@pve:~# cat /etc/network/interfaces
> # network interface settings; autogenerated
> # Please do NOT modify this file directly, unless you know what
> # you're doing.
> #
> # If you want to manage parts of the network configuration manually,
> # please utilize the 'source' or 'source-directory' directives to do
> # so.
> # PVE will preserve these directives, but will NOT read its network
> # configuration from sourced files, so do not attempt to move any of
> # the PVE managed interfaces into external files!
>
> auto lo
> iface lo inet loopback
>
> auto enp1s0
> iface enp1s0 inet manual
>          ovs_type OVSPort
>          ovs_bridge vmbr0
>
> auto enp2s0
> iface enp2s0 inet manual
>          ovs_type OVSPort
>          ovs_bridge vmbr1
>
> iface enp3s0 inet manual
> iface enp4s0 inet manual
> iface enp5s0 inet manual
> iface enp6s0 inet manual
>
> auto vmbr0
> iface vmbr0 inet static
>          address 192.168.10.254/24
>          gateway 192.168.10.1
>          ovs_type OVSBridge
>          ovs_ports enp1s0
>
> auto vmbr1
> iface vmbr1 inet manual
>          ovs_type OVSBridge
>          ovs_ports enp2s0
> </quote>
>
> Now I use scp to transfer file from pve (192.168.10.254) to another
> physical machine (192.168.10.100) or vice versa, but run the following
> two commands in order:
>
> root@pve:~# scp macOS-10.13.qcow2 werner@192.168.10.100:/dev/null
> werner@192.168.10.100's password:
> macOS-10.13.qcow2                              37% 6209MB  83.6MB/s   02:01 ETA
>
> werner@X10DAi:~$ scp macOS-10.13.qcow2 root@192.168.10.254:/dev/null
> root@192.168.10.254's password:
> macOS-10.13.qcow2                                        30% 1408MB
> 53.5MB/s   01:00 ETA
>
> As you can see, the show different network transfer speed. OTOH, if I
> run the above two commands simultaneously, the results will look like
> the following:
>
> werner@X10DAi:~$ scp macOS-10.13.qcow2 root@192.168.10.254:/dev/null
> root@192.168.10.254's password:
> macOS-10.13.qcow2                                        17%  811MB
> 26.0MB/s   02:26 ETA
>
> root@pve:~# scp macOS-10.13.qcow2 werner@192.168.10.100:/dev/null
> werner@192.168.10.100's password:
> macOS-10.13.qcow2                              13% 2205MB  51.7MB/s   04:34 ETA
>
>
> Any hints for the above observations and results will be highly appreciated.
>
> Regards

Re: [PVE-User] The low network throughput obsered on Open vSwitch bridge.

[Hongyi Zhao]

On Mon, Mar 15, 2021 at 12:00 PM Dmitry Petuhov <mityapetuhov@gmail.com> wrote:
>
> Hello.
>
> That's how TCP throughput mediation works. In basic case, you cannot
> canot fully saturate link by two opposite TCP connections regardless of
> using Open vSwitch.
>
> There's one dirty hack that can override this behavior: you could use
> qdisc on interface that prioritize packets that have ACK bit set over
> those that don't in same connection. It will saturate link, but may lead
> to other issues on certain types of traffic (because of packet
> reordering), so [almost] nobody use it in real life.
>
> Multiple parallel connections in same direction also saturate link more
> evenly.

Thanks a lot for your help and explanation. But I still can't
understand why the speed observed from pve (192.168.10.254) to another
physical machine (192.168.10.100) is always higher than that observed
in the reverse direction. Any hints for this phenomenon?

Regards
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China