Re: [PVE-User] replication bandwith limit not respected

Re: [PVE-User] replication bandwith limit not respected

[Stefan Nehlsen]

Am 06.09.24 um 15:00 schrieb Tonči Stipičević:
> Hello,
> 
> I'm running latest cluster (community support)

A little more information about your cluster might help.

> And today I've lowered replication speed down to 80MB/s  but replcation 
> still uses whole NIC bandwith (1G)  ... after few host restarts  still 
> the same
> 
> 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second 
> for transferring 'data2:subvol-1007-disk-0'
> 
> Does somebody else experience that too ?

Nowadays everything will be able to saturize 1Gbps links.

Better you add faster hardware to your cluster.

Cheapest solution will be connecting 2 nodes directly with 2 USB3 
ethernet devices and a cable. 2.5 or 5 Gbps just for cluster 
communication or/and replication will make a difference.

10Gbps cards and switches wound be even better.

Dual 40 Gbps connections probably may be overkill :-)

It all depends on what you are doing.


Stefan

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] replication bandwith limit not respected

[Stefan Radman via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 7553 bytes --]

From: Stefan Radman <stefan.radman@me.com>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Cc: PVE User List <pve-user@pve.proxmox.com>
Subject: Re: [PVE-User] replication bandwith limit not respected
Date: Fri, 6 Sep 2024 20:18:20 +0300
Message-ID: <C89D18A7-57D5-4AF0-926D-A651237997AD@me.com>

Hi Tonči

I am not familiar with PVE storage replication but with networks.
Your answers to the questions below might help everyone on this list (including me) to better understand the issues behind the problem you are facing.

> lowered replication speed down to 80MB/s

Have you tried much lower rates (e.g. 10MB/s)?
Did anything change in the bandwidth utilization?

> replcation still uses whole NIC bandwith (1G)


How do you know? 
How do know that storage replication cloggs your network and not some other apllication?
Where do you measure the replication traffic and how?

Are you running Jumbo frames (e.g. MTU 9000) between the replication targets or standard MTU 1500?

If you are not using a dedicated NIC for storage replication you are likely (and should be) running on standard MTU. 
Using the standard MTU will generate more packets than jumbo frames, thus more headers, thus more overhead.
A bandwidth limit of 80MB/s translates to rougly 640Mbit/s but I assume that this does not include the overhead of protocols used by PVE storage replication (Ethernet, TCP/IP, SSH).

How much overhead in % do you expect?

My conservative assumption (rule of thumb) for SSH with standard MTU is 20% overhead. 
That would be 768Mbit/s, leaving only 232Mbit/s for other traffic.

What is your output of “pveversion”?

Regards

Stefan

> On Sep 6, 2024, at 16:00, Tonči Stipičević <tonci@suma-informatika.hr> wrote:
> 
> Hello,
> 
> I'm running latest cluster (community support)
> 
> And today I've lowered replication speed down to 80MB/s  but replcation still uses whole NIC bandwith (1G)  ... after few host restarts  still the same
> 
> 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second for transferring 'data2:subvol-1007-disk-0'
> 
> Does somebody else experience that too ?
> 
> 
> Thank you in advance
> 
> BR
> 
> Tonči
> 
> -- 
> srdačan pozdrav / best regards
> 
> Tonči Stipičević, dipl. ing. elektr.
> direktor / manager
> 
> SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263
> 
> Podrška / Upravljanje IT sustavima za male i srednje tvrtke
> Small & Medium Business IT Support / Management
> 
> mob: 091 1234003
> www.suma-informatika.hr
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user


[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] replication bandwith limit not respected

[Stefan Radman via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 7563 bytes --]

From: Stefan Radman <stefan.radman@me.com>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Cc: PVE User List <pve-user@pve.proxmox.com>
Subject: Re: [PVE-User] replication bandwith limit not respected
Date: Fri, 6 Sep 2024 20:18:20 +0300
Message-ID: <C89D18A7-57D5-4AF0-926D-A651237997AD@me.com>

Hi Tonči

I am not familiar with PVE storage replication but with networks.
Your answers to the questions below might help everyone on this list (including me) to better understand the issues behind the problem you are facing.

> lowered replication speed down to 80MB/s

Have you tried much lower rates (e.g. 10MB/s)?
Did anything change in the bandwidth utilization?

> replcation still uses whole NIC bandwith (1G)


How do you know? 
How do know that storage replication cloggs your network and not some other apllication?
Where do you measure the replication traffic and how?

Are you running Jumbo frames (e.g. MTU 9000) between the replication targets or standard MTU 1500?

If you are not using a dedicated NIC for storage replication you are likely (and should be) running on standard MTU. 
Using the standard MTU will generate more packets than jumbo frames, thus more headers, thus more overhead.
A bandwidth limit of 80MB/s translates to rougly 640Mbit/s but I assume that this does not include the overhead of protocols used by PVE storage replication (Ethernet, TCP/IP, SSH).

How much overhead in % do you expect?

My conservative assumption (rule of thumb) for SSH with standard MTU is 20% overhead. 
That would be 768Mbit/s, leaving only 232Mbit/s for other traffic.

What is your output of “pveversion”?

Regards

Stefan

> On Sep 6, 2024, at 16:00, Tonči Stipičević <tonci@suma-informatika.hr> wrote:
> 
> Hello,
> 
> I'm running latest cluster (community support)
> 
> And today I've lowered replication speed down to 80MB/s  but replcation still uses whole NIC bandwith (1G)  ... after few host restarts  still the same
> 
> 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second for transferring 'data2:subvol-1007-disk-0'
> 
> Does somebody else experience that too ?
> 
> 
> Thank you in advance
> 
> BR
> 
> Tonči
> 
> -- 
> srdačan pozdrav / best regards
> 
> Tonči Stipičević, dipl. ing. elektr.
> direktor / manager
> 
> SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263
> 
> Podrška / Upravljanje IT sustavima za male i srednje tvrtke
> Small & Medium Business IT Support / Management
> 
> mob: 091 1234003
> www.suma-informatika.hr
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user


[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] replication bandwith limit not respected

[Stefan Radman via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 8347 bytes --]

From: Stefan Radman <stefan.radman@me.com>
To: "Tonči Stipičević" <tonci@suma-informatika.hr>
Cc: Proxmox VE user list <pve-user@lists.proxmox.com>
Subject: Re: [PVE-User] replication bandwith limit not respected
Date: Sun, 15 Sep 2024 23:37:51 +0300
Message-ID: <CCF3D6FC-EDA7-40B3-9278-F56CEA62B657@me.com>

Hi Tonči

> Now, two hosts are directly connected with optical patch cable (peer-to-peer) .. 10G

Then you should configure jumbo frames (e.g. MTU 9000) on those two interfaces.

Stefan

> On Sep 9, 2024, at 10:40, Tonči Stipičević <tonci@suma-informatika.hr> wrote:
> 
> Hello,
> 
> this was just FYI ..  I was just about to upgrade NICs to 10G , but before I had to "rearrange" my VMs around my cluster and that is why I lowered BW down to 80MBs
> 
> No, I  did not measured/tested the speed, haven't gone that deep, and this info was from network GUI graph ... it was saturated at 1Gbs
> 
> Anyway , I remember that custom replication bandwith adjustment was always followed by corresponding GUI network graph presentation ... but like you said , maybe this lowering was to small to reflect to the network GUI graph
> 
> Pretty soon I'll test it again (with 10G cards)  and write back the results      Now, two hosts are directly connected with optical patch cable (peer-to-peer) .. 10G
> 
>    ...srdačan pozdrav / best regards
> 
> Tonči Stipičević, dipl. ing. elektr.
> direktor / manager
> 
> SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263
> 
> Podrška / Upravljanje IT sustavima za male i srednje tvrtke
> Small & Medium Business IT Support / Management
> 
> mob: 091 1234003
> www.suma-informatika.hr
> 
> 06. 09. 2024. u 19:29, pve-user-request@lists.proxmox.com piše:
>> Send pve-user mailing list submissions to
>> 	pve-user@lists.proxmox.com
>> 
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> or, via email, send a message with subject or body 'help' to
>> 	pve-user-request@lists.proxmox.com
>> 
>> You can reach the person managing the list at
>> 	pve-user-owner@lists.proxmox.com
>> 
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of pve-user digest..."
>> 
>> 
>> Today's Topics:
>> 
>>    1. replication bandwith limit not respected (Ton?i Stipi?evi?)
>>    2. Re: replication bandwith limit not respected (Stefan Nehlsen)
>>    3. Re: replication bandwith limit not respected (Stefan Radman)
>>    4. Re: replication bandwith limit not respected (Stefan Radman)
>>    5. Re: 8.2-1 ISO doesn't boot via UEFI on usb (Bryan Fields)
>> 
>> 
>> ----------------------------------------------------------------------
>> 
>> Message: 1
>> Date: Fri, 6 Sep 2024 15:00:57 +0200
>> From: Ton?i Stipi?evi? <tonci@suma-informatika.hr>
>> To: PVE User List <pve-user@pve.proxmox.com>
>> Subject: [PVE-User] replication bandwith limit not respected
>> Message-ID: <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>> 
>> Hello,
>> 
>> I'm running latest cluster (community support)
>> 
>> And today I've lowered replication speed down to 80MB/s? but replcation
>> still uses whole NIC bandwith (1G)? ... after few host restarts? still
>> the same
>> 
>> 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second
>> for transferring 'data2:subvol-1007-disk-0'
>> 
>> Does somebody else experience that too ?
>> 
>> 
>> Thank you in advance
>> 
>> BR
>> 
>> Ton?i
>> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user



[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user