* Re: [PVE-User] pve-user Digest, Vol 198, Issue 6 [not found] <mailman.95.1725643774.414.pve-user@lists.proxmox.com> @ 2024-09-09 7:40 ` Tonči Stipičević 2024-09-15 20:37 ` [PVE-User] replication bandwith limit not respected Stefan Radman via pve-user 0 siblings, 1 reply; 5+ messages in thread From: Tonči Stipičević @ 2024-09-09 7:40 UTC (permalink / raw) To: pve-user Hello, this was just FYI .. I was just about to upgrade NICs to 10G , but before I had to "rearrange" my VMs around my cluster and that is why I lowered BW down to 80MBs No, I did not measured/tested the speed, haven't gone that deep, and this info was from network GUI graph ... it was saturated at 1Gbs Anyway , I remember that custom replication bandwith adjustment was always followed by corresponding GUI network graph presentation ... but like you said , maybe this lowering was to small to reflect to the network GUI graph Pretty soon I'll test it again (with 10G cards) and write back the results Now, two hosts are directly connected with optical patch cable (peer-to-peer) .. 10G ...srdačan pozdrav / best regards Tonči Stipičević, dipl. ing. elektr. direktor / manager SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263 Podrška / Upravljanje IT sustavima za male i srednje tvrtke Small & Medium Business IT Support / Management mob: 091 1234003 www.suma-informatika.hr 06. 09. 2024. u 19:29, pve-user-request@lists.proxmox.com piše: > Send pve-user mailing list submissions to > pve-user@lists.proxmox.com > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > or, via email, send a message with subject or body 'help' to > pve-user-request@lists.proxmox.com > > You can reach the person managing the list at > pve-user-owner@lists.proxmox.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of pve-user digest..." > > > Today's Topics: > > 1. replication bandwith limit not respected (Ton?i Stipi?evi?) > 2. Re: replication bandwith limit not respected (Stefan Nehlsen) > 3. Re: replication bandwith limit not respected (Stefan Radman) > 4. Re: replication bandwith limit not respected (Stefan Radman) > 5. Re: 8.2-1 ISO doesn't boot via UEFI on usb (Bryan Fields) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 6 Sep 2024 15:00:57 +0200 > From: Ton?i Stipi?evi? <tonci@suma-informatika.hr> > To: PVE User List <pve-user@pve.proxmox.com> > Subject: [PVE-User] replication bandwith limit not respected > Message-ID: <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr> > Content-Type: text/plain; charset=UTF-8; format=flowed > > Hello, > > I'm running latest cluster (community support) > > And today I've lowered replication speed down to 80MB/s? but replcation > still uses whole NIC bandwith (1G)? ... after few host restarts? still > the same > > 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second > for transferring 'data2:subvol-1007-disk-0' > > Does somebody else experience that too ? > > > Thank you in advance > > BR > > Ton?i > _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PVE-User] replication bandwith limit not respected 2024-09-09 7:40 ` [PVE-User] pve-user Digest, Vol 198, Issue 6 Tonči Stipičević @ 2024-09-15 20:37 ` Stefan Radman via pve-user 0 siblings, 0 replies; 5+ messages in thread From: Stefan Radman via pve-user @ 2024-09-15 20:37 UTC (permalink / raw) To: Tonči Stipičević; +Cc: Stefan Radman, Proxmox VE user list [-- Attachment #1: Type: message/rfc822, Size: 8347 bytes --] From: Stefan Radman <stefan.radman@me.com> To: "Tonči Stipičević" <tonci@suma-informatika.hr> Cc: Proxmox VE user list <pve-user@lists.proxmox.com> Subject: Re: [PVE-User] replication bandwith limit not respected Date: Sun, 15 Sep 2024 23:37:51 +0300 Message-ID: <CCF3D6FC-EDA7-40B3-9278-F56CEA62B657@me.com> Hi Tonči > Now, two hosts are directly connected with optical patch cable (peer-to-peer) .. 10G Then you should configure jumbo frames (e.g. MTU 9000) on those two interfaces. Stefan > On Sep 9, 2024, at 10:40, Tonči Stipičević <tonci@suma-informatika.hr> wrote: > > Hello, > > this was just FYI .. I was just about to upgrade NICs to 10G , but before I had to "rearrange" my VMs around my cluster and that is why I lowered BW down to 80MBs > > No, I did not measured/tested the speed, haven't gone that deep, and this info was from network GUI graph ... it was saturated at 1Gbs > > Anyway , I remember that custom replication bandwith adjustment was always followed by corresponding GUI network graph presentation ... but like you said , maybe this lowering was to small to reflect to the network GUI graph > > Pretty soon I'll test it again (with 10G cards) and write back the results Now, two hosts are directly connected with optical patch cable (peer-to-peer) .. 10G > > ...srdačan pozdrav / best regards > > Tonči Stipičević, dipl. ing. elektr. > direktor / manager > > SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263 > > Podrška / Upravljanje IT sustavima za male i srednje tvrtke > Small & Medium Business IT Support / Management > > mob: 091 1234003 > www.suma-informatika.hr > > 06. 09. 2024. u 19:29, pve-user-request@lists.proxmox.com piše: >> Send pve-user mailing list submissions to >> pve-user@lists.proxmox.com >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> or, via email, send a message with subject or body 'help' to >> pve-user-request@lists.proxmox.com >> >> You can reach the person managing the list at >> pve-user-owner@lists.proxmox.com >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of pve-user digest..." >> >> >> Today's Topics: >> >> 1. replication bandwith limit not respected (Ton?i Stipi?evi?) >> 2. Re: replication bandwith limit not respected (Stefan Nehlsen) >> 3. Re: replication bandwith limit not respected (Stefan Radman) >> 4. Re: replication bandwith limit not respected (Stefan Radman) >> 5. Re: 8.2-1 ISO doesn't boot via UEFI on usb (Bryan Fields) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Fri, 6 Sep 2024 15:00:57 +0200 >> From: Ton?i Stipi?evi? <tonci@suma-informatika.hr> >> To: PVE User List <pve-user@pve.proxmox.com> >> Subject: [PVE-User] replication bandwith limit not respected >> Message-ID: <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr> >> Content-Type: text/plain; charset=UTF-8; format=flowed >> >> Hello, >> >> I'm running latest cluster (community support) >> >> And today I've lowered replication speed down to 80MB/s? but replcation >> still uses whole NIC bandwith (1G)? ... after few host restarts? still >> the same >> >> 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second >> for transferring 'data2:subvol-1007-disk-0' >> >> Does somebody else experience that too ? >> >> >> Thank you in advance >> >> BR >> >> Ton?i >> > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user [-- Attachment #2: Type: text/plain, Size: 157 bytes --] _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 5+ messages in thread
[parent not found: <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr>]
* Re: [PVE-User] replication bandwith limit not respected [not found] <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr> @ 2024-09-06 15:10 ` Stefan Nehlsen 2024-09-06 17:18 ` Stefan Radman via pve-user 2024-09-06 17:18 ` Stefan Radman via pve-user 2 siblings, 0 replies; 5+ messages in thread From: Stefan Nehlsen @ 2024-09-06 15:10 UTC (permalink / raw) To: pve-user Am 06.09.24 um 15:00 schrieb Tonči Stipičević: > Hello, > > I'm running latest cluster (community support) A little more information about your cluster might help. > And today I've lowered replication speed down to 80MB/s but replcation > still uses whole NIC bandwith (1G) ... after few host restarts still > the same > > 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second > for transferring 'data2:subvol-1007-disk-0' > > Does somebody else experience that too ? Nowadays everything will be able to saturize 1Gbps links. Better you add faster hardware to your cluster. Cheapest solution will be connecting 2 nodes directly with 2 USB3 ethernet devices and a cable. 2.5 or 5 Gbps just for cluster communication or/and replication will make a difference. 10Gbps cards and switches wound be even better. Dual 40 Gbps connections probably may be overkill :-) It all depends on what you are doing. Stefan _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PVE-User] replication bandwith limit not respected [not found] <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr> 2024-09-06 15:10 ` Stefan Nehlsen @ 2024-09-06 17:18 ` Stefan Radman via pve-user 2024-09-06 17:18 ` Stefan Radman via pve-user 2 siblings, 0 replies; 5+ messages in thread From: Stefan Radman via pve-user @ 2024-09-06 17:18 UTC (permalink / raw) To: Proxmox VE user list; +Cc: Stefan Radman, PVE User List [-- Attachment #1: Type: message/rfc822, Size: 7563 bytes --] From: Stefan Radman <stefan.radman@me.com> To: Proxmox VE user list <pve-user@lists.proxmox.com> Cc: PVE User List <pve-user@pve.proxmox.com> Subject: Re: [PVE-User] replication bandwith limit not respected Date: Fri, 6 Sep 2024 20:18:20 +0300 Message-ID: <C89D18A7-57D5-4AF0-926D-A651237997AD@me.com> Hi Tonči I am not familiar with PVE storage replication but with networks. Your answers to the questions below might help everyone on this list (including me) to better understand the issues behind the problem you are facing. > lowered replication speed down to 80MB/s Have you tried much lower rates (e.g. 10MB/s)? Did anything change in the bandwidth utilization? > replcation still uses whole NIC bandwith (1G) How do you know? How do know that storage replication cloggs your network and not some other apllication? Where do you measure the replication traffic and how? Are you running Jumbo frames (e.g. MTU 9000) between the replication targets or standard MTU 1500? If you are not using a dedicated NIC for storage replication you are likely (and should be) running on standard MTU. Using the standard MTU will generate more packets than jumbo frames, thus more headers, thus more overhead. A bandwidth limit of 80MB/s translates to rougly 640Mbit/s but I assume that this does not include the overhead of protocols used by PVE storage replication (Ethernet, TCP/IP, SSH). How much overhead in % do you expect? My conservative assumption (rule of thumb) for SSH with standard MTU is 20% overhead. That would be 768Mbit/s, leaving only 232Mbit/s for other traffic. What is your output of “pveversion”? Regards Stefan > On Sep 6, 2024, at 16:00, Tonči Stipičević <tonci@suma-informatika.hr> wrote: > > Hello, > > I'm running latest cluster (community support) > > And today I've lowered replication speed down to 80MB/s but replcation still uses whole NIC bandwith (1G) ... after few host restarts still the same > > 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second for transferring 'data2:subvol-1007-disk-0' > > Does somebody else experience that too ? > > > Thank you in advance > > BR > > Tonči > > -- > srdačan pozdrav / best regards > > Tonči Stipičević, dipl. ing. elektr. > direktor / manager > > SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263 > > Podrška / Upravljanje IT sustavima za male i srednje tvrtke > Small & Medium Business IT Support / Management > > mob: 091 1234003 > www.suma-informatika.hr > > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user [-- Attachment #2: Type: text/plain, Size: 157 bytes --] _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PVE-User] replication bandwith limit not respected [not found] <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr> 2024-09-06 15:10 ` Stefan Nehlsen 2024-09-06 17:18 ` Stefan Radman via pve-user @ 2024-09-06 17:18 ` Stefan Radman via pve-user 2 siblings, 0 replies; 5+ messages in thread From: Stefan Radman via pve-user @ 2024-09-06 17:18 UTC (permalink / raw) To: Proxmox VE user list; +Cc: Stefan Radman, PVE User List [-- Attachment #1: Type: message/rfc822, Size: 7553 bytes --] From: Stefan Radman <stefan.radman@me.com> To: Proxmox VE user list <pve-user@lists.proxmox.com> Cc: PVE User List <pve-user@pve.proxmox.com> Subject: Re: [PVE-User] replication bandwith limit not respected Date: Fri, 6 Sep 2024 20:18:20 +0300 Message-ID: <C89D18A7-57D5-4AF0-926D-A651237997AD@me.com> Hi Tonči I am not familiar with PVE storage replication but with networks. Your answers to the questions below might help everyone on this list (including me) to better understand the issues behind the problem you are facing. > lowered replication speed down to 80MB/s Have you tried much lower rates (e.g. 10MB/s)? Did anything change in the bandwidth utilization? > replcation still uses whole NIC bandwith (1G) How do you know? How do know that storage replication cloggs your network and not some other apllication? Where do you measure the replication traffic and how? Are you running Jumbo frames (e.g. MTU 9000) between the replication targets or standard MTU 1500? If you are not using a dedicated NIC for storage replication you are likely (and should be) running on standard MTU. Using the standard MTU will generate more packets than jumbo frames, thus more headers, thus more overhead. A bandwidth limit of 80MB/s translates to rougly 640Mbit/s but I assume that this does not include the overhead of protocols used by PVE storage replication (Ethernet, TCP/IP, SSH). How much overhead in % do you expect? My conservative assumption (rule of thumb) for SSH with standard MTU is 20% overhead. That would be 768Mbit/s, leaving only 232Mbit/s for other traffic. What is your output of “pveversion”? Regards Stefan > On Sep 6, 2024, at 16:00, Tonči Stipičević <tonci@suma-informatika.hr> wrote: > > Hello, > > I'm running latest cluster (community support) > > And today I've lowered replication speed down to 80MB/s but replcation still uses whole NIC bandwith (1G) ... after few host restarts still the same > > 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second for transferring 'data2:subvol-1007-disk-0' > > Does somebody else experience that too ? > > > Thank you in advance > > BR > > Tonči > > -- > srdačan pozdrav / best regards > > Tonči Stipičević, dipl. ing. elektr. > direktor / manager > > SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263 > > Podrška / Upravljanje IT sustavima za male i srednje tvrtke > Small & Medium Business IT Support / Management > > mob: 091 1234003 > www.suma-informatika.hr > > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user [-- Attachment #2: Type: text/plain, Size: 157 bytes --] _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-09-15 20:38 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <mailman.95.1725643774.414.pve-user@lists.proxmox.com>
2024-09-09 7:40 ` [PVE-User] pve-user Digest, Vol 198, Issue 6 Tonči Stipičević
2024-09-15 20:37 ` [PVE-User] replication bandwith limit not respected Stefan Radman via pve-user
[not found] <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr>
2024-09-06 15:10 ` Stefan Nehlsen
2024-09-06 17:18 ` Stefan Radman via pve-user
2024-09-06 17:18 ` Stefan Radman via pve-user
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox