From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pve-user-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 669211FF15E for <inbox@lore.proxmox.com>; Fri, 6 Sep 2024 19:24:06 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 7044216316; Fri, 6 Sep 2024 19:24:40 +0200 (CEST) In-Reply-To: <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr> Date: Fri, 6 Sep 2024 20:18:20 +0300 References: <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr> To: Proxmox VE user list <pve-user@lists.proxmox.com> MIME-Version: 1.0 Message-ID: <mailman.93.1725643479.414.pve-user@lists.proxmox.com> List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Post: <mailto:pve-user@lists.proxmox.com> From: Stefan Radman via pve-user <pve-user@lists.proxmox.com> Precedence: list Cc: Stefan Radman <stefan.radman@me.com>, PVE User List <pve-user@pve.proxmox.com> X-Mailman-Version: 2.1.29 X-BeenThere: pve-user@lists.proxmox.com List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> Reply-To: Proxmox VE user list <pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> Subject: Re: [PVE-User] replication bandwith limit not respected Content-Type: multipart/mixed; boundary="===============0844335378325444310==" Errors-To: pve-user-bounces@lists.proxmox.com Sender: "pve-user" <pve-user-bounces@lists.proxmox.com> --===============0844335378325444310== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: <stefan.radman@me.com> X-Original-To: pve-user@lists.proxmox.com Delivered-To: pve-user@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 5D0EEC1112 for <pve-user@lists.proxmox.com>; Fri, 6 Sep 2024 19:24:38 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3FDC116152 for <pve-user@lists.proxmox.com>; Fri, 6 Sep 2024 19:24:08 +0200 (CEST) Received: from mr85p00im-ztdg06011901.me.com (mr85p00im-ztdg06011901.me.com [17.58.23.198]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pve-user@lists.proxmox.com>; Fri, 6 Sep 2024 19:24:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=me.com; s=1a1hai; t=1725643136; bh=+TEGALIp86MczN+bpcDDz4YOtdacRgdzayL0IdsOb8A=; h=Content-Type:Mime-Version:Subject:From:Date:Message-Id:To; b=zKe0cGOcFUI/zQv6WVjCOdf9mFqouvmt/Jxh+TxotGhZ6gJzC2CsK1mjiWQzCKczG 6mu+3fQvue4Tih+NZWBU7K5aQGpN6QtzaSmbPOVlqxKgnRZosE2inVsQBmXpfM1iYl An/YQp4UeFmz48aAkKwV7LGC7Orh9dxoMXasb2BXdVhZ/NPSCu3Zu29Li2xtyl+AZp HePOlH+tLJnzFsvEj15VifnnA/epOy2vOi5ehL1Itma9pVkQ+Zu3KjNVwXc05zyzOt rgL6Flfg/A+LbPJEue9/yoMaqK2vMd0XZQcBzOk9hHH0nSRzEFlu2HNdIFtrFszn2e cSqlUDbybKjNw== Received: from smtpclient.apple (mr38p00im-dlb-asmtp-mailmevip.me.com [17.57.152.18]) by mr85p00im-ztdg06011901.me.com (Postfix) with ESMTPSA id 1FB6E1349F1A; Fri, 6 Sep 2024 17:18:45 +0000 (UTC) Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: [PVE-User] replication bandwith limit not respected From: Stefan Radman <stefan.radman@me.com> In-Reply-To: <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr> Date: Fri, 6 Sep 2024 20:18:20 +0300 Cc: PVE User List <pve-user@pve.proxmox.com> Message-Id: <C89D18A7-57D5-4AF0-926D-A651237997AD@me.com> References: <3ee95219-6894-4ff6-a89c-9e4a8d81b74d@suma-informatika.hr> To: Proxmox VE user list <pve-user@lists.proxmox.com> X-Mailer: Apple Mail (2.3731.700.6) X-Proofpoint-GUID: 9EzgDgoVHXbAV9F6J4lYCZcspaSPlL8y X-Proofpoint-ORIG-GUID: 9EzgDgoVHXbAV9F6J4lYCZcspaSPlL8y X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.60.29 definitions=2024-09-06_03,2024-09-06_01,2024-09-02_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 phishscore=0 adultscore=0 mlxlogscore=999 spamscore=0 bulkscore=0 clxscore=1015 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2308100000 definitions=main-2409060127 X-Apple-Remote-Links: v=1;h=KCk=;charset=UTF-8 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.067 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider HTML_MESSAGE 0.001 HTML included in message POISEN_SPAM_PILL_4 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust RCVD_IN_MSPIKE_H3 0.001 Good reputation (+3) RCVD_IN_MSPIKE_WL 0.001 Mailspike good senders RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [me.com,proxmox.com,suma-informatika.hr] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Hi Ton=C4=8Di I am not familiar with PVE storage replication but with networks. Your answers to the questions below might help everyone on this list = (including me) to better understand the issues behind the problem you = are facing. > lowered replication speed down to 80MB/s Have you tried much lower rates (e.g. 10MB/s)? Did anything change in the bandwidth utilization? > replcation still uses whole NIC bandwith (1G) How do you know?=20 How do know that storage replication cloggs your network and not some = other apllication? Where do you measure the replication traffic and how? Are you running Jumbo frames (e.g. MTU 9000) between the replication = targets or standard MTU 1500? If you are not using a dedicated NIC for storage replication you are = likely (and should be) running on standard MTU.=20 Using the standard MTU will generate more packets than jumbo frames, = thus more headers, thus more overhead. A bandwidth limit of 80MB/s translates to rougly 640Mbit/s but I assume = that this does not include the overhead of protocols used by PVE storage = replication (Ethernet, TCP/IP, SSH). How much overhead in % do you expect? My conservative assumption (rule of thumb) for SSH with standard MTU is = 20% overhead.=20 That would be 768Mbit/s, leaving only 232Mbit/s for other traffic. What is your output of =E2=80=9Cpveversion=E2=80=9D? Regards Stefan > On Sep 6, 2024, at 16:00, Ton=C4=8Di Stipi=C4=8Devi=C4=87 = <tonci@suma-informatika.hr> wrote: >=20 > Hello, >=20 > I'm running latest cluster (community support) >=20 > And today I've lowered replication speed down to 80MB/s but = replcation still uses whole NIC bandwith (1G) ... after few host = restarts still the same >=20 > 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per = second for transferring 'data2:subvol-1007-disk-0' >=20 > Does somebody else experience that too ? >=20 >=20 > Thank you in advance >=20 > BR >=20 > Ton=C4=8Di >=20 > --=20 > srda=C4=8Dan pozdrav / best regards >=20 > Ton=C4=8Di Stipi=C4=8Devi=C4=87, dipl. ing. elektr. > direktor / manager >=20 > SUMA Informatika d.o.o., Badali=C4=87eva 27, OIB 93926415263 >=20 > Podr=C5=A1ka / Upravljanje IT sustavima za male i srednje tvrtke > Small & Medium Business IT Support / Management >=20 > mob: 091 1234003 > www.suma-informatika.hr >=20 >=20 > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user --===============0844335378325444310== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user --===============0844335378325444310==--