From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 2046C7111E for ; Thu, 23 Jun 2022 15:55:23 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 033AF26D59 for ; Thu, 23 Jun 2022 15:54:53 +0200 (CEST) Received: from mail2.mxhosting.eu (mail2.mxhosting.eu [5.196.214.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 9312626D46 for ; Thu, 23 Jun 2022 15:54:49 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mail2.mxhosting.eu (Postfix) with ESMTP id 53343385674; Thu, 23 Jun 2022 15:54:49 +0200 (CEST) Received: from mail2.mxhosting.eu ([127.0.0.1]) by localhost (mail2.mxhosting.eu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id j4sC-p_cY9yC; Thu, 23 Jun 2022 15:54:48 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mail2.mxhosting.eu (Postfix) with ESMTP id 25C9038671B; Thu, 23 Jun 2022 15:54:48 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail2.mxhosting.eu 25C9038671B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=branoviest.com; s=F39BEA7E-92E5-11E7-AA57-DA56C37E8AFF; t=1655992488; bh=44hQg4S/kx/FADUy/iNkhxlbF+6K+Y6acdk6x+r49M8=; h=Date:From:To:Message-ID:MIME-Version; b=ocw2bZ02YFHhIh5TlRzIITndutZPAjYpOLP4mdajJAaZ+HqoFqGcsqO/Dj8IJevVF VHZy9zZQwf5Un4cBI082izNT4KB17WIaz4D6SLoZ5tnLL/Xz+BFiS0a1zeVXAOmRtl 9362lDiMFUKSat5xylIfchyyaOjPROjlTn0XqFy8YSnX2LYCAFEDd7bIpXTy88B2lb /sRvyJV6FD32VusqkJZRCgks7j9VEApoJQUhjdr2P6fMRfPKpiUN0rjAFbrjko5ARS S6d0AFlh59ignwIMrQmuAWg3Y1AdQ17QSixTwY7Bs47mnsaxYxJ4vWtXnBBhVjaNDV I4lTSkk9iOqkA== X-Virus-Scanned: amavisd-new at mail2.mxhosting.eu Received: from mail2.mxhosting.eu ([127.0.0.1]) by localhost (mail2.mxhosting.eu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id cm7k5ZjR-Zlz; Thu, 23 Jun 2022 15:54:48 +0200 (CEST) Received: from mail2.mxhosting.eu (mail2.mxhosting.eu [5.196.214.12]) by mail2.mxhosting.eu (Postfix) with ESMTP id E43D4386713; Thu, 23 Jun 2022 15:54:47 +0200 (CEST) Date: Thu, 23 Jun 2022 15:54:47 +0200 (CEST) From: Branislav Viest To: Eneko Lacunza Cc: Proxmox VE user list Message-ID: <913364878.6289.1655992487687.JavaMail.zimbra@branoviest.com> In-Reply-To: <1cdc99da-0537-7533-2987-8678223e2971@binovo.es> References: <1918332790.5860.1655923185691.JavaMail.zimbra@branoviest.com> <1463978529.6086.1655967268966.JavaMail.zimbra@branoviest.com> <4c507d5f-5ad9-9fb1-6eab-6ca1913784d8@binovo.es> <842526590.6160.1655972650267.JavaMail.zimbra@branoviest.com> <1cdc99da-0537-7533-2987-8678223e2971@binovo.es> MIME-Version: 1.0 X-Originating-IP: [78.102.251.4] X-Mailer: Zimbra 8.8.15_GA_4272 (ZimbraWebClient - FF101 (Mac)/8.8.15_GA_4257) Thread-Topic: Ceph: sudden slow ops, freezes, and slow-downs Thread-Index: FXBlGisCyu0S4WdRe90krnvcRvvYsA== X-SPAM-LEVEL: Spam detection results: 0 AWL 0.228 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain HTML_MESSAGE 0.001 HTML included in message SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_REMOTE_IMAGE 0.01 Message contains an external image T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [branoviest.com, proxmox.com, binovo.es] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] Ceph: sudden slow ops, freezes, and slow-downs X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jun 2022 13:55:23 -0000 3 nodes have the same firmware on Drives (EDA5402Q) except the last one (ne= west) where is higher firmware version, EDA5702Q=20 ------------=20 Best Regards=20 Branislav Viest=20 ------------=20 Legal Disclaimer: This e-mail and any attached files are confidential and m= ay be legally privileged. If you are not the addressee, any disclosure, rep= roduction, copying, distribution, or other dissemination or use of this com= munication is strictly prohibited. If you have received this transmission i= n error please notify the sender immediately and then delete this e-mail. T= he sender does not accept liability for the correct and complete transmissi= on of the information, nor for any delay or interruption of the transmissio= n, nor for damages arising from the use of or reliance on the information. = All e-mail messages addressed to, received or sent by sender are deemed to = be professional in nature. Accordingly, the sender or recipient of these me= ssages agrees that they may be read by other sender employees than the offi= cial recipient or sender in order to ensure the continuity of work-related = activities and allow supervision thereof.=20 Od: "Eneko Lacunza" =20 Komu: "Branislav Viest" =20 K=C3=B3pia: "Proxmox VE user list" =20 Odoslan=C3=A9: =C5=A1tvrtok, 23. j=C3=BAn 2022 13:12:23=20 Predmet: Re: [PVE-User] Ceph: sudden slow ops, freezes, and slow-downs=20 Drives are DC so should be good. All have the same firmware version?=20 El 23/6/22 a las 10:24, Branislav Viest escribi=C3=B3:=20 OSD1,3,5,2,8=20 All drives are Samsung NVMe=20 Model Number: SAMSUNG MZQLB1T9HAJR-00007=20 According to SMART value "Percentage Used", all are up to 10%. All SMART ov= erall-health self-assessment test result are PASSED.=20 ------------=20 Best Regards=20 Branislav Brian Viest=20 ------------=20 Legal Disclaimer: This e-mail and any attached files are confidential and m= ay be legally privileged. If you are not the addressee, any disclosure, rep= roduction, copying, distribution, or other dissemination or use of this com= munication is strictly prohibited. If you have received this transmission i= n error please notify the sender immediately and then delete this e-mail. T= he sender does not accept liability for the correct and complete transmissi= on of the information, nor for any delay or interruption of the transmissio= n, nor for damages arising from the use of or reliance on the information. = All e-mail messages addressed to, received or sent by sender are deemed to = be professional in nature. Accordingly, the sender or recipient of these me= ssages agrees that they may be read by other sender employees than the offi= cial recipient or sender in order to ensure the continuity of work-related = activities and allow supervision thereof.=20 Od: "Eneko Lacunza" [ mailto:elacunza@binovo.es | ]=20 Komu: "Branislav Viest" [ mailto:info@branoviest.com | ] , "Proxmox VE user list" [ mailto:pve-user@lists.proxmox.com | ]=20 Odoslan=C3=A9: =C5=A1tvrtok, 23. j=C3=BAn 2022 9:22:08=20 Predmet: Re: [PVE-User] Ceph: sudden slow ops, freezes, and slow-downs=20 Hi,=20 What numbers are those 5 OSDs?=20 Hace you checked SSD drive manufacturer and models?=20 El 23/6/22 a las 8:54, Branislav Viest escribi=C3=B3:=20 BQ_BEGIN Hello, ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 15.52213 root default =20 -3 5.18097 host node1 =20 0 ssd 1.72699 osd.0 up 1.00000 1.00000 1 ssd 1.72699 osd.1 up 1.00000 1.00000 2 ssd 1.72699 osd.2 up 1.00000 1.00000 -5 3.45398 host node2 =20 3 ssd 1.72699 osd.3 up 1.00000 1.00000 5 ssd 1.72699 osd.5 up 1.00000 1.00000 -7 1.70740 host node3 =20 6 ssd 0.85370 osd.6 up 1.00000 1.00000 7 ssd 0.85370 osd.7 up 1.00000 1.00000 -9 5.17978 host node4 =20 8 ssd 1.72659 osd.8 up 1.00000 1.00000 9 ssd 1.72659 osd.9 up 1.00000 1.00000 10 ssd 1.72659 osd.10 up 1.00000 1.00000 Since slow ops are reported the most of the time within multiple OSDs, I di= d not try to perform tests with some OSDs out.=20 Now I check the logs from the last 2-3 days and slow ops are reported mostl= y on the 5 OSDs from total 10.=20 ------------ Best Regards Branislav Brian Viest ------------ Legal Disclaimer: This e-mail and any attached files are confidential and m= ay be legally privileged. If you are not the addressee, any disclosure, rep= roduction, copying, distribution, or other dissemination or use of this com= munication is strictly prohibited. If you have received this transmission i= n error please notify the sender immediately and then delete this e-mail. T= he sender does not accept liability for the correct and complete transmissi= on of the information, nor for any delay or interruption of the transmissio= n, nor for damages arising from the use of or reliance on the information. = All e-mail messages addressed to, received or sent by sender are deemed to = be professional in nature. Accordingly, the sender or recipient of these me= ssages agrees that they may be read by other sender employees than the offi= cial recipient or sender in order to ensure the continuity of work-related = activities and allow supervision thereof. ----- P=C3=B4vodn=C3=A1 spr=C3=A1va ----- Od: "Eneko Lacunza via pve-user" [ mailto:pve-user@lists.proxmox.com | ] Komu: "pve-user" [ mailto:pve-user@lists.proxmox= .com | ] K=C3=B3pia: "Eneko Lacunza" [ mailto:= elacunza@binovo.es | ] Odoslan=C3=A9: =C5=A1tvrtok, 23= . j=C3=BAn 2022 8:29:40 Predmet: Re: [PVE-User] Ceph: sudden slow ops, freezes, and slow-downs _______________________________________________ pve-user mailing list [ mailto:pve-user@lists.proxmox.com | pve-user@lists.= proxmox.com ] [ https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user= | https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ]=20 BQ_END Eneko=C2=A0Lacunza=20 Director T=C3=A9cnico | Zuzendari teknikoa=20 Binovo IT Human Project=20 =09943 569 206=20 =09[ mailto:elacunza@binovo.es | elacunza@binovo.es ]=20 =09[ https://binovo.es/ | binovo.es ]=20 =09Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun=20 =09[ https://www.youtube.com/user/CANALBINOVO/ ]=20 =09=09[ https://www.linkedin.com/company/37269706/ ]=20 =09