From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 98862CD0C for ; Tue, 19 Sep 2023 12:35:16 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 7A02499EA for ; Tue, 19 Sep 2023 12:35:16 +0200 (CEST) Received: from mx-out.tlen.pl (mx-out.tlen.pl [193.222.135.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 19 Sep 2023 12:35:15 +0200 (CEST) Received: (wp-smtpd smtp.tlen.pl 26899 invoked from network); 19 Sep 2023 12:28:34 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=o2.pl; s=1024a; t=1695119314; bh=l2kO2G8VOatPIvngtPa73kgVFBAWATNB+Q7dD/HBvXE=; h=To:From:Subject:Cc; b=gwxBVJr0hn8+mxxZQRLJLFLGhxmucCQbbw7din7/eA2Cpkz2rULD7XVhBQujPRzgm cF1cSAmHehjZ05JSoshH0sBrofUWDtlWg3wX/VqsPlFKVnu2opQuKlzqsFSHjx9QXT 8ZpppNc05N7uPlD7tw59C2M3SEjoYsavGbz2qot4= Received: from dns2.komandor.pl (HELO [172.20.0.200]) (sir_Misiek1@o2.pl@[80.48.162.20]) (envelope-sender ) by smtp.tlen.pl (WP-SMTPD) with ECDHE-RSA-AES256-GCM-SHA384 encrypted SMTP for ; 19 Sep 2023 12:28:34 +0200 Message-ID: <0f30b423-7ba9-8cdc-11c1-23929f8ff37d@o2.pl> Date: Tue, 19 Sep 2023 12:28:33 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Content-Language: en-US To: pve-user@pve.proxmox.com From: lord_Niedzwiedz Cc: "kkolator@poczta.onet.pl" X-Priority: 1 (Highest) Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-WP-MailID: 3e956d913f058cdbc4fd31eafa19e83d X-WP-AV: skaner antywirusowy Poczty o2 X-WP-SPAM: NO 0000002 [geHF] X-SPAM-LEVEL: Spam detection results: 0 AWL -0.637 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_ENVFROM_END_DIGIT 0.25 Envelope-from freemail username ends in digit FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider FREEMAIL_REPLY 1 From and body contain different freemails GB_FREEMAIL_DISPTO 0.001 Disposition-Notification-To/From or Disposition-Notification-To/body contain different freemails POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust RCVD_IN_MSPIKE_H5 0.001 Excellent reputation (+5) RCVD_IN_MSPIKE_WL 0.001 Mailspike good senders SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [ceph.sh, o2.pl] Subject: [PVE-User] CEPH - benchmark X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Sep 2023 10:35:16 -0000         I have non heterogenic network and hardware. CEPH about        Write 1160MB/sec,     Read 1820 MB/sec One nvme drive started going crazy. The performance of the entire array dropped catastrophically. The system said nothing. I wonder if there is any mechanism in CEPH/Proxmox that informs us about this automatically ?? I quickly wrote a script that periodically checks performance. root@tjall1:~# cat /Backup/script/Ceph.sh #!/bin/sh #        Ceph test by sir_Misiek@o2.pl #        Grzegorz Mi$kiewicz #        19.09.2023 MINWRITE=600 MINREAD=1200 POOL1=ceph-lxc POOL2=ceph-vm WRITE=`rados bench -p $POOL1 120 write --no-cleanup | grep "Bandwidth "| awk '{ print $3}'` READ=`rados bench -p $POOL1 60 rand| grep "Bandwidth "| awk '{ print $3}'`  echo Write = $WRITE  echo READ = $READ # We cut out everything before the dot. Converts a floating point value to an integer. WRITE_INT=${WRITE%%.*} READ_INT=${READ%%.*} if [ ${MINWRITE} -ge "${WRITE_INT}" ]; then     echo "Ceph slow write on pool($POOL1): ${WRITE} MB/sec"   fi if [ ${MINREAD} -ge "${READ_INT}" ]; then     echo "Ceph slow read on pool($POOL1): ${WRITE} MB/sec"   fi rados -p $POOL1 cleanup > /dev/null rados -p $POOL1 cleanup; sync; rados -p $POOL2 cleanup; sync exit