From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id A4A1898FA8 for ; Wed, 19 Apr 2023 19:40:48 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 83D9C23F47 for ; Wed, 19 Apr 2023 19:40:18 +0200 (CEST) Received: from picard.linux.it (picard.linux.it [IPv6:2001:1418:10:5::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 19 Apr 2023 19:40:17 +0200 (CEST) Received: by picard.linux.it (Postfix, from userid 10) id BCFBA3CC015; Wed, 19 Apr 2023 19:40:04 +0200 (CEST) Received: from news by eraldo.lilliput.linux.it with local (Exim 4.89) (envelope-from ) id 1ppBir-00055b-9m for pve-user@lists.proxmox.com; Wed, 19 Apr 2023 19:36:01 +0200 From: Marco Gaiarin Date: Wed, 19 Apr 2023 19:07:20 +0200 Organization: Il gaio usa sempre TIN per le liste, fallo anche tu!!! Message-ID: <6nj5hj-3a81.ln1@hermione.lilliput.linux.it> X-Trace: eraldo.lilliput.linux.it 1681924659 7927 192.168.24.2 (19 Apr 2023 17:17:39 GMT) X-Mailer: tin/2.6.2-20220130 ("Convalmore") (Linux/5.15.0-70-generic (x86_64)) X-Gateway-System: SmartGate 1.4.5 To: pve-user@lists.proxmox.com X-SPAM-LEVEL: Spam detection results: 0 AWL 0.227 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy JMQ_SPF_NEUTRAL 0.5 SPF set to ?all KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [PVE-User] DELL PowerEdge T440/PERC H750 and embarassing HDD performance... X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Apr 2023 17:40:48 -0000 Situation: some little PVE clusters with ZFS, still on PVE6. We have a set of PowerEdge T340 with PERC H330 Adapter in JBOD mode, that perform decently with HDD disks and ZFS. We have also a set of DELL PowerEdge T440 with PERC H750, that does NOT have a JBOD mode, but a 'Non-RAID' auto-RAID0 mode, and perform 'indecently' on HDD disks, for example: root@pppve1:~# fio --filename=/dev/sdc --direct=1 --rw=randrw --bs=128k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=hdd-rw-128 hdd-rw-128: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256 ... fio-3.12 Starting 4 processes Jobs: 4 (f=4): [m(4)][0.0%][eta 08d:17h:11m:29s] hdd-rw-128: (groupid=0, jobs=4): err= 0: pid=26198: Wed May 18 19:11:04 2022 read: IOPS=84, BW=10.5MiB/s (11.0MB/s)(1279MiB/121557msec) slat (usec): min=4, max=303887, avg=23029.19, stdev=61832.29 clat (msec): min=1329, max=6673, avg=4737.71, stdev=415.84 lat (msec): min=1543, max=6673, avg=4760.74, stdev=420.10 clat percentiles (msec): | 1.00th=[ 2802], 5.00th=[ 4329], 10.00th=[ 4463], 20.00th=[ 4530], | 30.00th=[ 4597], 40.00th=[ 4665], 50.00th=[ 4732], 60.00th=[ 4799], | 70.00th=[ 4866], 80.00th=[ 4933], 90.00th=[ 5134], 95.00th=[ 5336], | 99.00th=[ 5805], 99.50th=[ 6007], 99.90th=[ 6342], 99.95th=[ 6409], | 99.99th=[ 6611] bw ( KiB/s): min= 256, max= 5120, per=25.18%, avg=2713.08, stdev=780.45, samples=929 iops : min= 2, max= 40, avg=21.13, stdev= 6.10, samples=929 write: IOPS=87, BW=10.9MiB/s (11.5MB/s)(1328MiB/121557msec); 0 zone resets slat (usec): min=9, max=309914, avg=23025.13, stdev=61676.77 clat (msec): min=1444, max=13086, avg=6943.12, stdev=2068.26 lat (msec): min=1543, max=13086, avg=6966.15, stdev=2069.28 clat percentiles (msec): | 1.00th=[ 2769], 5.00th=[ 4597], 10.00th=[ 4799], 20.00th=[ 5067], | 30.00th=[ 5403], 40.00th=[ 5873], 50.00th=[ 6409], 60.00th=[ 7148], | 70.00th=[ 8020], 80.00th=[ 9060], 90.00th=[10134], 95.00th=[10671], | 99.00th=[11610], 99.50th=[11879], 99.90th=[12550], 99.95th=[12550], | 99.99th=[12684] bw ( KiB/s): min= 256, max= 5376, per=24.68%, avg=2762.20, stdev=841.30, samples=926 iops : min= 2, max= 42, avg=21.52, stdev= 6.56, samples=926 cpu : usr=0.05%, sys=0.09%, ctx=2847, majf=0, minf=49 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.6%, >=64=98.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=10233,10627,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=10.5MiB/s (11.0MB/s), 10.5MiB/s-10.5MiB/s (11.0MB/s-11.0MB/s), io=1279MiB (1341MB), run=121557-121557msec WRITE: bw=10.9MiB/s (11.5MB/s), 10.9MiB/s-10.9MiB/s (11.5MB/s-11.5MB/s), io=1328MiB (1393MB), run=121557-121557msec Disk stats (read/write): sdc: ios=10282/10601, merge=0/0, ticks=3041312/27373721, in_queue=30373472, util=99.99% note in particular the slow IOPS, very slow... Someone have some hint to share?! Thanks. -- La macchina del capo la guida Emilio Fede La macchina del capo la lava Emilio Fede La macchina del capo la parcheggia Emilio Fede ma la benzina gliela paghiamo noi [Dado]