From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 89D5998FEA for ; Thu, 20 Apr 2023 01:14:46 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6ED2425442 for ; Thu, 20 Apr 2023 01:14:46 +0200 (CEST) Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 20 Apr 2023 01:14:44 +0200 (CEST) Received: by mail-ot1-x332.google.com with SMTP id 46e09a7af769-6a60630574aso321827a34.1 for ; Wed, 19 Apr 2023 16:14:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wolfspyre-com.20221208.gappssmtp.com; s=20221208; t=1681946076; x=1684538076; h=date:to:in-reply-to:references:message-id:subject:mime-version:from :content-transfer-encoding:from:to:cc:subject:date:message-id :reply-to; bh=4/qDFRPx8h7bN/17qJDIlykPaxDQ0ie3rrCZl2b5j0M=; b=hs+P9AfjWjfPBSS91M5ijqjj27hUT8RLG/qdScDu8ObKCDJMVcFO4j/WZNxMZBKrmS SvdPUM25EuVJ/Nu39Yw1g8lcx6hXQGYFe4KLGMFjzuQBnGMi7a3whNCfdKhkCkxMkaZx sE0njiGPnfArjBWZnlu33TZw9lwEqBUa2zmimxiyPAC/d1c9jj3D2ZNWKK2wPA6Dig/S Bcm7HS/ghlObo596yIY5n5W8e2ixoOzqQSeynaYHiD+sWA1Gr9SK72wAj3eKZnTaFjtI m6xPrergiwtq8q6tEdvETcMrt8pdVZJ6DBITEup1Mcj+W+hH7h/aXqI0KvidlG85ShZS 0AZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681946076; x=1684538076; h=date:to:in-reply-to:references:message-id:subject:mime-version:from :content-transfer-encoding:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=4/qDFRPx8h7bN/17qJDIlykPaxDQ0ie3rrCZl2b5j0M=; b=QAyocZwdEhqPkC4SAJurVD7unyPklb0+fmCZkQ753iotfMGSGpJ7RWbUKxa7w+5uLO 7frypdYP9finszmENksLBoZmXa7qT9jSdgqaOQEvS92tqdFXvSSeL9FWu98WlFUiQnMn bxvGtlTwzkDmpVShh3luR13zfeseE9z6oZSHGfAUXuVVx5AFfUMM56r2Irh9+ux0E/8f je0+Dc/YY7jHYyhfvwN85PJUXQj8iVfd8Im/jGVaq7LbxtYW6wm+4yODpkbqMEu6KtwW lCi3LJvIgtjY5NxggZy1bI5xNppipeUvln+SqzT6pBhYJ14OK9RPJjpLUMRfkZKzUoDi xemg== X-Gm-Message-State: AAQBX9cwzdbYn+hA2rGVJ2Yr9Ycr+yr33GhXndhLhDuYQbt0gQz6CP+x wY67jKqaWU3Z0xz4sY4gsqmHAg== X-Google-Smtp-Source: AKy350ZelTqKdRwR+XIlMGt5r7Q0S8H517r9Gq3wOfl6SF3na9kpVbBhEY5zWeCoHGbdWyoo3QOuSQ== X-Received: by 2002:a9d:7352:0:b0:6a5:f146:6ddb with SMTP id l18-20020a9d7352000000b006a5f1466ddbmr683290otk.14.1681946075749; Wed, 19 Apr 2023 16:14:35 -0700 (PDT) Received: from smtpclient.apple (who.wolfspaw.com. [108.221.46.19]) by smtp.gmail.com with ESMTPSA id v28-20020a0568301bdc00b006a1394ea9f3sm154970ota.30.2023.04.19.16.14.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 19 Apr 2023 16:14:35 -0700 (PDT) From: Wolf Noble Mime-Version: 1.0 (1.0) Message-Id: References: <6nj5hj-3a81.ln1@hermione.lilliput.linux.it> In-Reply-To: <6nj5hj-3a81.ln1@hermione.lilliput.linux.it> To: Proxmox VE user list Date: Wed, 19 Apr 2023 18:14:23 -0500 X-Mailer: iPhone Mail (20F5039e) X-SPAM-LEVEL: Spam detection results: 0 AWL -0.048 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DMARC_MISSING 0.1 Missing DMARC policy HTML_MESSAGE 0.001 HTML included in message MIME_QP_LONG_LINE 0.001 Quoted-printable line longer than 76 chars RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] DELL PowerEdge T440/PERC H750 and embarassing HDD performance... X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Apr 2023 23:14:46 -0000 Hai Marco! The voodoo of zfs and how it talks to the disks is a vast and confusing subj= ect=E2=80=A6 there=E2=80=99s lots of moving pieces, and it=E2=80=99s a huge r= abbit hole=E2=80=A6. the broadest applicable advise is (fakeraid/passthru raid0/hardware raid) + z= fs is a recipe for pain and poor performance.=20 ZFS expects to talk directly to the disk and have an accurate understanding o= f the disk geometry=E2=80=A6 abstraction is problematic here=E2=80=A6 at bes= t it introduces unnecessary complexity. At worst it exposes you to data loss= . I=E2=80=99ve not seen a single reputable source that=E2=80=99s reported good= experiences combining zfs atop raid. even virtualized the benefit of zfs ge= ts questionable imo, but that=E2=80=99s tangental.. granted, Im open to be proven wrong; but if possible i=E2=80=99d change out t= he controller to one that isn=E2=80=99t playing games with the storage prese= nted to zfs and see if performance improves=E2=80=A6 physical sectorsize, logical sector size, queue depth and speed of the spinn= ing rust are also important.. jus my $.03 tho. [=3D The contents of this message have been written, read, processed, erased= , sorted, sniffed, compressed, rewritten, misspelled, overcompensated, lost,= found, and most importantly delivered entirely with recycled electrons =3D]= > On Apr 19, 2023, at 12:40, Marco Gaiarin wrote: >=20 > =EF=BB=BF > Situation: some little PVE clusters with ZFS, still on PVE6. >=20 > We have a set of PowerEdge T340 with PERC H330 Adapter in JBOD mode, that > perform decently with HDD disks and ZFS. >=20 > We have also a set of DELL PowerEdge T440 with PERC H750, that does NOT ha= ve > a JBOD mode, but a 'Non-RAID' auto-RAID0 mode, and perform 'indecently' on= > HDD disks, for example: >=20 > root@pppve1:~# fio --filename=3D/dev/sdc --direct=3D1 --rw=3Drandrw --bs=3D= 128k --ioengine=3Dlibaio --iodepth=3D256 --runtime=3D120 --numjobs=3D4 --tim= e_based --group_reporting --name=3Dhdd-rw-128 > hdd-rw-128: (g=3D0): rw=3Drandrw, bs=3D(R) 128KiB-128KiB, (W) 128KiB-128Ki= B, (T) 128KiB-128KiB, ioengine=3Dlibaio, iodepth=3D256 > ... > fio-3.12 > Starting 4 processes > Jobs: 4 (f=3D4): [m(4)][0.0%][eta 08d:17h:11m:29s] = =20 > hdd-rw-128: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D26198: Wed May 18 19:= 11:04 2022 > read: IOPS=3D84, BW=3D10.5MiB/s (11.0MB/s)(1279MiB/121557msec) > slat (usec): min=3D4, max=3D303887, avg=3D23029.19, stdev=3D61832.29 > clat (msec): min=3D1329, max=3D6673, avg=3D4737.71, stdev=3D415.84 > lat (msec): min=3D1543, max=3D6673, avg=3D4760.74, stdev=3D420.10 > clat percentiles (msec): > | 1.00th=3D[ 2802], 5.00th=3D[ 4329], 10.00th=3D[ 4463], 20.00th=3D[= 4530], > | 30.00th=3D[ 4597], 40.00th=3D[ 4665], 50.00th=3D[ 4732], 60.00th=3D[= 4799], > | 70.00th=3D[ 4866], 80.00th=3D[ 4933], 90.00th=3D[ 5134], 95.00th=3D[= 5336], > | 99.00th=3D[ 5805], 99.50th=3D[ 6007], 99.90th=3D[ 6342], 99.95th=3D[= 6409], > | 99.99th=3D[ 6611] > bw ( KiB/s): min=3D 256, max=3D 5120, per=3D25.18%, avg=3D2713.08, std= ev=3D780.45, samples=3D929 > iops : min=3D 2, max=3D 40, avg=3D21.13, stdev=3D 6.10, samp= les=3D929 > write: IOPS=3D87, BW=3D10.9MiB/s (11.5MB/s)(1328MiB/121557msec); 0 zone r= esets > slat (usec): min=3D9, max=3D309914, avg=3D23025.13, stdev=3D61676.77 > clat (msec): min=3D1444, max=3D13086, avg=3D6943.12, stdev=3D2068.26 > lat (msec): min=3D1543, max=3D13086, avg=3D6966.15, stdev=3D2069.28 > clat percentiles (msec): > | 1.00th=3D[ 2769], 5.00th=3D[ 4597], 10.00th=3D[ 4799], 20.00th=3D[= 5067], > | 30.00th=3D[ 5403], 40.00th=3D[ 5873], 50.00th=3D[ 6409], 60.00th=3D[= 7148], > | 70.00th=3D[ 8020], 80.00th=3D[ 9060], 90.00th=3D[10134], 95.00th=3D[= 10671], > | 99.00th=3D[11610], 99.50th=3D[11879], 99.90th=3D[12550], 99.95th=3D[= 12550], > | 99.99th=3D[12684] > bw ( KiB/s): min=3D 256, max=3D 5376, per=3D24.68%, avg=3D2762.20, std= ev=3D841.30, samples=3D926 > iops : min=3D 2, max=3D 42, avg=3D21.52, stdev=3D 6.56, samp= les=3D926 > cpu : usr=3D0.05%, sys=3D0.09%, ctx=3D2847, majf=3D0, minf=3D49 > IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.2%, 16=3D0.3%, 32=3D0.= 6%, >=3D64=3D98.8% > submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=3D= 0.0%, >=3D64=3D0.0% > complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=3D= 0.0%, >=3D64=3D0.1% > issued rwts: total=3D10233,10627,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0= > latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D256 >=20 > Run status group 0 (all jobs): > READ: bw=3D10.5MiB/s (11.0MB/s), 10.5MiB/s-10.5MiB/s (11.0MB/s-11.0MB/s)= , io=3D1279MiB (1341MB), run=3D121557-121557msec > WRITE: bw=3D10.9MiB/s (11.5MB/s), 10.9MiB/s-10.9MiB/s (11.5MB/s-11.5MB/s)= , io=3D1328MiB (1393MB), run=3D121557-121557msec >=20 > Disk stats (read/write): > sdc: ios=3D10282/10601, merge=3D0/0, ticks=3D3041312/27373721, in_queue=3D= 30373472, util=3D99.99% >=20 > note in particular the slow IOPS, very slow... >=20 >=20 > Someone have some hint to share?! Thanks. >=20 > --=20 > La macchina del capo la guida Emilio Fede > La macchina del capo la lava Emilio Fede > La macchina del capo la parcheggia Emilio Fede > ma la benzina gliela paghiamo noi [Dado] >=20 >=20 >=20 > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >=20