From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <wolf@wolfspyre.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 89D5998FEA
for <pve-user@lists.proxmox.com>; Thu, 20 Apr 2023 01:14:46 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 6ED2425442
for <pve-user@lists.proxmox.com>; Thu, 20 Apr 2023 01:14:46 +0200 (CEST)
Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com
[IPv6:2607:f8b0:4864:20::332])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-user@lists.proxmox.com>; Thu, 20 Apr 2023 01:14:44 +0200 (CEST)
Received: by mail-ot1-x332.google.com with SMTP id
46e09a7af769-6a60630574aso321827a34.1
for <pve-user@lists.proxmox.com>; Wed, 19 Apr 2023 16:14:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=wolfspyre-com.20221208.gappssmtp.com; s=20221208; t=1681946076; x=1684538076;
h=date:to:in-reply-to:references:message-id:subject:mime-version:from
:content-transfer-encoding:from:to:cc:subject:date:message-id
:reply-to; bh=4/qDFRPx8h7bN/17qJDIlykPaxDQ0ie3rrCZl2b5j0M=;
b=hs+P9AfjWjfPBSS91M5ijqjj27hUT8RLG/qdScDu8ObKCDJMVcFO4j/WZNxMZBKrmS
SvdPUM25EuVJ/Nu39Yw1g8lcx6hXQGYFe4KLGMFjzuQBnGMi7a3whNCfdKhkCkxMkaZx
sE0njiGPnfArjBWZnlu33TZw9lwEqBUa2zmimxiyPAC/d1c9jj3D2ZNWKK2wPA6Dig/S
Bcm7HS/ghlObo596yIY5n5W8e2ixoOzqQSeynaYHiD+sWA1Gr9SK72wAj3eKZnTaFjtI
m6xPrergiwtq8q6tEdvETcMrt8pdVZJ6DBITEup1Mcj+W+hH7h/aXqI0KvidlG85ShZS
0AZA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20221208; t=1681946076; x=1684538076;
h=date:to:in-reply-to:references:message-id:subject:mime-version:from
:content-transfer-encoding:x-gm-message-state:from:to:cc:subject
:date:message-id:reply-to;
bh=4/qDFRPx8h7bN/17qJDIlykPaxDQ0ie3rrCZl2b5j0M=;
b=QAyocZwdEhqPkC4SAJurVD7unyPklb0+fmCZkQ753iotfMGSGpJ7RWbUKxa7w+5uLO
7frypdYP9finszmENksLBoZmXa7qT9jSdgqaOQEvS92tqdFXvSSeL9FWu98WlFUiQnMn
bxvGtlTwzkDmpVShh3luR13zfeseE9z6oZSHGfAUXuVVx5AFfUMM56r2Irh9+ux0E/8f
je0+Dc/YY7jHYyhfvwN85PJUXQj8iVfd8Im/jGVaq7LbxtYW6wm+4yODpkbqMEu6KtwW
lCi3LJvIgtjY5NxggZy1bI5xNppipeUvln+SqzT6pBhYJ14OK9RPJjpLUMRfkZKzUoDi
xemg==
X-Gm-Message-State: AAQBX9cwzdbYn+hA2rGVJ2Yr9Ycr+yr33GhXndhLhDuYQbt0gQz6CP+x
wY67jKqaWU3Z0xz4sY4gsqmHAg==
X-Google-Smtp-Source: AKy350ZelTqKdRwR+XIlMGt5r7Q0S8H517r9Gq3wOfl6SF3na9kpVbBhEY5zWeCoHGbdWyoo3QOuSQ==
X-Received: by 2002:a9d:7352:0:b0:6a5:f146:6ddb with SMTP id
l18-20020a9d7352000000b006a5f1466ddbmr683290otk.14.1681946075749;
Wed, 19 Apr 2023 16:14:35 -0700 (PDT)
Received: from smtpclient.apple (who.wolfspaw.com. [108.221.46.19])
by smtp.gmail.com with ESMTPSA id
v28-20020a0568301bdc00b006a1394ea9f3sm154970ota.30.2023.04.19.16.14.35
(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
Wed, 19 Apr 2023 16:14:35 -0700 (PDT)
From: Wolf Noble <wolf@wolfspyre.com>
Mime-Version: 1.0 (1.0)
Message-Id: <C8002500-3A9E-4DAB-A607-5DF62CFB7CE4@wolfspyre.com>
References: <6nj5hj-3a81.ln1@hermione.lilliput.linux.it>
In-Reply-To: <6nj5hj-3a81.ln1@hermione.lilliput.linux.it>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Date: Wed, 19 Apr 2023 18:14:23 -0500
X-Mailer: iPhone Mail (20F5039e)
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.048 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DKIM_SIGNED 0.1 Message has a DKIM or DK signature,
not necessarily valid
DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
DMARC_MISSING 0.1 Missing DMARC policy
HTML_MESSAGE 0.001 HTML included in message
MIME_QP_LONG_LINE 0.001 Quoted-printable line longer than 76 chars
RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/,
no trust
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
T_SCC_BODY_TEXT_LINE -0.01 -
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [proxmox.com]
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.29
Subject: Re: [PVE-User] DELL PowerEdge T440/PERC H750 and embarassing HDD
performance...
X-BeenThere: pve-user@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE user list <pve-user.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-user/>
List-Post: <mailto:pve-user@lists.proxmox.com>
List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Wed, 19 Apr 2023 23:14:46 -0000
Hai Marco!
The voodoo of zfs and how it talks to the disks is a vast and confusing subj=
ect=E2=80=A6 there=E2=80=99s lots of moving pieces, and it=E2=80=99s a huge r=
abbit hole=E2=80=A6.
the broadest applicable advise is (fakeraid/passthru raid0/hardware raid) + z=
fs is a recipe for pain and poor performance.=20
ZFS expects to talk directly to the disk and have an accurate understanding o=
f the disk geometry=E2=80=A6 abstraction is problematic here=E2=80=A6 at bes=
t it introduces unnecessary complexity. At worst it exposes you to data loss=
.
I=E2=80=99ve not seen a single reputable source that=E2=80=99s reported good=
experiences combining zfs atop raid. even virtualized the benefit of zfs ge=
ts questionable imo, but that=E2=80=99s tangental..
granted, Im open to be proven wrong; but if possible i=E2=80=99d change out t=
he controller to one that isn=E2=80=99t playing games with the storage prese=
nted to zfs and see if performance improves=E2=80=A6
physical sectorsize, logical sector size, queue depth and speed of the spinn=
ing rust are also important..
jus my $.03 tho.
[=3D The contents of this message have been written, read, processed, erased=
, sorted, sniffed, compressed, rewritten, misspelled, overcompensated, lost,=
found, and most importantly delivered entirely with recycled electrons =3D]=
> On Apr 19, 2023, at 12:40, Marco Gaiarin <gaio@lilliput.linux.it> wrote:
>=20
> =EF=BB=BF
> Situation: some little PVE clusters with ZFS, still on PVE6.
>=20
> We have a set of PowerEdge T340 with PERC H330 Adapter in JBOD mode, that
> perform decently with HDD disks and ZFS.
>=20
> We have also a set of DELL PowerEdge T440 with PERC H750, that does NOT ha=
ve
> a JBOD mode, but a 'Non-RAID' auto-RAID0 mode, and perform 'indecently' on=
> HDD disks, for example:
>=20
> root@pppve1:~# fio --filename=3D/dev/sdc --direct=3D1 --rw=3Drandrw --bs=3D=
128k --ioengine=3Dlibaio --iodepth=3D256 --runtime=3D120 --numjobs=3D4 --tim=
e_based --group_reporting --name=3Dhdd-rw-128
> hdd-rw-128: (g=3D0): rw=3Drandrw, bs=3D(R) 128KiB-128KiB, (W) 128KiB-128Ki=
B, (T) 128KiB-128KiB, ioengine=3Dlibaio, iodepth=3D256
> ...
> fio-3.12
> Starting 4 processes
> Jobs: 4 (f=3D4): [m(4)][0.0%][eta 08d:17h:11m:29s] =
=20
> hdd-rw-128: (groupid=3D0, jobs=3D4): err=3D 0: pid=3D26198: Wed May 18 19:=
11:04 2022
> read: IOPS=3D84, BW=3D10.5MiB/s (11.0MB/s)(1279MiB/121557msec)
> slat (usec): min=3D4, max=3D303887, avg=3D23029.19, stdev=3D61832.29
> clat (msec): min=3D1329, max=3D6673, avg=3D4737.71, stdev=3D415.84
> lat (msec): min=3D1543, max=3D6673, avg=3D4760.74, stdev=3D420.10
> clat percentiles (msec):
> | 1.00th=3D[ 2802], 5.00th=3D[ 4329], 10.00th=3D[ 4463], 20.00th=3D[=
4530],
> | 30.00th=3D[ 4597], 40.00th=3D[ 4665], 50.00th=3D[ 4732], 60.00th=3D[=
4799],
> | 70.00th=3D[ 4866], 80.00th=3D[ 4933], 90.00th=3D[ 5134], 95.00th=3D[=
5336],
> | 99.00th=3D[ 5805], 99.50th=3D[ 6007], 99.90th=3D[ 6342], 99.95th=3D[=
6409],
> | 99.99th=3D[ 6611]
> bw ( KiB/s): min=3D 256, max=3D 5120, per=3D25.18%, avg=3D2713.08, std=
ev=3D780.45, samples=3D929
> iops : min=3D 2, max=3D 40, avg=3D21.13, stdev=3D 6.10, samp=
les=3D929
> write: IOPS=3D87, BW=3D10.9MiB/s (11.5MB/s)(1328MiB/121557msec); 0 zone r=
esets
> slat (usec): min=3D9, max=3D309914, avg=3D23025.13, stdev=3D61676.77
> clat (msec): min=3D1444, max=3D13086, avg=3D6943.12, stdev=3D2068.26
> lat (msec): min=3D1543, max=3D13086, avg=3D6966.15, stdev=3D2069.28
> clat percentiles (msec):
> | 1.00th=3D[ 2769], 5.00th=3D[ 4597], 10.00th=3D[ 4799], 20.00th=3D[=
5067],
> | 30.00th=3D[ 5403], 40.00th=3D[ 5873], 50.00th=3D[ 6409], 60.00th=3D[=
7148],
> | 70.00th=3D[ 8020], 80.00th=3D[ 9060], 90.00th=3D[10134], 95.00th=3D[=
10671],
> | 99.00th=3D[11610], 99.50th=3D[11879], 99.90th=3D[12550], 99.95th=3D[=
12550],
> | 99.99th=3D[12684]
> bw ( KiB/s): min=3D 256, max=3D 5376, per=3D24.68%, avg=3D2762.20, std=
ev=3D841.30, samples=3D926
> iops : min=3D 2, max=3D 42, avg=3D21.52, stdev=3D 6.56, samp=
les=3D926
> cpu : usr=3D0.05%, sys=3D0.09%, ctx=3D2847, majf=3D0, minf=3D49
> IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.2%, 16=3D0.3%, 32=3D0.=
6%, >=3D64=3D98.8%
> submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=3D=
0.0%, >=3D64=3D0.0%
> complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64=3D=
0.0%, >=3D64=3D0.1%
> issued rwts: total=3D10233,10627,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0=
> latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D256
>=20
> Run status group 0 (all jobs):
> READ: bw=3D10.5MiB/s (11.0MB/s), 10.5MiB/s-10.5MiB/s (11.0MB/s-11.0MB/s)=
, io=3D1279MiB (1341MB), run=3D121557-121557msec
> WRITE: bw=3D10.9MiB/s (11.5MB/s), 10.9MiB/s-10.9MiB/s (11.5MB/s-11.5MB/s)=
, io=3D1328MiB (1393MB), run=3D121557-121557msec
>=20
> Disk stats (read/write):
> sdc: ios=3D10282/10601, merge=3D0/0, ticks=3D3041312/27373721, in_queue=3D=
30373472, util=3D99.99%
>=20
> note in particular the slow IOPS, very slow...
>=20
>=20
> Someone have some hint to share?! Thanks.
>=20
> --=20
> La macchina del capo la guida Emilio Fede
> La macchina del capo la lava Emilio Fede
> La macchina del capo la parcheggia Emilio Fede
> ma la benzina gliela paghiamo noi [Dado]
>=20
>=20
>=20
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>=20