From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dietmar@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
 (No client certificate requested)
 by lists.proxmox.com (Postfix) with ESMTPS id 3D3D973129
 for <pbs-devel@lists.proxmox.com>; Wed, 14 Apr 2021 18:45:03 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
 by firstgate.proxmox.com (Proxmox) with ESMTP id 2B02011EAA
 for <pbs-devel@lists.proxmox.com>; Wed, 14 Apr 2021 18:45:03 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
 [212.186.127.180])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
 (No client certificate requested)
 by firstgate.proxmox.com (Proxmox) with ESMTPS id EFC3C11E9E
 for <pbs-devel@lists.proxmox.com>; Wed, 14 Apr 2021 18:45:01 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
 by proxmox-new.maurer-it.com (Proxmox) with ESMTP id BF7F545A99
 for <pbs-devel@lists.proxmox.com>; Wed, 14 Apr 2021 18:45:01 +0200 (CEST)
Date: Wed, 14 Apr 2021 18:44:46 +0200 (CEST)
From: Dietmar Maurer <dietmar@proxmox.com>
To: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com>, 
 =?UTF-8?Q?Fabian_Gr=C3=BCnbichler?= <f.gruenbichler@proxmox.com>
Message-ID: <1688290929.3134.1618418686372@webmail.proxmox.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Priority: 3
Importance: Normal
X-Mailer: Open-Xchange Mailer v7.10.5-Rev5
X-Originating-Client: open-xchange-appsuite
X-SPAM-LEVEL: Spam detection results:  0
 AWL 0.109 Adjusted score from AWL reputation of From: address
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 RCVD_IN_DNSWL_MED        -2.3 Sender listed at https://www.dnswl.org/,
 medium trust
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
Subject: Re: [pbs-devel] [PATCH proxmox-backup] backup/verify: improve speed
 by sorting chunks by inode
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
 <pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Wed, 14 Apr 2021 16:45:03 -0000

> zfs with single spinner + fast special device, with a (not counted ;))=20
> warmup run and everything fitting into cache:
>=20
> Benchmark #1: stock
>   Time (mean =C2=B1 =CF=83):     21.407 s =C2=B1  0.819 s    [User: 20.1 =
ms, System: 15.2 ms]
>   Range (min =E2=80=A6 max):   21.070 s =E2=80=A6 23.078 s    6 runs
>=20
> Benchmark #2: patched
>   Time (mean =C2=B1 =CF=83):     47.119 s =C2=B1  0.018 s    [User: 29.5 =
ms, System: 15.1 ms]
>   Range (min =E2=80=A6 max):   47.107 s =E2=80=A6 47.154 s    6 runs
>=20
> Summary
>   'stock' ran
>     2.20 =C2=B1 0.08 times faster than 'patched'

I assume you have about 3000 chunks? On XFS, I can do 3000 stat() calls per=
 second (after cache cleared).
So why the hell are stat calls that slow on ZFS with special device? Or wha=
t causes that delay (stat or sort)?