From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <gregor@aeppelbroe.de>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id DF08970F67
for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 08:52:15 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id D249727C8B
for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 08:52:15 +0200 (CEST)
Received: from mail.ud08.udmedia.de (ud08.udmedia.de [194.117.254.48])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS id 34AD427C82
for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 08:52:15 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=aeppelbroe.de; h=from:to
:subject:message-id:date:content-type:mime-version; s=k1; bh=ZoO
upe3844BihH6Chvw88aVWUoUOWKAP2FhJculaC08=; b=ErBcJdX6ujp7jLLxe1r
sJKJIdWAuDxAZJN9+R01NTjCIiOOe2EX/AQwLh0ZSDt/eD2+TjEc8qlaQWaRjVe/
Fso+Rblbqm8YJPLezFUQ+nNY4xGjsmn9SDv5Ms5IwaLNlPOCgF/yXrYSYdVmkkdJ
0Rb5/DVoIHJ+V9YG/oWjyPfI=
Received: (qmail 3568759 invoked from network); 1 Oct 2021 08:52:14 +0200
Received: by mail.ud08.udmedia.de with ESMTPSA (TLS_AES_256_GCM_SHA384
encrypted, authenticated); 1 Oct 2021 08:52:14 +0200
X-UD-Smtp-Session: ud08?166p2@wNrZAEXNY+pf3hNG
Received: from localhost (localhost [127.0.0.1])
by baikonur.fritz.box (Postfix) with ESMTP id 397E4D27865
for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 08:52:14 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at baikonur.orbit.local
Received: from baikonur.fritz.box ([127.0.0.1])
by localhost (baikonur.fritz.box [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id 2dWANgwL7njT for <pve-user@lists.proxmox.com>;
Fri, 1 Oct 2021 08:52:13 +0200 (CEST)
Received: from baikonur (localhost [127.0.0.1])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest
SHA256) (No client certificate requested)
by baikonur.fritz.box (Postfix) with ESMTPS id D773CD27862
for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 08:52:13 +0200 (CEST)
User-Agent: EGroupware API 17.1.007
From: Gregor Burck <gregor@aeppelbroe.de>
X-Priority: 3
X-Mailer: EGroupware-Mail
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Message-ID: <20211001085213.EGroupware.sb0JmHulYuMBOtEh9bqxti9@heim.aeppelbroe.de>
Date: Fri, 01 Oct 2021 08:52:13 +0200
Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.660 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DKIM_INVALID 0.1 DKIM or DK signature exists, but is not valid
DKIM_SIGNED 0.1 Message has a DKIM or DK signature,
not necessarily valid FORGED_SPF_HELO 1 -
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
KHOP_HELO_FCRDNS 0.399 Relay HELO differs from its IP's reverse DNS
POISEN_SPAM_PILL 0.1 Meta: its spam
POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes
POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes
SPF_HELO_PASS -0.001 SPF: HELO matches SPF record
SPF_NONE 0.001 SPF: sender does not publish an SPF Record
T_FILL_THIS_FORM_SHORT 0.01 Fill in a short form with personal information
Subject: Re: [PVE-User] proxmox-restore - performance issues
X-BeenThere: pve-user@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE user list <pve-user.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-user/>
List-Post: <mailto:pve-user@lists.proxmox.com>
List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Fri, 01 Oct 2021 06:52:15 -0000
Hi,
thank you for reply. I made a lot of different tests and setups, but
this the setup I want to use:
Original setup:
HP DL380 Gen9 with
E5-2640 v3 @ 2.60GHz
256 GB RAM
2x SSDs for host OS
For an ZFS Rais 10:
2x 1TB SAMSUNG NVME PM983 for spezial devices
12x 8 TB HP SAS HDDs
root@ph-pbs:~# zpool status
pool: ZFSPOOL
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
ZFSPOOL ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
sdi ONLINE 0 0 0
sdj ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
sdk ONLINE 0 0 0
sdl ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
sdm ONLINE 0 0 0
sdn ONLINE 0 0 0
special
mirror-6 ONLINE 0 0 0
nvme0n1 ONLINE 0 0 0
nvme1n1 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scan: scrub repaired 0B in 00:02:40 with 0 errors on Sun Aug 8
00:26:43 2021
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda3 ONLINE 0 0 0
sdb3 ONLINE 0 0 0
errors: No known data errors
The VMSTORE and the BACKUPSTORE is on the zsf as an dataset:
root@ph-pbs:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
ZFSPOOL 10.1T 32.1T 96K /ZFSPOOL
ZFSPOOL/BACKUPSTORE001 5.63T 32.1T 5.63T /ZFSPOOL/BACKUPSTORE001
ZFSPOOL/VMSTORE001 4.52T 32.1T 4.52T /ZFSPOOL/VMSTORE001
rpool 27.3G 80.2G 96K /rpool
rpool/ROOT 27.3G 80.2G 96K /rpool/ROOT
rpool/ROOT/pbs-1 27.3G 80.2G 27.3G /
The VM I tested with is our Exchange Server. Raw image size 500GB,
netto ~400GB content
First Test with one restore job:
Virtual
Environment 7.0-11
Datacenter
Search:
Logs
new
volume ID is 'VMSTORE:vm-101-disk-0'
restore
proxmox backup image: /usr/bin/pbs-restore --repository
root@pam@ph-pbs.peiker-holding.de:ZFSPOOLBACKUP
vm/121/2021-07-23T19:00:03Z drive-virtio0.img.fidx
/dev/zvol/ZFSPOOLVMSTORE/vm-101-disk-0 --verbose --format raw
--skip-zero
connecting
to repository 'root@pam@ph-pbs.peiker-holding.de:ZFSPOOLBACKUP'
open
block backend for target '/dev/zvol/ZFSPOOLVMSTORE/vm-101-disk-0'
starting
to restore snapshot 'vm/121/2021-07-23T19:00:03Z'
download
and verify backup index
progress
1% (read 5368709120 bytes, zeroes = 2% (125829120 bytes), duration 86
sec)
progress
2% (read 10737418240 bytes, zeroes = 1% (159383552 bytes), duration
181 sec)
progress
3% (read 16106127360 bytes, zeroes = 0% (159383552 bytes), duration
270 sec)
.
.
progress
98% (read 526133493760 bytes, zeroes = 0% (3628072960 bytes),
duration 9492 sec)
progress
99% (read 531502202880 bytes, zeroes = 0% (3628072960 bytes),
duration 9583 sec)
progress
100% (read 536870912000 bytes, zeroes = 0% (3628072960 bytes),
duration 9676 sec)
restore
image complete (bytes=536870912000, duration=9676.97s,
speed=52.91MB/s)
rescan
volumes...
TASK
OK
When I regard iotop I see about the same rate.
But when I start multiple restore jobs parallel, I see that the single
jon is still on IO 40-50 MB/s but the total IO is multiple of the
rate. I see on iotop rates to 200-250 MB/s
So I guess it isn't the store. In some Test with an Setup where I used
the nvmes as source and target I could reach a singele restore rate
about 70 MB/s
Now I test an other CPU in this machine, cause on other test machines
with other CPU (AMD Ryzen or others) I get an higher rate.
Unfortunaly the rate on the current machine doesn't rise with the other CPU.
Now I confused if there is any chance to get the restore rate better.
Bye
Gregor