From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id DF08970F67 for ; Fri, 1 Oct 2021 08:52:15 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D249727C8B for ; Fri, 1 Oct 2021 08:52:15 +0200 (CEST) Received: from mail.ud08.udmedia.de (ud08.udmedia.de [194.117.254.48]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 34AD427C82 for ; Fri, 1 Oct 2021 08:52:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=aeppelbroe.de; h=from:to :subject:message-id:date:content-type:mime-version; s=k1; bh=ZoO upe3844BihH6Chvw88aVWUoUOWKAP2FhJculaC08=; b=ErBcJdX6ujp7jLLxe1r sJKJIdWAuDxAZJN9+R01NTjCIiOOe2EX/AQwLh0ZSDt/eD2+TjEc8qlaQWaRjVe/ Fso+Rblbqm8YJPLezFUQ+nNY4xGjsmn9SDv5Ms5IwaLNlPOCgF/yXrYSYdVmkkdJ 0Rb5/DVoIHJ+V9YG/oWjyPfI= Received: (qmail 3568759 invoked from network); 1 Oct 2021 08:52:14 +0200 Received: by mail.ud08.udmedia.de with ESMTPSA (TLS_AES_256_GCM_SHA384 encrypted, authenticated); 1 Oct 2021 08:52:14 +0200 X-UD-Smtp-Session: ud08?166p2@wNrZAEXNY+pf3hNG Received: from localhost (localhost [127.0.0.1]) by baikonur.fritz.box (Postfix) with ESMTP id 397E4D27865 for ; Fri, 1 Oct 2021 08:52:14 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at baikonur.orbit.local Received: from baikonur.fritz.box ([127.0.0.1]) by localhost (baikonur.fritz.box [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2dWANgwL7njT for ; Fri, 1 Oct 2021 08:52:13 +0200 (CEST) Received: from baikonur (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by baikonur.fritz.box (Postfix) with ESMTPS id D773CD27862 for ; Fri, 1 Oct 2021 08:52:13 +0200 (CEST) User-Agent: EGroupware API 17.1.007 From: Gregor Burck X-Priority: 3 X-Mailer: EGroupware-Mail To: Proxmox VE user list Message-ID: <20211001085213.EGroupware.sb0JmHulYuMBOtEh9bqxti9@heim.aeppelbroe.de> Date: Fri, 01 Oct 2021 08:52:13 +0200 Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.660 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_INVALID 0.1 DKIM or DK signature exists, but is not valid DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid FORGED_SPF_HELO 1 - KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KHOP_HELO_FCRDNS 0.399 Relay HELO differs from its IP's reverse DNS POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_NONE 0.001 SPF: sender does not publish an SPF Record T_FILL_THIS_FORM_SHORT 0.01 Fill in a short form with personal information Subject: Re: [PVE-User] proxmox-restore - performance issues X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Oct 2021 06:52:15 -0000 Hi, thank you for reply. I made a lot of different tests and setups, but this the setup I want to use: Original setup: HP DL380 Gen9 with E5-2640 v3 @ 2.60GHz 256 GB RAM 2x SSDs for host OS For an ZFS Rais 10: 2x 1TB SAMSUNG NVME PM983 for spezial devices 12x 8 TB HP SAS HDDs root@ph-pbs:~# zpool status pool: ZFSPOOL state: ONLINE config: NAME STATE READ WRITE CKSUM ZFSPOOL ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 sdi ONLINE 0 0 0 sdj ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 sdk ONLINE 0 0 0 sdl ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 sdm ONLINE 0 0 0 sdn ONLINE 0 0 0 special mirror-6 ONLINE 0 0 0 nvme0n1 ONLINE 0 0 0 nvme1n1 ONLINE 0 0 0 errors: No known data errors pool: rpool state: ONLINE scan: scrub repaired 0B in 00:02:40 with 0 errors on Sun Aug 8 00:26:43 2021 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda3 ONLINE 0 0 0 sdb3 ONLINE 0 0 0 errors: No known data errors The VMSTORE and the BACKUPSTORE is on the zsf as an dataset: root@ph-pbs:~# zfs list NAME USED AVAIL REFER MOUNTPOINT ZFSPOOL 10.1T 32.1T 96K /ZFSPOOL ZFSPOOL/BACKUPSTORE001 5.63T 32.1T 5.63T /ZFSPOOL/BACKUPSTORE001 ZFSPOOL/VMSTORE001 4.52T 32.1T 4.52T /ZFSPOOL/VMSTORE001 rpool 27.3G 80.2G 96K /rpool rpool/ROOT 27.3G 80.2G 96K /rpool/ROOT rpool/ROOT/pbs-1 27.3G 80.2G 27.3G / The VM I tested with is our Exchange Server. Raw image size 500GB, netto ~400GB content First Test with one restore job: Virtual Environment 7.0-11 Datacenter Search: Logs new volume ID is 'VMSTORE:vm-101-disk-0' restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@ph-pbs.peiker-holding.de:ZFSPOOLBACKUP vm/121/2021-07-23T19:00:03Z drive-virtio0.img.fidx /dev/zvol/ZFSPOOLVMSTORE/vm-101-disk-0 --verbose --format raw --skip-zero connecting to repository 'root@pam@ph-pbs.peiker-holding.de:ZFSPOOLBACKUP' open block backend for target '/dev/zvol/ZFSPOOLVMSTORE/vm-101-disk-0' starting to restore snapshot 'vm/121/2021-07-23T19:00:03Z' download and verify backup index progress 1% (read 5368709120 bytes, zeroes = 2% (125829120 bytes), duration 86 sec) progress 2% (read 10737418240 bytes, zeroes = 1% (159383552 bytes), duration 181 sec) progress 3% (read 16106127360 bytes, zeroes = 0% (159383552 bytes), duration 270 sec) . . progress 98% (read 526133493760 bytes, zeroes = 0% (3628072960 bytes), duration 9492 sec) progress 99% (read 531502202880 bytes, zeroes = 0% (3628072960 bytes), duration 9583 sec) progress 100% (read 536870912000 bytes, zeroes = 0% (3628072960 bytes), duration 9676 sec) restore image complete (bytes=536870912000, duration=9676.97s, speed=52.91MB/s) rescan volumes... TASK OK When I regard iotop I see about the same rate. But when I start multiple restore jobs parallel, I see that the single jon is still on IO 40-50 MB/s but the total IO is multiple of the rate. I see on iotop rates to 200-250 MB/s So I guess it isn't the store. In some Test with an Setup where I used the nvmes as source and target I could reach a singele restore rate about 70 MB/s Now I test an other CPU in this machine, cause on other test machines with other CPU (AMD Ryzen or others) I get an higher rate. Unfortunaly the rate on the current machine doesn't rise with the other CPU. Now I confused if there is any chance to get the restore rate better. Bye Gregor