From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <d.csapak@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 409D370F8F for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 09:00:20 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3E83D27E1A for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 09:00:20 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 1FACC27E0E for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 09:00:19 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id D9B2444AB6; Fri, 1 Oct 2021 09:00:18 +0200 (CEST) Message-ID: <e12bd5f6-2304-5674-e5cb-394695c9f924@proxmox.com> Date: Fri, 1 Oct 2021 09:00:17 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:93.0) Gecko/20100101 Thunderbird/93.0 Content-Language: en-US To: Proxmox VE user list <pve-user@lists.proxmox.com>, Gregor Burck <gregor@aeppelbroe.de> References: <20211001085213.EGroupware.sb0JmHulYuMBOtEh9bqxti9@heim.aeppelbroe.de> From: Dominik Csapak <d.csapak@proxmox.com> In-Reply-To: <20211001085213.EGroupware.sb0JmHulYuMBOtEh9bqxti9@heim.aeppelbroe.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 1.937 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -3.499 Looks like a legit reply (A) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] proxmox-restore - performance issues X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Fri, 01 Oct 2021 07:00:20 -0000 On 10/1/21 08:52, Gregor Burck wrote: > Hi, hi, > > thank you for reply. I made a lot of different tests and setups, but > this the setup I want to use: > > Original setup: > > HP DL380 Gen9 with > > E5-2640 v3 @ 2.60GHz > 256 GB RAM > > 2x SSDs for host OS > > For an ZFS Rais 10: > > 2x 1TB SAMSUNG NVME PM983 for spezial devices > 12x 8 TB HP SAS HDDs i guess thats the server? what about the restore client? encryption/sha/etc. will be done by the client > > root@ph-pbs:~# zpool status > pool: ZFSPOOL > state: ONLINE > config: > > NAME STATE READ WRITE CKSUM > ZFSPOOL ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > sdc ONLINE 0 0 0 > sdd ONLINE 0 0 0 > mirror-1 ONLINE 0 0 0 > sde ONLINE 0 0 0 > sdf ONLINE 0 0 0 > mirror-2 ONLINE 0 0 0 > sdg ONLINE 0 0 0 > sdh ONLINE 0 0 0 > mirror-3 ONLINE 0 0 0 > sdi ONLINE 0 0 0 > sdj ONLINE 0 0 0 > mirror-4 ONLINE 0 0 0 > sdk ONLINE 0 0 0 > sdl ONLINE 0 0 0 > mirror-5 ONLINE 0 0 0 > sdm ONLINE 0 0 0 > sdn ONLINE 0 0 0 > special > mirror-6 ONLINE 0 0 0 > nvme0n1 ONLINE 0 0 0 > nvme1n1 ONLINE 0 0 0 > > errors: No known data errors > > pool: rpool > state: ONLINE > scan: scrub repaired 0B in 00:02:40 with 0 errors on Sun Aug 8 > 00:26:43 2021 > config: > > NAME STATE READ WRITE CKSUM > rpool ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > sda3 ONLINE 0 0 0 > sdb3 ONLINE 0 0 0 > > errors: No known data errors > > The VMSTORE and the BACKUPSTORE is on the zsf as an dataset: > > root@ph-pbs:~# zfs list > NAME USED AVAIL REFER MOUNTPOINT > ZFSPOOL 10.1T 32.1T 96K /ZFSPOOL > ZFSPOOL/BACKUPSTORE001 5.63T 32.1T 5.63T /ZFSPOOL/BACKUPSTORE001 > ZFSPOOL/VMSTORE001 4.52T 32.1T 4.52T /ZFSPOOL/VMSTORE001 > rpool 27.3G 80.2G 96K /rpool > rpool/ROOT 27.3G 80.2G 96K /rpool/ROOT > rpool/ROOT/pbs-1 27.3G 80.2G 27.3G / > > The VM I tested with is our Exchange Server. Raw image size 500GB, > netto ~400GB content > > First Test with one restore job: > > Virtual > Environment 7.0-11 > Datacenter > Search: > Logs > new > volume ID is 'VMSTORE:vm-101-disk-0' > restore > proxmox backup image: /usr/bin/pbs-restore --repository > root@pam@ph-pbs.peiker-holding.de:ZFSPOOLBACKUP > vm/121/2021-07-23T19:00:03Z drive-virtio0.img.fidx > /dev/zvol/ZFSPOOLVMSTORE/vm-101-disk-0 --verbose --format raw > --skip-zero > connecting > to repository 'root@pam@ph-pbs.peiker-holding.de:ZFSPOOLBACKUP' > open > block backend for target '/dev/zvol/ZFSPOOLVMSTORE/vm-101-disk-0' > starting > to restore snapshot 'vm/121/2021-07-23T19:00:03Z' > download > and verify backup index > progress > 1% (read 5368709120 bytes, zeroes = 2% (125829120 bytes), duration 86 > sec) > progress > 2% (read 10737418240 bytes, zeroes = 1% (159383552 bytes), duration > 181 sec) > progress > 3% (read 16106127360 bytes, zeroes = 0% (159383552 bytes), duration > 270 sec) > . > . > progress > 98% (read 526133493760 bytes, zeroes = 0% (3628072960 bytes), > duration 9492 sec) > progress > 99% (read 531502202880 bytes, zeroes = 0% (3628072960 bytes), > duration 9583 sec) > progress > 100% (read 536870912000 bytes, zeroes = 0% (3628072960 bytes), > duration 9676 sec) > restore > image complete (bytes=536870912000, duration=9676.97s, > speed=52.91MB/s) > rescan > volumes... > TASK > OK > > When I regard iotop I see about the same rate. > > But when I start multiple restore jobs parallel, I see that the single > jon is still on IO 40-50 MB/s but the total IO is multiple of the > rate. I see on iotop rates to 200-250 MB/s > So I guess it isn't the store. In some Test with an Setup where I used > the nvmes as source and target I could reach a singele restore rate > about 70 MB/s some disks/storages do not scale with single threaded workloads (and AFAIR, the pbs-restore must restore a disk single threaded because of qemu limitations?), but will scale with multiple threads just fine a 'fio' benchmark of the source as well as the target storage would be good to get a baseline storage perfomance see for example: https://pve.proxmox.com/wiki/Benchmarking_Storage > > Now I test an other CPU in this machine, cause on other test machines > with other CPU (AMD Ryzen or others) I get an higher rate. > Unfortunaly the rate on the current machine doesn't rise with the other > CPU can you do a proxmox-backup-client benchmark on all machines and their respective restore speed (especially the clients; also specify a repository to see tls speed) kind regards