From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 409D370F8F for ; Fri, 1 Oct 2021 09:00:20 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3E83D27E1A for ; Fri, 1 Oct 2021 09:00:20 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 1FACC27E0E for ; Fri, 1 Oct 2021 09:00:19 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id D9B2444AB6; Fri, 1 Oct 2021 09:00:18 +0200 (CEST) Message-ID: Date: Fri, 1 Oct 2021 09:00:17 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:93.0) Gecko/20100101 Thunderbird/93.0 Content-Language: en-US To: Proxmox VE user list , Gregor Burck References: <20211001085213.EGroupware.sb0JmHulYuMBOtEh9bqxti9@heim.aeppelbroe.de> From: Dominik Csapak In-Reply-To: <20211001085213.EGroupware.sb0JmHulYuMBOtEh9bqxti9@heim.aeppelbroe.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 1.937 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -3.499 Looks like a legit reply (A) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] proxmox-restore - performance issues X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Oct 2021 07:00:20 -0000 On 10/1/21 08:52, Gregor Burck wrote: > Hi, hi, > > thank you for reply. I made a lot of different tests and setups, but > this the setup I want to use: > > Original setup: > > HP DL380 Gen9 with > > E5-2640 v3 @ 2.60GHz > 256 GB RAM > > 2x SSDs for host OS > > For an ZFS Rais 10: > > 2x 1TB SAMSUNG NVME PM983 for spezial devices > 12x 8 TB HP SAS HDDs i guess thats the server? what about the restore client? encryption/sha/etc. will be done by the client > > root@ph-pbs:~# zpool status >    pool: ZFSPOOL >   state: ONLINE > config: > >          NAME         STATE     READ WRITE CKSUM >          ZFSPOOL      ONLINE       0     0     0 >            mirror-0   ONLINE       0     0     0 >              sdc      ONLINE       0     0     0 >              sdd      ONLINE       0     0     0 >            mirror-1   ONLINE       0     0     0 >              sde      ONLINE       0     0     0 >              sdf      ONLINE       0     0     0 >            mirror-2   ONLINE       0     0     0 >              sdg      ONLINE       0     0     0 >              sdh      ONLINE       0     0     0 >            mirror-3   ONLINE       0     0     0 >              sdi      ONLINE       0     0     0 >              sdj      ONLINE       0     0     0 >            mirror-4   ONLINE       0     0     0 >              sdk      ONLINE       0     0     0 >              sdl      ONLINE       0     0     0 >            mirror-5   ONLINE       0     0     0 >              sdm      ONLINE       0     0     0 >              sdn      ONLINE       0     0     0 >          special >            mirror-6   ONLINE       0     0     0 >              nvme0n1  ONLINE       0     0     0 >              nvme1n1  ONLINE       0     0     0 > > errors: No known data errors > >    pool: rpool >   state: ONLINE >    scan: scrub repaired 0B in 00:02:40 with 0 errors on Sun Aug  8 > 00:26:43 2021 > config: > >          NAME        STATE     READ WRITE CKSUM >          rpool       ONLINE       0     0     0 >            mirror-0  ONLINE       0     0     0 >              sda3    ONLINE       0     0     0 >              sdb3    ONLINE       0     0     0 > > errors: No known data errors > > The VMSTORE and the BACKUPSTORE is on the zsf as an dataset: > > root@ph-pbs:~# zfs list > NAME                     USED  AVAIL     REFER  MOUNTPOINT > ZFSPOOL                 10.1T  32.1T       96K  /ZFSPOOL > ZFSPOOL/BACKUPSTORE001  5.63T  32.1T     5.63T  /ZFSPOOL/BACKUPSTORE001 > ZFSPOOL/VMSTORE001      4.52T  32.1T     4.52T  /ZFSPOOL/VMSTORE001 > rpool                   27.3G  80.2G       96K  /rpool > rpool/ROOT              27.3G  80.2G       96K  /rpool/ROOT > rpool/ROOT/pbs-1        27.3G  80.2G     27.3G  / > > The VM I tested with is our Exchange Server. Raw image size 500GB, > netto ~400GB content > > First Test with one restore job: > > Virtual > Environment 7.0-11 > Datacenter > Search: > Logs > new > volume ID is 'VMSTORE:vm-101-disk-0' > restore > proxmox backup image: /usr/bin/pbs-restore --repository > root@pam@ph-pbs.peiker-holding.de:ZFSPOOLBACKUP > vm/121/2021-07-23T19:00:03Z drive-virtio0.img.fidx > /dev/zvol/ZFSPOOLVMSTORE/vm-101-disk-0 --verbose --format raw > --skip-zero > connecting > to repository 'root@pam@ph-pbs.peiker-holding.de:ZFSPOOLBACKUP' > open > block backend for target '/dev/zvol/ZFSPOOLVMSTORE/vm-101-disk-0' > starting > to restore snapshot 'vm/121/2021-07-23T19:00:03Z' > download > and verify backup index > progress > 1% (read 5368709120 bytes, zeroes = 2% (125829120 bytes), duration 86 > sec) > progress > 2% (read 10737418240 bytes, zeroes = 1% (159383552 bytes), duration > 181 sec) > progress > 3% (read 16106127360 bytes, zeroes = 0% (159383552 bytes), duration > 270 sec) > . > . > progress > 98% (read 526133493760 bytes, zeroes = 0% (3628072960 bytes), > duration 9492 sec) > progress > 99% (read 531502202880 bytes, zeroes = 0% (3628072960 bytes), > duration 9583 sec) > progress > 100% (read 536870912000 bytes, zeroes = 0% (3628072960 bytes), > duration 9676 sec) > restore > image complete (bytes=536870912000, duration=9676.97s, > speed=52.91MB/s) > rescan > volumes... > TASK > OK > > When I regard iotop I see about the same rate. > > But when I start multiple restore jobs parallel, I see that the single > jon is still on IO 40-50 MB/s but the total IO is multiple of the > rate. I see on iotop rates to 200-250 MB/s > So I guess it isn't the store. In some Test with an Setup where I used > the nvmes as source and target I could reach a singele restore rate > about 70 MB/s some disks/storages do not scale with single threaded workloads (and AFAIR, the pbs-restore must restore a disk single threaded because of qemu limitations?), but will scale with multiple threads just fine a 'fio' benchmark of the source as well as the target storage would be good to get a baseline storage perfomance see for example: https://pve.proxmox.com/wiki/Benchmarking_Storage > > Now I test an other CPU in this machine, cause on other test machines > with other CPU (AMD Ryzen or others) I get an higher rate. > Unfortunaly the rate on the current machine doesn't rise with the other > CPU can you do a proxmox-backup-client benchmark on all machines and their respective restore speed (especially the clients; also specify a repository to see tls speed) kind regards