From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <d.csapak@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 604007107C for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 11:30:09 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 4B3D729B78 for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 11:29:39 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 3E10929B6D for <pve-user@lists.proxmox.com>; Fri, 1 Oct 2021 11:29:38 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0A09A453CB; Fri, 1 Oct 2021 11:29:38 +0200 (CEST) Message-ID: <453ff6f1-081e-72e4-1325-a78b46934240@proxmox.com> Date: Fri, 1 Oct 2021 11:29:36 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:93.0) Gecko/20100101 Thunderbird/93.0 Content-Language: en-US To: Proxmox VE user list <pve-user@lists.proxmox.com>, Gregor Burck <gregor@aeppelbroe.de> References: <20211001091851.EGroupware.HVxe1rjAizNRB0o9XSoY2QH@heim.aeppelbroe.de> <20211001110037.EGroupware.Ez22YX0Ys-I09F4bK5VPukF@heim.aeppelbroe.de> From: Dominik Csapak <d.csapak@proxmox.com> In-Reply-To: <20211001110037.EGroupware.Ez22YX0Ys-I09F4bK5VPukF@heim.aeppelbroe.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 1.921 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -3.499 Looks like a legit reply (A) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] proxmox-restore - performance issues X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Fri, 01 Oct 2021 09:30:09 -0000 On 10/1/21 11:00, Gregor Burck wrote: > And more information: > > I test an fio job, I got the settings not by my own instaed someone from > the forum give me this for testing ZFS: is that the source as well as the target storage? if not please benchmark both > > > root@ph-pbs:~# fio --name=typical-vm --size=8G --rw=readwrite > --rwmixread=69 --direct=1 --bs=4K --numjobs=4 --ioengine=libaio > --iodepth=12 --group_reporting --runtime=20m --time_based^C > root@ph-pbs:~# cd /ZFSPOOL/ > BACKUPSTORE001/ VMSTORE001/ > root@ph-pbs:~# cd /ZFSPOOL/VMSTORE001/ > root@ph-pbs:/ZFSPOOL/VMSTORE001# fio --name=typical-vm --size=8G > --rw=readwrite --rwmixread=69 --direct=1 --bs=4K --numjobs=4 > --ioengine=libaio --iodepth=12 --group_reporting --runtime=20m --time_based > typical-vm: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) > 4096B-4096B, ioengine=libaio, iodepth=12 > ... > fio-3.25 > Starting 4 processes > typical-vm: Laying out IO file (1 file / 8192MiB) > typical-vm: Laying out IO file (1 file / 8192MiB) > typical-vm: Laying out IO file (1 file / 8192MiB) > typical-vm: Laying out IO file (1 file / 8192MiB) > Jobs: 4 (f=0): [f(4)][100.0%][r=1518MiB/s,w=682MiB/s][r=389k,w=175k > IOPS][eta 00m:00s] > typical-vm: (groupid=0, jobs=4): err= 0: pid=3804786: Fri Oct 1 > 10:56:30 2021 > read: IOPS=356k, BW=1392MiB/s (1460MB/s)(1631GiB/1200001msec) this looks too high for the storage array, so i guess something is off with the benchmark (may be cache or missing filename parameter) and the size is too little (i'd use something that cannot fit into the cache) in any case, i'd do read and write benchmarks seperately as well as setting iodepth and numjobs to 1, to get a baseline single thread performance as i wrote in my previous message, check out examples at: https://pve.proxmox.com/wiki/Benchmarking_Storage > slat (nsec): min=1854, max=176589k, avg=5156.08, stdev=39010.68 > clat (usec): min=4, max=191637, avg=85.89, stdev=133.21 > lat (usec): min=32, max=191640, avg=91.13, stdev=139.42 > clat percentiles (usec): > | 1.00th=[ 42], 5.00th=[ 46], 10.00th=[ 49], 20.00th=[ 52], > | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 65], 60.00th=[ 85], > | 70.00th=[ 97], 80.00th=[ 111], 90.00th=[ 141], 95.00th=[ 176], > | 99.00th=[ 265], 99.50th=[ 318], 99.90th=[ 570], 99.95th=[ 693], > | 99.99th=[ 1090] > bw ( MiB/s): min= 250, max= 2159, per=100.00%, avg=1392.77, > stdev=63.78, samples=9596 > iops : min=64218, max=552858, avg=356548.75, stdev=16328.20, > samples=9596 > write: IOPS=160k, BW=626MiB/s (656MB/s)(733GiB/1200001msec); 0 zone > resets > slat (usec): min=3, max=191425, avg= 9.71, stdev=34.41 > clat (usec): min=2, max=191641, avg=86.02, stdev=137.32 > lat (usec): min=35, max=191650, avg=95.85, stdev=144.10 > clat percentiles (usec): > | 1.00th=[ 42], 5.00th=[ 46], 10.00th=[ 49], 20.00th=[ 52], > | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 65], 60.00th=[ 85], > | 70.00th=[ 98], 80.00th=[ 111], 90.00th=[ 141], 95.00th=[ 178], > | 99.00th=[ 265], 99.50th=[ 318], 99.90th=[ 578], 99.95th=[ 701], > | 99.99th=[ 1106] > bw ( KiB/s): min=114464, max=995856, per=100.00%, avg=640817.51, > stdev=29342.79, samples=9596 > iops : min=28616, max=248964, avg=160204.26, stdev=7335.70, > samples=9596 > lat (usec) : 4=0.01%, 10=0.01%, 50=13.69%, 100=58.80%, 250=26.29% > lat (usec) : 500=1.08%, 750=0.10%, 1000=0.02% > lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% > lat (msec) : 100=0.01%, 250=0.01% > cpu : usr=18.17%, sys=79.17%, ctx=982498, majf=10, minf=2977 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, > >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, > >=64=0.0% > issued rwts: total=427672030,192161509,0,0 short=0,0,0,0 > dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=12 > > Run status group 0 (all jobs): > READ: bw=1392MiB/s (1460MB/s), 1392MiB/s-1392MiB/s > (1460MB/s-1460MB/s), io=1631GiB (1752GB), run=1200001-1200001msec > WRITE: bw=626MiB/s (656MB/s), 626MiB/s-626MiB/s (656MB/s-656MB/s), > io=733GiB (787GB), run=1200001-1200001msec > > > And this is while two of the restore jobs still running. > > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >