Re: [PVE-User] proxmox-restore - performance issues

[Dominik Csapak <d.csapak@proxmox.com>]

On 10/1/21 11:00, Gregor Burck wrote:
> And more information:
> 
> I test an fio job, I got the settings not by my own instaed someone from 
> the forum give me this for testing ZFS:

is that the source as well as the target storage?
if not please benchmark both

> 
> 
> root@ph-pbs:~# fio --name=typical-vm --size=8G --rw=readwrite 
> --rwmixread=69 --direct=1 --bs=4K --numjobs=4 --ioengine=libaio 
> --iodepth=12 --group_reporting --runtime=20m --time_based^C
> root@ph-pbs:~# cd /ZFSPOOL/
> BACKUPSTORE001/ VMSTORE001/
> root@ph-pbs:~# cd /ZFSPOOL/VMSTORE001/
> root@ph-pbs:/ZFSPOOL/VMSTORE001# fio --name=typical-vm --size=8G 
> --rw=readwrite --rwmixread=69 --direct=1 --bs=4K --numjobs=4 
> --ioengine=libaio --iodepth=12 --group_reporting --runtime=20m --time_based
> typical-vm: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 
> 4096B-4096B, ioengine=libaio, iodepth=12
> ...
> fio-3.25
> Starting 4 processes
> typical-vm: Laying out IO file (1 file / 8192MiB)
> typical-vm: Laying out IO file (1 file / 8192MiB)
> typical-vm: Laying out IO file (1 file / 8192MiB)
> typical-vm: Laying out IO file (1 file / 8192MiB)
> Jobs: 4 (f=0): [f(4)][100.0%][r=1518MiB/s,w=682MiB/s][r=389k,w=175k 
> IOPS][eta 00m:00s]
> typical-vm: (groupid=0, jobs=4): err= 0: pid=3804786: Fri Oct  1 
> 10:56:30 2021
>    read: IOPS=356k, BW=1392MiB/s (1460MB/s)(1631GiB/1200001msec)

this looks too high for the storage array, so i guess something is
off with the benchmark (may be cache or missing filename parameter)
and the size is too little (i'd use something that cannot fit
into the cache)

in any case, i'd do read and write benchmarks seperately
as well as setting iodepth and numjobs to 1, to get a baseline
single thread performance

as i wrote in my previous message, check out examples at:
https://pve.proxmox.com/wiki/Benchmarking_Storage


>      slat (nsec): min=1854, max=176589k, avg=5156.08, stdev=39010.68
>      clat (usec): min=4, max=191637, avg=85.89, stdev=133.21
>       lat (usec): min=32, max=191640, avg=91.13, stdev=139.42
>      clat percentiles (usec):
>       |  1.00th=[   42],  5.00th=[   46], 10.00th=[   49], 20.00th=[   52],
>       | 30.00th=[   56], 40.00th=[   59], 50.00th=[   65], 60.00th=[   85],
>       | 70.00th=[   97], 80.00th=[  111], 90.00th=[  141], 95.00th=[  176],
>       | 99.00th=[  265], 99.50th=[  318], 99.90th=[  570], 99.95th=[  693],
>       | 99.99th=[ 1090]
>     bw (  MiB/s): min=  250, max= 2159, per=100.00%, avg=1392.77, 
> stdev=63.78, samples=9596
>     iops        : min=64218, max=552858, avg=356548.75, stdev=16328.20, 
> samples=9596
>    write: IOPS=160k, BW=626MiB/s (656MB/s)(733GiB/1200001msec); 0 zone 
> resets
>      slat (usec): min=3, max=191425, avg= 9.71, stdev=34.41
>      clat (usec): min=2, max=191641, avg=86.02, stdev=137.32
>       lat (usec): min=35, max=191650, avg=95.85, stdev=144.10
>      clat percentiles (usec):
>       |  1.00th=[   42],  5.00th=[   46], 10.00th=[   49], 20.00th=[   52],
>       | 30.00th=[   56], 40.00th=[   59], 50.00th=[   65], 60.00th=[   85],
>       | 70.00th=[   98], 80.00th=[  111], 90.00th=[  141], 95.00th=[  178],
>       | 99.00th=[  265], 99.50th=[  318], 99.90th=[  578], 99.95th=[  701],
>       | 99.99th=[ 1106]
>     bw (  KiB/s): min=114464, max=995856, per=100.00%, avg=640817.51, 
> stdev=29342.79, samples=9596
>     iops        : min=28616, max=248964, avg=160204.26, stdev=7335.70, 
> samples=9596
>    lat (usec)   : 4=0.01%, 10=0.01%, 50=13.69%, 100=58.80%, 250=26.29%
>    lat (usec)   : 500=1.08%, 750=0.10%, 1000=0.02%
>    lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
>    lat (msec)   : 100=0.01%, 250=0.01%
>    cpu          : usr=18.17%, sys=79.17%, ctx=982498, majf=10, minf=2977
>    IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, 
>  >=64=0.0%
>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>  >=64=0.0%
>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, 
>  >=64=0.0%
>       issued rwts: total=427672030,192161509,0,0 short=0,0,0,0 
> dropped=0,0,0,0
>       latency   : target=0, window=0, percentile=100.00%, depth=12
> 
> Run status group 0 (all jobs):
>     READ: bw=1392MiB/s (1460MB/s), 1392MiB/s-1392MiB/s 
> (1460MB/s-1460MB/s), io=1631GiB (1752GB), run=1200001-1200001msec
>    WRITE: bw=626MiB/s (656MB/s), 626MiB/s-626MiB/s (656MB/s-656MB/s), 
> io=733GiB (787GB), run=1200001-1200001msec
> 
> 
> And this is while two of the restore jobs still running.
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 
>