Re: [PVE-User] NFS Backups

["Сергей Цаболов" <tsabolov@t8.ru>]

[-- Attachment #1: Type: text/plain, Size: 3824 bytes --]

Hello,

20.07.2022 20:13, GM пишет:
> Have you tried to copy a large file directly from the host(s) to the nf
> server, how's the speed on that?
Yes, I try copy large file iso with rsync and the speed is no  more than 
180MB/s - 210MB/s
> Have you checked if there's a bandwidth limit set for the backups in pve
> datacenter options?
Yes, no limits Bandwidth set in Options.
> Try running a single, manual backup on a specific vm to the nfs server,
> how's its speed compare to the scheduled backups?

Yes, in GUI I try the start one backup, the speed when start is up after 
is falling to same speed:

In attach is from one host when is start backup and show speed to Synology.

> On Wed, 20 Jul 2022 at 10:12, Сергей Цаболов<tsabolov@t8.ru>  wrote:
>
>> Hi, to all.
>>
>> In our PVE cluster have 5 hosts with ceph storage on all hosts disks.
>>
>> I setup backups to save on Synology with NFS storage.
>>
>> For 2-3 moths right now backups load very long time and not fast to
>> finish backup from all hosts.
>>
>> If we change the MTU from default 1500 on Synology and on hosts to 9000
>> MTU is possible to make it fast.
>>
>> Synology and the hosts connect to the Fiber Optic 10GB switch but the
>> mtu on switch and on the hosts for now is default.
>>
>> If I ping to Synology from hosts :
>>
>> ping -s 9000 -d 192.168.8.23
>> PING 192.168.8.23 (192.168.8.23) 9000(9028) bytes of data.
>> 9008 bytes from 192.168.8.23: icmp_seq=1 ttl=63 time=1.03 ms
>> 9008 bytes from 192.168.8.23: icmp_seq=2 ttl=63 time=0.223 ms
>> 9008 bytes from 192.168.8.23: icmp_seq=3 ttl=63 time=0.229 ms
>>
>> I have the answer.
>>
>> When I test connection with between the hosts PVE connect :
>>
>> iperf3 -c 10.10.29.16
>> Connecting to host 10.10.29.16, port 5201
>> [  5] local 10.10.29.10 port 51502 connected to 10.10.29.16 port 5201
>> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
>> [  5]   0.00-1.00   sec  1.07 GBytes  9.19 Gbits/sec  1089    639 KBytes
>> [  5]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec   52   1.17 MBytes
>> [  5]   2.00-3.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.23 MBytes
>> [  5]   3.00-4.00   sec  1.09 GBytes  9.37 Gbits/sec    0   1.27 MBytes
>> [  5]   4.00-5.00   sec  1.07 GBytes  9.20 Gbits/sec  608    789 KBytes
>> [  5]   5.00-6.00   sec  1.09 GBytes  9.38 Gbits/sec   45   1.09 MBytes
>> [  5]   6.00-7.00   sec  1.09 GBytes  9.37 Gbits/sec    0   1.26 MBytes
>> [  5]   7.00-8.00   sec  1.08 GBytes  9.28 Gbits/sec  411    872 KBytes
>> [  5]   8.00-9.00   sec  1.09 GBytes  9.33 Gbits/sec  114   1.14 MBytes
>> [  5]   9.00-10.00  sec  1.08 GBytes  9.30 Gbits/sec  321    676 KBytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval           Transfer     Bitrate         Retr
>> [  5]   0.00-10.00  sec  10.8 GBytes  9.32 Gbits/sec 2640
>> sender
>> [  5]   0.00-10.00  sec  10.8 GBytes  9.31 Gbits/sec
>> receiver
>>
>> But the speed to backup to NFS is very low.
>>
>> Can some one give me suggestion  how I can make faster the NFS backup on
>> hosts (all hosts is now production).
>>
>> --
>> Sergey TS
>> The best Regard
>>
>> _______________________________________________
>> pve-user mailing list
>> pve-user@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> _______________________________________________
>> pve-user mailing list
>> pve-user@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

-- 
Sergey TS
The best Regard

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[-- Attachment #2: backup job: vzdump 111 106 124 129.txt --]
[-- Type: text/plain, Size: 32586 bytes --]

Proxmox
Virtual Environment 6.4-13
Node 't8ntc-3103'
Status


INFO: starting new backup job: vzdump 111 106 124 129 --compress zstd --storage nfs-synology --node pve-3111 --mailnotification always --quiet 1 --mailto support@t8.ru --mode snapshot
INFO: Starting Backup of VM 106 (qemu)
INFO: Backup started at 2022-06-19 22:15:01
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: prometheus
INFO: include disk 'virtio0' 'vm.pool:vm-106-disk-1' 82G
/dev/rbd4
INFO: creating vzdump archive '/mnt/pve/nfs-synology/dump/vzdump-qemu-106-2022_06_19-22_15_01.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '9780e427-5a0f-4e54-aade-2c0b88ba6d09'
INFO:   0% (45.5 MiB of 82.0 GiB) in 3s, read: 15.2 MiB/s, write: 3.9 MiB/s
INFO:   1% (899.0 MiB of 82.0 GiB) in 1m, read: 15.0 MiB/s, write: 8.6 MiB/s
INFO:   2% (1.7 GiB of 82.0 GiB) in 1m 7s, read: 115.0 MiB/s, write: 2.5 MiB/s
INFO:   3% (2.5 GiB of 82.0 GiB) in 2m 55s, read: 7.6 MiB/s, write: 1.5 MiB/s
INFO:   4% (3.3 GiB of 82.0 GiB) in 4m 31s, read: 8.7 MiB/s, write: 58.2 KiB/s
INFO:   5% (4.1 GiB of 82.0 GiB) in 6m 1s, read: 9.7 MiB/s, write: 133.2 KiB/s
INFO:   6% (4.9 GiB of 82.0 GiB) in 7m 16s, read: 10.8 MiB/s, write: 442.9 KiB/s
INFO:   7% (5.7 GiB of 82.0 GiB) in 8m 39s, read: 10.1 MiB/s, write: 0 B/s
INFO:   8% (6.6 GiB of 82.0 GiB) in 9m 50s, read: 12.2 MiB/s, write: 1.7 MiB/s
INFO:   9% (7.6 GiB of 82.0 GiB) in 11m 1s, read: 14.2 MiB/s, write: 6.7 MiB/s
INFO:  10% (8.2 GiB of 82.0 GiB) in 11m 9s, read: 86.0 MiB/s, write: 12.9 MiB/s
INFO:  11% (9.0 GiB of 82.0 GiB) in 11m 22s, read: 60.7 MiB/s, write: 8.5 MiB/s
INFO:  12% (9.9 GiB of 82.0 GiB) in 11m 33s, read: 81.1 MiB/s, write: 2.8 MiB/s
INFO:  13% (11.0 GiB of 82.0 GiB) in 12m 15s, read: 26.3 MiB/s, write: 4.2 MiB/s
INFO:  14% (11.8 GiB of 82.0 GiB) in 12m 31s, read: 51.1 MiB/s, write: 6.8 MiB/s
INFO:  15% (12.6 GiB of 82.0 GiB) in 13m 10s, read: 22.6 MiB/s, write: 4.0 MiB/s
INFO:  16% (13.3 GiB of 82.0 GiB) in 13m 24s, read: 46.0 MiB/s, write: 3.2 MiB/s
INFO:  17% (14.0 GiB of 82.0 GiB) in 14m 17s, read: 13.5 MiB/s, write: 6.1 MiB/s
INFO:  18% (14.8 GiB of 82.0 GiB) in 15m 29s, read: 11.7 MiB/s, write: 6.7 MiB/s
INFO:  19% (15.6 GiB of 82.0 GiB) in 16m 2s, read: 25.8 MiB/s, write: 5.2 MiB/s
INFO:  20% (16.4 GiB of 82.0 GiB) in 17m 39s, read: 8.5 MiB/s, write: 2.8 MiB/s
INFO:  21% (17.2 GiB of 82.0 GiB) in 19m 13s, read: 8.9 MiB/s, write: 5.1 MiB/s
INFO:  22% (18.1 GiB of 82.0 GiB) in 20m 18s, read: 13.2 MiB/s, write: 6.4 MiB/s
INFO:  23% (18.9 GiB of 82.0 GiB) in 20m 41s, read: 37.2 MiB/s, write: 5.4 MiB/s
INFO:  24% (19.7 GiB of 82.0 GiB) in 20m 45s, read: 209.9 MiB/s, write: 8.3 MiB/s
INFO:  25% (20.5 GiB of 82.0 GiB) in 21m 4s, read: 42.8 MiB/s, write: 6.0 MiB/s
INFO:  26% (21.5 GiB of 82.0 GiB) in 21m 14s, read: 98.3 MiB/s, write: 4.9 MiB/s
INFO:  27% (22.2 GiB of 82.0 GiB) in 21m 33s, read: 37.3 MiB/s, write: 4.3 MiB/s
INFO:  28% (23.1 GiB of 82.0 GiB) in 21m 58s, read: 38.7 MiB/s, write: 5.6 MiB/s
INFO:  29% (23.9 GiB of 82.0 GiB) in 22m 21s, read: 34.2 MiB/s, write: 4.1 MiB/s
INFO:  30% (24.8 GiB of 82.0 GiB) in 22m 37s, read: 56.2 MiB/s, write: 2.5 MiB/s
INFO:  31% (25.5 GiB of 82.0 GiB) in 23m 27s, read: 15.4 MiB/s, write: 4.7 MiB/s
INFO:  32% (26.2 GiB of 82.0 GiB) in 23m 43s, read: 46.9 MiB/s, write: 6.4 MiB/s
INFO:  33% (27.6 GiB of 82.0 GiB) in 23m 51s, read: 178.5 MiB/s, write: 6.4 MiB/s
INFO:  34% (28.0 GiB of 82.0 GiB) in 23m 54s, read: 129.9 MiB/s, write: 8.6 MiB/s
INFO:  35% (28.8 GiB of 82.0 GiB) in 24m 4s, read: 76.7 MiB/s, write: 7.5 MiB/s
INFO:  36% (30.1 GiB of 82.0 GiB) in 24m 7s, read: 464.0 MiB/s, write: 6.3 MiB/s
INFO:  37% (30.5 GiB of 82.0 GiB) in 24m 11s, read: 100.2 MiB/s, write: 1.0 MiB/s
INFO:  38% (31.6 GiB of 82.0 GiB) in 24m 17s, read: 190.5 MiB/s, write: 4.9 MiB/s
INFO:  39% (32.7 GiB of 82.0 GiB) in 24m 21s, read: 280.8 MiB/s, write: 7.0 MiB/s
INFO:  43% (35.8 GiB of 82.0 GiB) in 24m 24s, read: 1.0 GiB/s, write: 0 B/s
INFO:  47% (38.7 GiB of 82.0 GiB) in 24m 27s, read: 1017.8 MiB/s, write: 0 B/s
INFO:  50% (41.5 GiB of 82.0 GiB) in 24m 30s, read: 950.5 MiB/s, write: 0 B/s
INFO:  54% (44.5 GiB of 82.0 GiB) in 24m 33s, read: 1007.2 MiB/s, write: 0 B/s
INFO:  57% (47.4 GiB of 82.0 GiB) in 24m 36s, read: 1014.1 MiB/s, write: 0 B/s
INFO:  61% (50.4 GiB of 82.0 GiB) in 24m 39s, read: 1006.0 MiB/s, write: 0 B/s
INFO:  64% (53.3 GiB of 82.0 GiB) in 24m 42s, read: 986.6 MiB/s, write: 0 B/s
INFO:  68% (56.0 GiB of 82.0 GiB) in 24m 45s, read: 910.3 MiB/s, write: 0 B/s
INFO:  71% (58.6 GiB of 82.0 GiB) in 24m 48s, read: 903.4 MiB/s, write: 0 B/s
INFO:  75% (61.6 GiB of 82.0 GiB) in 24m 51s, read: 1.0 GiB/s, write: 0 B/s
INFO:  78% (64.6 GiB of 82.0 GiB) in 24m 54s, read: 1.0 GiB/s, write: 0 B/s
INFO:  82% (67.3 GiB of 82.0 GiB) in 24m 57s, read: 918.0 MiB/s, write: 0 B/s
INFO:  85% (69.9 GiB of 82.0 GiB) in 25m, read: 900.2 MiB/s, write: 0 B/s
INFO:  87% (71.4 GiB of 82.0 GiB) in 25m 3s, read: 487.0 MiB/s, write: 0 B/s
INFO:  90% (73.8 GiB of 82.0 GiB) in 25m 6s, read: 841.6 MiB/s, write: 0 B/s
INFO:  93% (76.5 GiB of 82.0 GiB) in 25m 9s, read: 910.6 MiB/s, write: 0 B/s
INFO:  96% (79.4 GiB of 82.0 GiB) in 25m 12s, read: 1002.1 MiB/s, write: 0 B/s
INFO: 100% (82.0 GiB of 82.0 GiB) in 25m 15s, read: 871.9 MiB/s, write: 2.7 KiB/s
INFO: backup is sparse: 76.67 GiB (93%) total zero data
INFO: transferred 82.00 GiB in 1515 seconds (55.4 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 2.15GB
INFO: removing backup 'nfs-synology:backup/vzdump-qemu-106-2022_06_17-22_15_02.vma.zst'
INFO: Finished Backup of VM 106 (00:25:22)
INFO: Backup finished at 2022-06-19 22:40:23
INFO: Starting Backup of VM 111 (qemu)
INFO: Backup started at 2022-06-19 22:40:23
INFO: status = running
INFO: VM Name: rnd-sborka2
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/nfs-synology/dump/vzdump-qemu-111-2022_06_19-22_40_23.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'a1d3a5a9-6d02-4f06-8dbb-b328548bb55a'
INFO: resuming VM again
INFO:   0% (66.1 MiB of 200.0 GiB) in 3s, read: 22.0 MiB/s, write: 12.8 MiB/s
INFO:   1% (2.0 GiB of 200.0 GiB) in 6m 55s, read: 4.8 MiB/s, write: 4.7 MiB/s
INFO:   2% (4.0 GiB of 200.0 GiB) in 13m 48s, read: 5.0 MiB/s, write: 4.9 MiB/s
INFO:   3% (6.0 GiB of 200.0 GiB) in 20m 36s, read: 5.0 MiB/s, write: 5.0 MiB/s
INFO:   4% (8.0 GiB of 200.0 GiB) in 28m 20s, read: 4.4 MiB/s, write: 4.4 MiB/s
INFO:   5% (10.0 GiB of 200.0 GiB) in 35m 24s, read: 4.8 MiB/s, write: 4.7 MiB/s
INFO:   6% (12.0 GiB of 200.0 GiB) in 40m 12s, read: 7.1 MiB/s, write: 4.5 MiB/s
INFO:   7% (14.0 GiB of 200.0 GiB) in 43m 18s, read: 11.1 MiB/s, write: 9.9 MiB/s
INFO:   8% (16.1 GiB of 200.0 GiB) in 43m 42s, read: 88.9 MiB/s, write: 13.7 MiB/s
INFO:   9% (18.1 GiB of 200.0 GiB) in 43m 47s, read: 399.9 MiB/s, write: 0 B/s
INFO:  10% (20.0 GiB of 200.0 GiB) in 46m 3s, read: 14.7 MiB/s, write: 6.4 MiB/s
INFO:  11% (22.0 GiB of 200.0 GiB) in 52m 4s, read: 5.7 MiB/s, write: 5.6 MiB/s
INFO:  12% (24.0 GiB of 200.0 GiB) in 57m 56s, read: 5.8 MiB/s, write: 5.8 MiB/s
INFO:  13% (26.0 GiB of 200.0 GiB) in 1h 3m 50s, read: 5.8 MiB/s, write: 5.8 MiB/s
INFO:  14% (28.0 GiB of 200.0 GiB) in 1h 8m 47s, read: 6.9 MiB/s, write: 6.9 MiB/s
INFO:  15% (30.0 GiB of 200.0 GiB) in 1h 12m 38s, read: 8.9 MiB/s, write: 8.9 MiB/s
INFO:  16% (32.0 GiB of 200.0 GiB) in 1h 18m 8s, read: 6.2 MiB/s, write: 6.2 MiB/s
INFO:  17% (34.0 GiB of 200.0 GiB) in 1h 25m 23s, read: 4.7 MiB/s, write: 4.6 MiB/s
INFO:  18% (36.0 GiB of 200.0 GiB) in 1h 31m 19s, read: 5.8 MiB/s, write: 5.7 MiB/s
INFO:  19% (38.0 GiB of 200.0 GiB) in 1h 37m 51s, read: 5.2 MiB/s, write: 5.2 MiB/s
INFO:  20% (40.0 GiB of 200.0 GiB) in 1h 43m 16s, read: 6.3 MiB/s, write: 6.2 MiB/s
INFO:  21% (42.0 GiB of 200.0 GiB) in 1h 47m 3s, read: 9.1 MiB/s, write: 9.0 MiB/s
INFO:  22% (44.0 GiB of 200.0 GiB) in 1h 52m 47s, read: 5.9 MiB/s, write: 5.9 MiB/s
INFO:  23% (46.0 GiB of 200.0 GiB) in 1h 57m 58s, read: 6.6 MiB/s, write: 6.6 MiB/s
INFO:  24% (48.0 GiB of 200.0 GiB) in 2h 3m 43s, read: 5.9 MiB/s, write: 5.9 MiB/s
INFO:  25% (50.0 GiB of 200.0 GiB) in 2h 9m 8s, read: 6.3 MiB/s, write: 6.3 MiB/s
INFO:  26% (52.0 GiB of 200.0 GiB) in 2h 14m 4s, read: 7.0 MiB/s, write: 6.9 MiB/s
INFO:  27% (54.0 GiB of 200.0 GiB) in 2h 19m 43s, read: 6.0 MiB/s, write: 5.9 MiB/s
INFO:  28% (56.0 GiB of 200.0 GiB) in 2h 25m 43s, read: 5.7 MiB/s, write: 5.7 MiB/s
INFO:  29% (58.0 GiB of 200.0 GiB) in 2h 30m 45s, read: 6.8 MiB/s, write: 6.7 MiB/s
INFO:  30% (60.0 GiB of 200.0 GiB) in 2h 36m 26s, read: 6.0 MiB/s, write: 6.0 MiB/s
INFO:  31% (62.2 GiB of 200.0 GiB) in 2h 39m 11s, read: 13.6 MiB/s, write: 6.4 MiB/s
INFO:  32% (64.0 GiB of 200.0 GiB) in 2h 41m 2s, read: 16.6 MiB/s, write: 6.6 MiB/s
INFO:  33% (66.0 GiB of 200.0 GiB) in 2h 44m 34s, read: 9.7 MiB/s, write: 5.9 MiB/s
INFO:  34% (68.0 GiB of 200.0 GiB) in 2h 47m 43s, read: 10.9 MiB/s, write: 8.6 MiB/s
INFO:  35% (70.0 GiB of 200.0 GiB) in 2h 49m 43s, read: 17.0 MiB/s, write: 7.4 MiB/s
INFO:  36% (72.0 GiB of 200.0 GiB) in 2h 52m 2s, read: 14.8 MiB/s, write: 5.5 MiB/s
INFO:  37% (74.1 GiB of 200.0 GiB) in 2h 54m 23s, read: 15.2 MiB/s, write: 4.9 MiB/s
INFO:  38% (76.0 GiB of 200.0 GiB) in 2h 57m 16s, read: 11.2 MiB/s, write: 6.9 MiB/s
INFO:  39% (78.0 GiB of 200.0 GiB) in 3h 1m 43s, read: 7.7 MiB/s, write: 5.7 MiB/s
INFO:  40% (80.0 GiB of 200.0 GiB) in 3h 7m 46s, read: 5.7 MiB/s, write: 5.4 MiB/s
INFO:  41% (82.2 GiB of 200.0 GiB) in 3h 9m 27s, read: 22.7 MiB/s, write: 8.3 MiB/s
INFO:  42% (84.0 GiB of 200.0 GiB) in 3h 10m 30s, read: 28.7 MiB/s, write: 7.9 MiB/s
INFO:  43% (86.0 GiB of 200.0 GiB) in 3h 13m 58s, read: 9.9 MiB/s, write: 8.4 MiB/s
INFO:  44% (88.0 GiB of 200.0 GiB) in 3h 16m 24s, read: 14.1 MiB/s, write: 4.9 MiB/s
INFO:  45% (90.0 GiB of 200.0 GiB) in 3h 17m 36s, read: 28.3 MiB/s, write: 5.6 MiB/s
INFO:  46% (92.0 GiB of 200.0 GiB) in 3h 22m 16s, read: 7.3 MiB/s, write: 5.6 MiB/s
INFO:  47% (94.1 GiB of 200.0 GiB) in 3h 25m 6s, read: 12.8 MiB/s, write: 5.6 MiB/s
INFO:  48% (96.0 GiB of 200.0 GiB) in 3h 28m 3s, read: 10.9 MiB/s, write: 5.0 MiB/s
INFO:  49% (98.0 GiB of 200.0 GiB) in 3h 34m 5s, read: 5.7 MiB/s, write: 5.5 MiB/s
INFO:  50% (100.1 GiB of 200.0 GiB) in 3h 35m 57s, read: 19.6 MiB/s, write: 2.9 MiB/s
INFO:  51% (102.2 GiB of 200.0 GiB) in 3h 36m 33s, read: 57.6 MiB/s, write: 5.1 MiB/s
INFO:  52% (104.0 GiB of 200.0 GiB) in 3h 37m 6s, read: 56.8 MiB/s, write: 7.5 MiB/s
INFO:  53% (106.1 GiB of 200.0 GiB) in 3h 37m 33s, read: 78.4 MiB/s, write: 5.4 MiB/s
INFO:  54% (108.0 GiB of 200.0 GiB) in 3h 40m 4s, read: 13.2 MiB/s, write: 6.9 MiB/s
INFO:  55% (110.0 GiB of 200.0 GiB) in 3h 42m 25s, read: 14.7 MiB/s, write: 6.7 MiB/s
INFO:  56% (112.0 GiB of 200.0 GiB) in 3h 47m 27s, read: 6.6 MiB/s, write: 5.4 MiB/s
INFO:  57% (114.1 GiB of 200.0 GiB) in 3h 50m 18s, read: 12.3 MiB/s, write: 5.4 MiB/s
INFO:  58% (116.0 GiB of 200.0 GiB) in 3h 50m 32s, read: 142.8 MiB/s, write: 6.5 MiB/s
INFO:  59% (118.0 GiB of 200.0 GiB) in 3h 52m 1s, read: 22.9 MiB/s, write: 4.3 MiB/s
INFO:  60% (120.0 GiB of 200.0 GiB) in 3h 53m 40s, read: 20.7 MiB/s, write: 6.4 MiB/s
INFO:  61% (122.1 GiB of 200.0 GiB) in 3h 55m 7s, read: 24.5 MiB/s, write: 7.2 MiB/s
INFO:  62% (124.0 GiB of 200.0 GiB) in 3h 56m 1s, read: 36.4 MiB/s, write: 6.5 MiB/s
INFO:  63% (126.1 GiB of 200.0 GiB) in 3h 56m 52s, read: 42.3 MiB/s, write: 4.9 MiB/s
INFO:  64% (128.0 GiB of 200.0 GiB) in 3h 58m 44s, read: 17.3 MiB/s, write: 6.7 MiB/s
INFO:  65% (130.3 GiB of 200.0 GiB) in 4h 2m 12s, read: 11.4 MiB/s, write: 6.2 MiB/s
INFO:  66% (132.0 GiB of 200.0 GiB) in 4h 4m 12s, read: 14.4 MiB/s, write: 4.8 MiB/s
INFO:  67% (134.0 GiB of 200.0 GiB) in 4h 7m 34s, read: 10.1 MiB/s, write: 5.6 MiB/s
INFO:  68% (136.1 GiB of 200.0 GiB) in 4h 11m 26s, read: 9.4 MiB/s, write: 5.9 MiB/s
INFO:  69% (138.0 GiB of 200.0 GiB) in 4h 14m 33s, read: 10.2 MiB/s, write: 5.0 MiB/s
INFO:  70% (140.3 GiB of 200.0 GiB) in 4h 15m 25s, read: 44.4 MiB/s, write: 6.1 MiB/s
INFO:  71% (142.2 GiB of 200.0 GiB) in 4h 17m 25s, read: 16.3 MiB/s, write: 3.5 MiB/s
INFO:  72% (144.0 GiB of 200.0 GiB) in 4h 18m 57s, read: 20.5 MiB/s, write: 4.8 MiB/s
INFO:  73% (146.0 GiB of 200.0 GiB) in 4h 22m 55s, read: 8.7 MiB/s, write: 6.0 MiB/s
INFO:  74% (148.2 GiB of 200.0 GiB) in 4h 27m 51s, read: 7.6 MiB/s, write: 6.2 MiB/s
INFO:  75% (150.1 GiB of 200.0 GiB) in 4h 31m 2s, read: 10.3 MiB/s, write: 4.3 MiB/s
INFO:  76% (152.0 GiB of 200.0 GiB) in 4h 35m 52s, read: 6.6 MiB/s, write: 5.2 MiB/s
INFO:  77% (154.0 GiB of 200.0 GiB) in 4h 38m 58s, read: 11.0 MiB/s, write: 5.7 MiB/s
INFO:  78% (156.0 GiB of 200.0 GiB) in 4h 39m 49s, read: 40.0 MiB/s, write: 6.3 MiB/s
INFO:  79% (158.0 GiB of 200.0 GiB) in 4h 42m 38s, read: 12.1 MiB/s, write: 7.5 MiB/s
INFO:  80% (160.0 GiB of 200.0 GiB) in 4h 45m 23s, read: 12.4 MiB/s, write: 7.2 MiB/s
INFO:  81% (162.0 GiB of 200.0 GiB) in 4h 47m 8s, read: 19.8 MiB/s, write: 6.3 MiB/s
INFO:  82% (164.3 GiB of 200.0 GiB) in 4h 50m 17s, read: 12.3 MiB/s, write: 7.1 MiB/s
INFO:  83% (166.0 GiB of 200.0 GiB) in 4h 52m 22s, read: 13.9 MiB/s, write: 6.0 MiB/s
INFO:  84% (168.2 GiB of 200.0 GiB) in 4h 54m 38s, read: 16.7 MiB/s, write: 7.3 MiB/s
INFO:  85% (170.2 GiB of 200.0 GiB) in 4h 56m 47s, read: 15.8 MiB/s, write: 6.3 MiB/s
INFO:  86% (172.3 GiB of 200.0 GiB) in 4h 57m 24s, read: 59.0 MiB/s, write: 4.5 MiB/s
INFO:  87% (174.0 GiB of 200.0 GiB) in 4h 58m 42s, read: 21.9 MiB/s, write: 10.0 MiB/s
INFO:  88% (176.0 GiB of 200.0 GiB) in 5h 17s, read: 21.6 MiB/s, write: 12.4 MiB/s
INFO:  89% (178.0 GiB of 200.0 GiB) in 5h 3m 27s, read: 10.8 MiB/s, write: 10.8 MiB/s
INFO:  90% (180.2 GiB of 200.0 GiB) in 5h 6m 10s, read: 13.9 MiB/s, write: 8.7 MiB/s
INFO:  91% (182.0 GiB of 200.0 GiB) in 5h 8m 22s, read: 13.9 MiB/s, write: 10.0 MiB/s
INFO:  92% (184.0 GiB of 200.0 GiB) in 5h 9m 13s, read: 40.1 MiB/s, write: 8.8 MiB/s
INFO:  93% (186.0 GiB of 200.0 GiB) in 5h 10m 54s, read: 20.2 MiB/s, write: 9.7 MiB/s
INFO:  94% (188.0 GiB of 200.0 GiB) in 5h 12m 20s, read: 23.9 MiB/s, write: 8.6 MiB/s
INFO:  95% (190.0 GiB of 200.0 GiB) in 5h 14m 49s, read: 13.8 MiB/s, write: 11.3 MiB/s
INFO:  96% (192.0 GiB of 200.0 GiB) in 5h 16m 30s, read: 20.2 MiB/s, write: 9.6 MiB/s
INFO:  97% (194.2 GiB of 200.0 GiB) in 5h 19m 12s, read: 14.0 MiB/s, write: 8.0 MiB/s
INFO:  98% (196.2 GiB of 200.0 GiB) in 5h 22m 9s, read: 11.7 MiB/s, write: 7.2 MiB/s
INFO:  99% (198.1 GiB of 200.0 GiB) in 5h 23m 43s, read: 20.8 MiB/s, write: 9.7 MiB/s
INFO: 100% (200.0 GiB of 200.0 GiB) in 5h 24m 27s, read: 43.3 MiB/s, write: 11.0 MiB/s
INFO: backup is sparse: 81.68 GiB (40%) total zero data
INFO: transferred 200.00 GiB in 19467 seconds (10.5 MiB/s)
INFO: archive file size: 53.42GB
INFO: removing backup 'nfs-synology:backup/vzdump-qemu-111-2022_06_17-22_35_39.vma.zst'
INFO: Finished Backup of VM 111 (05:25:55)
INFO: Backup finished at 2022-06-20 04:06:18
INFO: Starting Backup of VM 124 (qemu)
INFO: Backup started at 2022-06-20 04:06:18
INFO: status = running
INFO: VM Name: nms-test3
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/nfs-synology/dump/vzdump-qemu-124-2022_06_20-04_06_18.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '56fb0ace-97db-4fe3-97b0-df3e605688c5'
INFO: resuming VM again
INFO:   0% (61.2 MiB of 240.0 GiB) in 3s, read: 20.4 MiB/s, write: 11.5 MiB/s
INFO:   1% (2.5 GiB of 240.0 GiB) in 1m 12s, read: 35.7 MiB/s, write: 4.7 MiB/s
INFO:   2% (4.9 GiB of 240.0 GiB) in 2m 2s, read: 50.0 MiB/s, write: 6.5 MiB/s
INFO:   3% (7.3 GiB of 240.0 GiB) in 2m 37s, read: 69.5 MiB/s, write: 679.5 KiB/s
INFO:   4% (9.6 GiB of 240.0 GiB) in 3m 8s, read: 78.0 MiB/s, write: 2.3 MiB/s
INFO:   5% (12.1 GiB of 240.0 GiB) in 4m, read: 47.6 MiB/s, write: 3.7 MiB/s
INFO:   6% (14.4 GiB of 240.0 GiB) in 4m 49s, read: 49.8 MiB/s, write: 1.9 MiB/s
INFO:   7% (16.8 GiB of 240.0 GiB) in 5m 25s, read: 67.9 MiB/s, write: 3.1 MiB/s
INFO:   8% (19.2 GiB of 240.0 GiB) in 6m 8s, read: 57.3 MiB/s, write: 1.9 MiB/s
INFO:   9% (21.6 GiB of 240.0 GiB) in 6m 54s, read: 53.0 MiB/s, write: 657.9 KiB/s
INFO:  10% (24.1 GiB of 240.0 GiB) in 7m 44s, read: 50.6 MiB/s, write: 3.6 MiB/s
INFO:  11% (26.4 GiB of 240.0 GiB) in 8m 31s, read: 50.8 MiB/s, write: 2.5 MiB/s
INFO:  12% (28.8 GiB of 240.0 GiB) in 9m 49s, read: 31.3 MiB/s, write: 3.0 MiB/s
INFO:  13% (31.3 GiB of 240.0 GiB) in 11m 21s, read: 27.5 MiB/s, write: 3.8 MiB/s
INFO:  14% (33.6 GiB of 240.0 GiB) in 12m 37s, read: 31.3 MiB/s, write: 1.3 MiB/s
INFO:  15% (36.0 GiB of 240.0 GiB) in 13m 27s, read: 49.2 MiB/s, write: 2.8 MiB/s
INFO:  16% (38.4 GiB of 240.0 GiB) in 15m 23s, read: 21.3 MiB/s, write: 6.1 MiB/s
INFO:  17% (40.9 GiB of 240.0 GiB) in 16m 11s, read: 52.8 MiB/s, write: 4.2 MiB/s
INFO:  18% (43.3 GiB of 240.0 GiB) in 17m 13s, read: 38.9 MiB/s, write: 5.1 MiB/s
INFO:  19% (45.7 GiB of 240.0 GiB) in 17m 58s, read: 55.5 MiB/s, write: 1.8 MiB/s
INFO:  20% (48.1 GiB of 240.0 GiB) in 18m 49s, read: 48.0 MiB/s, write: 2.3 MiB/s
INFO:  21% (50.4 GiB of 240.0 GiB) in 19m 24s, read: 68.3 MiB/s, write: 1.9 MiB/s
INFO:  22% (52.8 GiB of 240.0 GiB) in 20m 17s, read: 46.1 MiB/s, write: 2.2 MiB/s
INFO:  23% (55.3 GiB of 240.0 GiB) in 21m 27s, read: 37.0 MiB/s, write: 2.1 MiB/s
INFO:  24% (57.7 GiB of 240.0 GiB) in 22m 16s, read: 49.6 MiB/s, write: 440.2 KiB/s
INFO:  25% (60.0 GiB of 240.0 GiB) in 22m 55s, read: 61.2 MiB/s, write: 586.5 KiB/s
INFO:  26% (62.4 GiB of 240.0 GiB) in 23m 32s, read: 66.5 MiB/s, write: 288.9 KiB/s
INFO:  27% (64.8 GiB of 240.0 GiB) in 24m 33s, read: 40.3 MiB/s, write: 176.1 KiB/s
INFO:  28% (67.2 GiB of 240.0 GiB) in 25m 33s, read: 40.7 MiB/s, write: 2.9 MiB/s
INFO:  29% (69.6 GiB of 240.0 GiB) in 26m 48s, read: 33.0 MiB/s, write: 1.4 MiB/s
INFO:  30% (72.0 GiB of 240.0 GiB) in 27m 27s, read: 62.5 MiB/s, write: 3.8 MiB/s
INFO:  31% (74.5 GiB of 240.0 GiB) in 28m 5s, read: 67.4 MiB/s, write: 1.5 MiB/s
INFO:  32% (76.8 GiB of 240.0 GiB) in 28m 56s, read: 46.1 MiB/s, write: 696.4 KiB/s
INFO:  33% (79.3 GiB of 240.0 GiB) in 29m 36s, read: 63.2 MiB/s, write: 52.4 KiB/s
INFO:  34% (81.6 GiB of 240.0 GiB) in 30m 18s, read: 57.2 MiB/s, write: 103.0 KiB/s
INFO:  35% (84.0 GiB of 240.0 GiB) in 31m 6s, read: 51.2 MiB/s, write: 658.8 KiB/s
INFO:  36% (86.5 GiB of 240.0 GiB) in 31m 51s, read: 55.3 MiB/s, write: 1.1 MiB/s
INFO:  37% (88.8 GiB of 240.0 GiB) in 32m 31s, read: 60.4 MiB/s, write: 71.1 KiB/s
INFO:  38% (91.2 GiB of 240.0 GiB) in 34m 50s, read: 17.6 MiB/s, write: 9.6 MiB/s
INFO:  39% (93.6 GiB of 240.0 GiB) in 36m 23s, read: 26.6 MiB/s, write: 9.9 MiB/s
INFO:  40% (96.0 GiB of 240.0 GiB) in 37m 35s, read: 34.2 MiB/s, write: 9.4 MiB/s
INFO:  41% (98.4 GiB of 240.0 GiB) in 39m 52s, read: 18.0 MiB/s, write: 9.2 MiB/s
INFO:  42% (100.8 GiB of 240.0 GiB) in 41m 3s, read: 34.3 MiB/s, write: 9.7 MiB/s
INFO:  43% (103.3 GiB of 240.0 GiB) in 41m 44s, read: 61.2 MiB/s, write: 2.8 MiB/s
INFO:  44% (105.6 GiB of 240.0 GiB) in 42m 17s, read: 73.8 MiB/s, write: 1.3 MiB/s
INFO:  45% (108.0 GiB of 240.0 GiB) in 43m 52s, read: 25.6 MiB/s, write: 2.4 MiB/s
INFO:  46% (110.4 GiB of 240.0 GiB) in 44m 44s, read: 47.2 MiB/s, write: 6.4 MiB/s
INFO:  47% (112.8 GiB of 240.0 GiB) in 45m 35s, read: 48.7 MiB/s, write: 5.2 MiB/s
INFO:  48% (115.2 GiB of 240.0 GiB) in 46m 56s, read: 30.1 MiB/s, write: 7.1 MiB/s
INFO:  49% (117.8 GiB of 240.0 GiB) in 47m 25s, read: 90.4 MiB/s, write: 223.2 KiB/s
INFO:  50% (120.1 GiB of 240.0 GiB) in 47m 56s, read: 77.2 MiB/s, write: 62.8 KiB/s
INFO:  51% (122.5 GiB of 240.0 GiB) in 48m 28s, read: 77.9 MiB/s, write: 935.1 KiB/s
INFO:  52% (124.8 GiB of 240.0 GiB) in 49m 14s, read: 50.9 MiB/s, write: 1.8 MiB/s
INFO:  53% (127.2 GiB of 240.0 GiB) in 49m 59s, read: 54.7 MiB/s, write: 914.8 KiB/s
INFO:  54% (129.6 GiB of 240.0 GiB) in 50m 42s, read: 56.7 MiB/s, write: 2.8 MiB/s
INFO:  55% (132.0 GiB of 240.0 GiB) in 56m 24s, read: 7.2 MiB/s, write: 6.9 MiB/s
INFO:  56% (134.4 GiB of 240.0 GiB) in 58m 18s, read: 21.9 MiB/s, write: 5.8 MiB/s
INFO:  57% (137.1 GiB of 240.0 GiB) in 58m 26s, read: 344.2 MiB/s, write: 16.0 KiB/s
INFO:  58% (139.3 GiB of 240.0 GiB) in 58m 33s, read: 311.2 MiB/s, write: 9.1 KiB/s
INFO:  59% (141.7 GiB of 240.0 GiB) in 58m 40s, read: 354.4 MiB/s, write: 36.6 KiB/s
INFO:  60% (144.3 GiB of 240.0 GiB) in 58m 48s, read: 333.7 MiB/s, write: 16.0 KiB/s
INFO:  61% (146.6 GiB of 240.0 GiB) in 58m 55s, read: 337.7 MiB/s, write: 0 B/s
INFO:  62% (149.0 GiB of 240.0 GiB) in 59m 2s, read: 350.3 MiB/s, write: 9.1 KiB/s
INFO:  63% (151.2 GiB of 240.0 GiB) in 59m 10s, read: 289.4 MiB/s, write: 8.0 KiB/s
INFO:  64% (153.8 GiB of 240.0 GiB) in 59m 18s, read: 324.2 MiB/s, write: 88.0 KiB/s
INFO:  65% (156.0 GiB of 240.0 GiB) in 1h 30s, read: 31.6 MiB/s, write: 6.5 MiB/s
INFO:  66% (158.4 GiB of 240.0 GiB) in 1h 3m 25s, read: 14.1 MiB/s, write: 14.1 MiB/s
INFO:  67% (160.8 GiB of 240.0 GiB) in 1h 6m 50s, read: 12.0 MiB/s, write: 12.0 MiB/s
INFO:  68% (163.2 GiB of 240.0 GiB) in 1h 10m 36s, read: 10.9 MiB/s, write: 10.9 MiB/s
INFO:  69% (165.6 GiB of 240.0 GiB) in 1h 15m 54s, read: 7.8 MiB/s, write: 7.5 MiB/s
INFO:  70% (168.0 GiB of 240.0 GiB) in 1h 20m 34s, read: 8.7 MiB/s, write: 8.3 MiB/s
INFO:  71% (170.4 GiB of 240.0 GiB) in 1h 27m 27s, read: 6.0 MiB/s, write: 5.4 MiB/s
INFO:  72% (172.8 GiB of 240.0 GiB) in 1h 33m 46s, read: 6.5 MiB/s, write: 5.8 MiB/s
INFO:  73% (175.2 GiB of 240.0 GiB) in 1h 38m 44s, read: 8.2 MiB/s, write: 7.0 MiB/s
INFO:  74% (177.9 GiB of 240.0 GiB) in 1h 39m 43s, read: 46.6 MiB/s, write: 2.7 MiB/s
INFO:  75% (180.1 GiB of 240.0 GiB) in 1h 39m 49s, read: 385.7 MiB/s, write: 0 B/s
INFO:  76% (182.4 GiB of 240.0 GiB) in 1h 42m 1s, read: 17.7 MiB/s, write: 17.0 MiB/s
INFO:  77% (184.8 GiB of 240.0 GiB) in 1h 44m 3s, read: 20.0 MiB/s, write: 20.0 MiB/s
INFO:  78% (187.2 GiB of 240.0 GiB) in 1h 45m 54s, read: 22.2 MiB/s, write: 22.2 MiB/s
INFO:  79% (189.6 GiB of 240.0 GiB) in 1h 48m 1s, read: 19.3 MiB/s, write: 19.3 MiB/s
INFO:  80% (192.0 GiB of 240.0 GiB) in 1h 49m 53s, read: 21.9 MiB/s, write: 21.9 MiB/s
INFO:  81% (194.4 GiB of 240.0 GiB) in 1h 51m 40s, read: 23.1 MiB/s, write: 23.1 MiB/s
INFO:  82% (196.8 GiB of 240.0 GiB) in 1h 54m 27s, read: 14.7 MiB/s, write: 14.7 MiB/s
INFO:  83% (199.2 GiB of 240.0 GiB) in 1h 57m 43s, read: 12.5 MiB/s, write: 12.5 MiB/s
INFO:  84% (201.6 GiB of 240.0 GiB) in 2h 3m 21s, read: 7.3 MiB/s, write: 7.3 MiB/s
INFO:  85% (204.0 GiB of 240.0 GiB) in 2h 7m 33s, read: 9.8 MiB/s, write: 9.8 MiB/s
INFO:  86% (206.4 GiB of 240.0 GiB) in 2h 11m 15s, read: 11.1 MiB/s, write: 10.9 MiB/s
INFO:  87% (208.8 GiB of 240.0 GiB) in 2h 13m 46s, read: 16.4 MiB/s, write: 16.4 MiB/s
INFO:  88% (211.2 GiB of 240.0 GiB) in 2h 16m 11s, read: 16.9 MiB/s, write: 16.9 MiB/s
INFO:  89% (213.6 GiB of 240.0 GiB) in 2h 20m 34s, read: 9.3 MiB/s, write: 8.7 MiB/s
INFO:  90% (216.0 GiB of 240.0 GiB) in 2h 27m 28s, read: 5.9 MiB/s, write: 5.2 MiB/s
INFO:  91% (218.4 GiB of 240.0 GiB) in 2h 31m 16s, read: 10.8 MiB/s, write: 9.7 MiB/s
INFO:  92% (220.8 GiB of 240.0 GiB) in 2h 34m 9s, read: 14.2 MiB/s, write: 14.2 MiB/s
INFO:  93% (223.2 GiB of 240.0 GiB) in 2h 35m 28s, read: 31.4 MiB/s, write: 14.2 MiB/s
INFO:  94% (225.7 GiB of 240.0 GiB) in 2h 35m 35s, read: 358.2 MiB/s, write: 0 B/s
INFO:  95% (228.2 GiB of 240.0 GiB) in 2h 35m 42s, read: 369.8 MiB/s, write: 0 B/s
INFO:  96% (230.5 GiB of 240.0 GiB) in 2h 35m 49s, read: 336.6 MiB/s, write: 0 B/s
INFO:  97% (232.8 GiB of 240.0 GiB) in 2h 37m 42s, read: 20.8 MiB/s, write: 5.3 MiB/s
INFO:  98% (235.2 GiB of 240.0 GiB) in 2h 41m 25s, read: 11.2 MiB/s, write: 5.2 MiB/s
INFO:  99% (237.9 GiB of 240.0 GiB) in 2h 41m 33s, read: 333.6 MiB/s, write: 0 B/s
INFO: 100% (240.0 GiB of 240.0 GiB) in 2h 41m 40s, read: 314.2 MiB/s, write: 1.1 KiB/s
INFO: backup is sparse: 164.38 GiB (68%) total zero data
INFO: transferred 240.00 GiB in 9700 seconds (25.3 MiB/s)
INFO: archive file size: 44.14GB
INFO: removing backup 'nfs-synology:backup/vzdump-qemu-124-2022_06_18-01_59_13.vma.zst'
INFO: Finished Backup of VM 124 (02:41:44)
INFO: Backup finished at 2022-06-20 06:48:02
INFO: Starting Backup of VM 129 (qemu)
INFO: Backup started at 2022-06-20 06:48:02
INFO: status = running
INFO: VM Name: nms-test-100-clone
INFO: include disk 'virtio0' 'vm.pool:vm-129-disk-0' 310G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: skip unused drive 'vm.pool:vm-129-disk-1' (not included into backup)
INFO: creating vzdump archive '/mnt/pve/nfs-synology/dump/vzdump-qemu-129-2022_06_20-06_48_02.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '0ca96cf5-0317-42d4-a721-0cfdf151ac8c'
INFO: resuming VM again
INFO:   0% (19.2 MiB of 310.0 GiB) in 3s, read: 6.4 MiB/s, write: 3.5 MiB/s
INFO:   1% (3.1 GiB of 310.0 GiB) in 3m 19s, read: 16.1 MiB/s, write: 3.8 MiB/s
INFO:   2% (6.2 GiB of 310.0 GiB) in 6m 10s, read: 18.6 MiB/s, write: 11.7 MiB/s
INFO:   3% (9.3 GiB of 310.0 GiB) in 10m 13s, read: 13.1 MiB/s, write: 13.1 MiB/s
INFO:   4% (12.4 GiB of 310.0 GiB) in 13m 58s, read: 14.1 MiB/s, write: 14.0 MiB/s
INFO:   5% (15.5 GiB of 310.0 GiB) in 19m 5s, read: 10.3 MiB/s, write: 10.3 MiB/s
INFO:   6% (18.6 GiB of 310.0 GiB) in 24m 2s, read: 10.7 MiB/s, write: 10.4 MiB/s
INFO:   7% (21.7 GiB of 310.0 GiB) in 28m 49s, read: 11.0 MiB/s, write: 10.1 MiB/s
INFO:   8% (24.8 GiB of 310.0 GiB) in 32m 26s, read: 14.6 MiB/s, write: 14.5 MiB/s
INFO:   9% (27.9 GiB of 310.0 GiB) in 35m 7s, read: 19.8 MiB/s, write: 19.4 MiB/s
INFO:  10% (31.0 GiB of 310.0 GiB) in 37m 42s, read: 20.5 MiB/s, write: 20.2 MiB/s
INFO:  11% (34.1 GiB of 310.0 GiB) in 40m 40s, read: 17.8 MiB/s, write: 17.6 MiB/s
INFO:  12% (37.2 GiB of 310.0 GiB) in 44m 7s, read: 15.4 MiB/s, write: 15.2 MiB/s
INFO:  13% (40.3 GiB of 310.0 GiB) in 47m 35s, read: 15.3 MiB/s, write: 15.3 MiB/s
INFO:  14% (43.4 GiB of 310.0 GiB) in 51m 29s, read: 13.5 MiB/s, write: 13.3 MiB/s
INFO:  15% (46.5 GiB of 310.0 GiB) in 55m 21s, read: 13.7 MiB/s, write: 13.5 MiB/s
INFO:  16% (49.6 GiB of 310.0 GiB) in 1h 35s, read: 10.1 MiB/s, write: 9.8 MiB/s
INFO:  17% (52.7 GiB of 310.0 GiB) in 1h 5m 8s, read: 11.6 MiB/s, write: 10.4 MiB/s
INFO:  18% (55.8 GiB of 310.0 GiB) in 1h 8m 30s, read: 15.7 MiB/s, write: 13.4 MiB/s
INFO:  19% (58.9 GiB of 310.0 GiB) in 1h 10m 51s, read: 22.5 MiB/s, write: 22.3 MiB/s
INFO:  20% (62.0 GiB of 310.0 GiB) in 1h 14m 3s, read: 16.5 MiB/s, write: 16.2 MiB/s
INFO:  21% (65.1 GiB of 310.0 GiB) in 1h 17m 15s, read: 16.5 MiB/s, write: 16.3 MiB/s
INFO:  22% (68.2 GiB of 310.0 GiB) in 1h 20m 18s, read: 17.4 MiB/s, write: 17.1 MiB/s
INFO:  23% (71.3 GiB of 310.0 GiB) in 1h 23m 35s, read: 16.1 MiB/s, write: 15.9 MiB/s
INFO:  24% (74.4 GiB of 310.0 GiB) in 1h 27m 22s, read: 14.0 MiB/s, write: 14.0 MiB/s
INFO:  25% (77.5 GiB of 310.0 GiB) in 1h 31m 24s, read: 13.1 MiB/s, write: 13.1 MiB/s
INFO:  26% (80.6 GiB of 310.0 GiB) in 1h 35m 20s, read: 13.4 MiB/s, write: 13.3 MiB/s
INFO:  27% (83.7 GiB of 310.0 GiB) in 1h 38m 30s, read: 16.8 MiB/s, write: 15.8 MiB/s
INFO:  28% (86.8 GiB of 310.0 GiB) in 1h 40m 32s, read: 25.9 MiB/s, write: 24.7 MiB/s
INFO:  29% (89.9 GiB of 310.0 GiB) in 1h 43m 27s, read: 18.2 MiB/s, write: 17.4 MiB/s
INFO:  30% (93.0 GiB of 310.0 GiB) in 1h 45m 39s, read: 24.0 MiB/s, write: 22.0 MiB/s
INFO:  31% (96.1 GiB of 310.0 GiB) in 1h 47m 59s, read: 22.9 MiB/s, write: 22.0 MiB/s
INFO:  32% (99.2 GiB of 310.0 GiB) in 1h 50m 3s, read: 25.3 MiB/s, write: 23.3 MiB/s
INFO:  33% (102.3 GiB of 310.0 GiB) in 1h 51m 54s, read: 28.9 MiB/s, write: 26.9 MiB/s
INFO:  34% (105.4 GiB of 310.0 GiB) in 1h 53m 51s, read: 26.9 MiB/s, write: 26.0 MiB/s
INFO:  35% (108.5 GiB of 310.0 GiB) in 1h 56m 4s, read: 24.0 MiB/s, write: 23.0 MiB/s
INFO:  36% (111.6 GiB of 310.0 GiB) in 1h 59m 54s, read: 13.7 MiB/s, write: 13.6 MiB/s
INFO:  37% (114.7 GiB of 310.0 GiB) in 2h 3m 17s, read: 15.6 MiB/s, write: 14.9 MiB/s
INFO:  38% (117.8 GiB of 310.0 GiB) in 2h 6m 32s, read: 16.3 MiB/s, write: 16.1 MiB/s
INFO:  39% (120.9 GiB of 310.0 GiB) in 2h 9m 11s, read: 20.1 MiB/s, write: 19.2 MiB/s
INFO:  40% (124.0 GiB of 310.0 GiB) in 2h 12m 4s, read: 18.3 MiB/s, write: 17.5 MiB/s
INFO:  41% (127.1 GiB of 310.0 GiB) in 2h 15m 11s, read: 16.9 MiB/s, write: 15.6 MiB/s
INFO:  42% (130.2 GiB of 310.0 GiB) in 2h 18m 36s, read: 15.5 MiB/s, write: 15.3 MiB/s
INFO:  43% (133.3 GiB of 310.0 GiB) in 2h 21m 33s, read: 17.9 MiB/s, write: 17.5 MiB/s
INFO:  44% (136.4 GiB of 310.0 GiB) in 2h 24m 16s, read: 19.5 MiB/s, write: 19.1 MiB/s
INFO:  45% (139.5 GiB of 310.0 GiB) in 2h 27m 5s, read: 18.8 MiB/s, write: 18.4 MiB/s
INFO:  46% (142.6 GiB of 310.0 GiB) in 2h 30m 5s, read: 17.6 MiB/s, write: 17.0 MiB/s
INFO:  47% (145.7 GiB of 310.0 GiB) in 2h 32m 38s, read: 20.7 MiB/s, write: 20.7 MiB/s
INFO:  48% (148.8 GiB of 310.0 GiB) in 2h 37m 24s, read: 11.1 MiB/s, write: 11.1 MiB/s
INFO:  49% (151.9 GiB of 310.0 GiB) in 2h 41m 13s, read: 14.0 MiB/s, write: 14.0 MiB/s
INFO:  50% (155.0 GiB of 310.0 GiB) in 2h 45m 48s, read: 11.5 MiB/s, write: 11.5 MiB/s
INFO:  51% (158.1 GiB of 310.0 GiB) in 2h 49m 38s, read: 13.8 MiB/s, write: 13.8 MiB/s
INFO:  52% (161.2 GiB of 310.0 GiB) in 2h 53m 12s, read: 14.8 MiB/s, write: 14.1 MiB/s
INFO:  53% (164.3 GiB of 310.0 GiB) in 2h 57m 52s, read: 11.4 MiB/s, write: 11.0 MiB/s
INFO:  54% (167.4 GiB of 310.0 GiB) in 3h 2m 41s, read: 10.9 MiB/s, write: 10.3 MiB/s
INFO:  55% (170.5 GiB of 310.0 GiB) in 3h 8m 34s, read: 9.0 MiB/s, write: 8.8 MiB/s
INFO:  56% (173.6 GiB of 310.0 GiB) in 3h 10m 55s, read: 22.6 MiB/s, write: 20.7 MiB/s
INFO:  57% (176.7 GiB of 310.0 GiB) in 3h 15m 3s, read: 12.7 MiB/s, write: 11.9 MiB/s
INFO:  58% (179.8 GiB of 310.0 GiB) in 3h 18m 36s, read: 14.9 MiB/s, write: 12.9 MiB/s
INFO:  59% (182.9 GiB of 310.0 GiB) in 3h 22m 10s, read: 14.8 MiB/s, write: 11.8 MiB/s
INFO:  60% (186.0 GiB of 310.0 GiB) in 3h 24m 32s, read: 22.3 MiB/s, write: 4.1 MiB/s
INFO:  61% (189.1 GiB of 310.0 GiB) in 3h 26m 20s, read: 29.4 MiB/s, write: 7.2 MiB/s
INFO:  62% (192.2 GiB of 310.0 GiB) in 3h 28m 35s, read: 23.5 MiB/s, write: 4.6 MiB/s
INFO:  63% (195.3 GiB of 310.0 GiB) in 3h 30m 50s, read: 23.6 MiB/s, write: 20.0 KiB/s
INFO:  64% (198.4 GiB of 310.0 GiB) in 3h 32m 33s, read: 30.7 MiB/s, write: 57.8 KiB/s
INFO:  65% (201.5 GiB of 310.0 GiB) in 3h 34m 43s, read: 24.4 MiB/s, write: 22.2 KiB/s
INFO:  66% (204.6 GiB of 310.0 GiB) in 3h 37m 41s, read: 17.8 MiB/s, write: 47.1 KiB/s
INFO:  67% (207.7 GiB of 310.0 GiB) in 3h 44m 12s, read: 8.1 MiB/s, write: 5.4 MiB/s
INFO:  68% (210.8 GiB of 310.0 GiB) in 3h 48m 33s, read: 12.2 MiB/s, write: 12.2 MiB/s
INFO:  69% (213.9 GiB of 310.0 GiB) in 3h 52m 55s, read: 12.1 MiB/s, write: 12.1 MiB/s
INFO:  70% (217.0 GiB of 310.0 GiB) in 3h 57m 33s, read: 11.4 MiB/s, write: 11.4 MiB/s
INFO:  71% (220.1 GiB of 310.0 GiB) in 4h 2m 29s, read: 10.7 MiB/s, write: 10.7 MiB/s
INFO:  72% (223.2 GiB of 310.0 GiB) in 4h 6m 31s, read: 13.1 MiB/s, write: 13.1 MiB/s
INFO:  73% (226.3 GiB of 310.0 GiB) in 4h 9m 50s, read: 15.9 MiB/s, write: 15.9 MiB/s
INFO:  74% (229.4 GiB of 310.0 GiB) in 4h 15m 24s, read: 9.5 MiB/s, write: 9.5 MiB/s
INFO:  75% (232.5 GiB of 310.0 GiB) in 4h 21m 46s, read: 8.3 MiB/s, write: 8.3 MiB/s
INFO:  76% (235.6 GiB of 310.0 GiB) in 4h 24m 57s, read: 16.6 MiB/s, write: 16.6 MiB/s
INFO:  77% (238.7 GiB of 310.0 GiB) in 4h 29m 24s, read: 11.9 MiB/s, write: 11.6 MiB/s
INFO:  78% (241.8 GiB of 310.0 GiB) in 4h 33m 18s, read: 13.6 MiB/s, write: 13.6 MiB/s
INFO:  79% (244.9 GiB of 310.0 GiB) in 4h 35m 37s, read: 22.8 MiB/s, write: 22.8 MiB/s
INFO:  80% (248.0 GiB of 310.0 GiB) in 4h 38m 2s, read: 21.8 MiB/s, write: 21.8 MiB/s
INFO:  81% (251.1 GiB of 310.0 GiB) in 4h 40m 43s, read: 19.9 MiB/s, write: 19.9 MiB/s
INFO:  82% (254.2 GiB of 310.0 GiB) in 4h 43m 29s, read: 19.1 MiB/s, write: 19.1 MiB/s
INFO:  83% (257.3 GiB of 310.0 GiB) in 4h 46m 42s, read: 16.3 MiB/s, write: 16.3 MiB/s
INFO:  84% (260.4 GiB of 310.0 GiB) in 4h 49m 49s, read: 17.0 MiB/s, write: 17.0 MiB/s
INFO:  85% (263.5 GiB of 310.0 GiB) in 4h 53m 18s, read: 15.2 MiB/s, write: 15.2 MiB/s
INFO:  86% (266.6 GiB of 310.0 GiB) in 4h 56m 58s, read: 14.4 MiB/s, write: 14.4 MiB/s
INFO:  87% (269.7 GiB of 310.0 GiB) in 5h 46s, read: 14.0 MiB/s, write: 13.9 MiB/s
INFO:  88% (272.8 GiB of 310.0 GiB) in 5h 7m 2s, read: 8.4 MiB/s, write: 7.7 MiB/s
INFO:  89% (275.9 GiB of 310.0 GiB) in 5h 14m 40s, read: 6.9 MiB/s, write: 6.5 MiB/s
INFO:  90% (279.0 GiB of 310.0 GiB) in 5h 20m 48s, read: 8.6 MiB/s, write: 8.4 MiB/s