[PVE-User] Failed backups of stopped machines

[PVE-User] Failed backups of stopped machines

[Sven]

Hello,

in the last few days, there have been some backup errors. The log is 
always the same:

> INFO: Starting Backup of VM 147 (qemu)
> INFO: Backup started at 2023-01-06 02:55:37
> INFO: status = stopped
> INFO: backup mode: stop
> INFO: ionice priority: 7
> INFO: VM Name: <censored>
> INFO: include disk 'virtio0' 'ceph_disks:vm-147-disk-0' 10G
> INFO: include disk 'virtio1' 'ceph_disks:vm-147-disk-1' 20G
> INFO: include disk 'virtio2' 'ceph_disks:vm-147-disk-2' 150G
> INFO: include disk 'virtio3' 'ceph_disks:vm-147-disk-3' 4G
> INFO: creating Proxmox Backup Server archive 
> 'vm/147/2023-01-06T01:55:37Z'
> INFO: starting kvm to execute backup task
> ERROR: VM 147 qmp command 'backup' failed - got timeout
> INFO: aborting backup job
> INFO: stopping kvm after backup task
> ERROR: Backup of VM 147 failed - VM 147 qmp command 'backup' failed - 
> got timeout
> INFO: Failed at 2023-01-06 02:58:20

Status: and config:

> $ qm status 147 --verbose
> cpus: 4
> disk: 0
> diskread: 0
> diskwrite: 0
> maxdisk: 10737418240
> maxmem: 8589934592
> mem: 0
> name: <censored>
> netin: 0
> netout: 0
> qmpstatus: stopped
> status: stopped
> uptime: 0
> vmid: 147

> $ qm config 147
> boot: order=net0;virtio0
> cores: 4
> ide2: none,media=cdrom
> memory: 8192
> name: <censored>
> net0: virtio=F2:F3:75:8D:92:9E,bridge=vmbr0,tag=1
> numa: 0
> ostype: l26
> scsihw: virtio-scsi-pci
> smbios1: uuid=743f558d-823a-4213-baa3-558a663aa273
> sockets: 1
> virtio0: ceph_disks:vm-147-disk-0,size=10G
> virtio1: ceph_disks:vm-147-disk-1,size=20G
> virtio2: ceph_disks:vm-147-disk-2,size=150G
> virtio3: ceph_disks:vm-147-disk-3,size=4G
> vmgenid: 90e25752-16b4-431d-92fe-8d71b364fb63

Proxmox-Version: 7.2-1
Version pve-qemu-kvm: 7.0.0-4
Version of proxmox-backup-server: 2.2.5-1

There are several VMs failing each night, not always the same. There is 
an accumulation of failed backups of stopped VMs.

Does anybody has an idea? What timed out (connection to backup server, 
...)?

Thanks!
Regards,
Sven

Re: [PVE-User] Failed backups of stopped machines

[Konold, Martin]

Hi Sven,

how many nodes are in your pve cluster?

I am experiencing the same as all nodes seem to sequentially backup 
their vms but the nodes run in parallel.

This can overwhelm the PBS.

---
Regards
ppa. Martin Konold

--
Martin Konold - Prokurist, CTO
KONSEC GmbH -⁠ make things real
Amtsgericht Stuttgart, HRB 23690
Geschäftsführer: Andreas Mack
Im Köller 3, 70794 Filderstadt, Germany

On 2023-01-09 13:08, Sven wrote:
> Hello,
> 
> in the last few days, there have been some backup errors. The log is
> always the same:
> 
>> INFO: Starting Backup of VM 147 (qemu)
>> INFO: Backup started at 2023-01-06 02:55:37
>> INFO: status = stopped
>> INFO: backup mode: stop
>> INFO: ionice priority: 7
>> INFO: VM Name: <censored>
>> INFO: include disk 'virtio0' 'ceph_disks:vm-147-disk-0' 10G
>> INFO: include disk 'virtio1' 'ceph_disks:vm-147-disk-1' 20G
>> INFO: include disk 'virtio2' 'ceph_disks:vm-147-disk-2' 150G
>> INFO: include disk 'virtio3' 'ceph_disks:vm-147-disk-3' 4G
>> INFO: creating Proxmox Backup Server archive 
>> 'vm/147/2023-01-06T01:55:37Z'
>> INFO: starting kvm to execute backup task
>> ERROR: VM 147 qmp command 'backup' failed - got timeout
>> INFO: aborting backup job
>> INFO: stopping kvm after backup task
>> ERROR: Backup of VM 147 failed - VM 147 qmp command 'backup' failed - 
>> got timeout
>> INFO: Failed at 2023-01-06 02:58:20
> 
> Status: and config:
> 
>> $ qm status 147 --verbose
>> cpus: 4
>> disk: 0
>> diskread: 0
>> diskwrite: 0
>> maxdisk: 10737418240
>> maxmem: 8589934592
>> mem: 0
>> name: <censored>
>> netin: 0
>> netout: 0
>> qmpstatus: stopped
>> status: stopped
>> uptime: 0
>> vmid: 147
> 
>> $ qm config 147
>> boot: order=net0;virtio0
>> cores: 4
>> ide2: none,media=cdrom
>> memory: 8192
>> name: <censored>
>> net0: virtio=F2:F3:75:8D:92:9E,bridge=vmbr0,tag=1
>> numa: 0
>> ostype: l26
>> scsihw: virtio-scsi-pci
>> smbios1: uuid=743f558d-823a-4213-baa3-558a663aa273
>> sockets: 1
>> virtio0: ceph_disks:vm-147-disk-0,size=10G
>> virtio1: ceph_disks:vm-147-disk-1,size=20G
>> virtio2: ceph_disks:vm-147-disk-2,size=150G
>> virtio3: ceph_disks:vm-147-disk-3,size=4G
>> vmgenid: 90e25752-16b4-431d-92fe-8d71b364fb63
> 
> Proxmox-Version: 7.2-1
> Version pve-qemu-kvm: 7.0.0-4
> Version of proxmox-backup-server: 2.2.5-1
> 
> There are several VMs failing each night, not always the same. There
> is an accumulation of failed backups of stopped VMs.
> 
> Does anybody has an idea? What timed out (connection to backup server, 
> ...)?
> 
> Thanks!
> Regards,
> Sven
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Failed backups of stopped machines

[Sven]

Hi Martin,

thanks, I will try to edit the backup-schedule not to run 
simultaneously. There are five nodes in the cluster.

And I'm going to update the nodes.

Regards,
Sven

Am 2023-01-09 13:46, schrieb Konold, Martin:
> Hi Sven,
> 
> how many nodes are in your pve cluster?
> 
> I am experiencing the same as all nodes seem to sequentially backup
> their vms but the nodes run in parallel.
> 
> This can overwhelm the PBS.
> 
> ---
> Regards
> ppa. Martin Konold
> 
> --
> Martin Konold - Prokurist, CTO
> KONSEC GmbH -⁠ make things real
> Amtsgericht Stuttgart, HRB 23690
> Geschäftsführer: Andreas Mack
> Im Köller 3, 70794 Filderstadt, Germany
>