Re: [PVE-User] Backup of one VM always fails

[Yannis Milios <yannis.milios@gmail.com>]

Can you try removing this specific VM from the normal backup schedule and
then create a new test schedule for it, if possible to a different backup
target (nfs, local etc) ?



On Fri, 4 Dec 2020 at 11:10, Frank Thommen <f.thommen@dkfz-heidelberg.de>
wrote:

> On 04/12/2020 11:36, Arjen via pve-user wrote:
> > On Fri, 2020-12-04 at 11:22 +0100, Frank Thommen wrote:
> >>
> >> On 04/12/2020 09:30, Frank Thommen wrote:
> >>>> On Thursday, December 3, 2020 10:16 PM, Frank Thommen
> >>>> <f.thommen@dkfz-heidelberg.de> wrote:
> >>>>
> >>>>>
> >>>>> Dear all,
> >>>>>
> >>>>> on our PVE cluster, the backup of a specific VM always fails
> >>>>> (which
> >>>>> makes us worry, as it is our GitLab instance). The general
> >>>>> backup plan
> >>>>> is "back up all VMs at 00:30". In the confirmation email we
> >>>>> see, that
> >>>>> the backup of this specific VM takes six to seven hours and
> >>>>> then fails.
> >>>>> The error message in the overview table used to be:
> >>>>>
> >>>>> vma_queue_write: write error - Broken pipe
> >>>>>
> >>>>> With detailed log
> >>>>>
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -----------------------------------------------
> >>>>>
> >>>>>
> >>>>> 123: 2020-12-01 02:53:08 INFO: Starting Backup of VM 123 (qemu)
> >>>>> 123: 2020-12-01 02:53:08 INFO: status = running
> >>>>> 123: 2020-12-01 02:53:09 INFO: update VM 123: -lock backup
> >>>>> 123: 2020-12-01 02:53:09 INFO: VM Name: odcf-vm123
> >>>>> 123: 2020-12-01 02:53:09 INFO: include disk 'virtio0'
> >>>>> 'ceph-rbd:vm-123-disk-0' 20G
> >>>>> 123: 2020-12-01 02:53:09 INFO: include disk 'virtio1'
> >>>>> 'ceph-rbd:vm-123-disk-2' 1000G
> >>>>> 123: 2020-12-01 02:53:09 INFO: include disk 'virtio2'
> >>>>> 'ceph-rbd:vm-123-disk-3' 2T
> >>>>> 123: 2020-12-01 02:53:09 INFO: backup mode: snapshot
> >>>>> 123: 2020-12-01 02:53:09 INFO: ionice priority: 7
> >>>>> 123: 2020-12-01 02:53:09 INFO: creating archive
> >>>>> '/mnt/pve/cephfs/dump/vzdump-qemu-123-2020_12_01-
> >>>>> 02_53_08.vma.lzo'
> >>>>> 123: 2020-12-01 02:53:09 INFO: started backup task
> >>>>> 'a38ff50a-f474-4b0a-a052-01a835d5c5c7'
> >>>>> 123: 2020-12-01 02:53:12 INFO: status: 0%
> >>>>> (167772160/3294239916032),
> >>>>> sparse 0% (31563776), duration 3, read/write 55/45 MB/s
> >>>>> [... ecc. ecc. ...]
> >>>>> 123: 2020-12-01 09:42:14 INFO: status: 35%
> >>>>> (1170252365824/3294239916032), sparse 0% (26845003776),
> >>>>> duration 24545,
> >>>>> read/write 59/56 MB/s
> >>>>> 123: 2020-12-01 09:42:14 ERROR: vma_queue_write: write error -
> >>>>> Broken
> >>>>> pipe
> >>>>> 123: 2020-12-01 09:42:14 INFO: aborting backup job
> >>>>> 123: 2020-12-01 09:42:15 ERROR: Backup of VM 123 failed -
> >>>>> vma_queue_write: write error - Broken pipe
> >>>>>
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>> ----------!
> >>>>>
> >>> ---------
> >>> -----------------------------------------------------------------
> >>> -----------------------------------------------------------------
> >>> -----------------------------------------------------------------
> >>> -----------------------------------------------------------------
> >>> ----------------------------------
> >>>
> >>>>> Since lately (upgrade to the newest PVE release) it's
> >>>>>
> >>>>> VM 123 qmp command 'query-backup' failed - got timeout
> >>>>>
> >>>>> with log
> >>>>>
> >>>>> -------------------------------------------------------------
> >>>>> -------------------------------------------------------------
> >>>>>
> >>>>>
> >>>>> 123: 2020-12-03 03:29:00 INFO: Starting Backup of VM 123 (qemu)
> >>>>> 123: 2020-12-03 03:29:00 INFO: status = running
> >>>>> 123: 2020-12-03 03:29:00 INFO: VM Name: odcf-vm123
> >>>>> 123: 2020-12-03 03:29:00 INFO: include disk 'virtio0'
> >>>>> 'ceph-rbd:vm-123-disk-0' 20G
> >>>>> 123: 2020-12-03 03:29:00 INFO: include disk 'virtio1'
> >>>>> 'ceph-rbd:vm-123-disk-2' 1000G
> >>>>> 123: 2020-12-03 03:29:00 INFO: include disk 'virtio2'
> >>>>> 'ceph-rbd:vm-123-disk-3' 2T
> >>>>> 123: 2020-12-03 03:29:01 INFO: backup mode: snapshot
> >>>>> 123: 2020-12-03 03:29:01 INFO: ionice priority: 7
> >>>>> 123: 2020-12-03 03:29:01 INFO: creating vzdump archive
> >>>>> '/mnt/pve/cephfs/dump/vzdump-qemu-123-2020_12_03-
> >>>>> 03_29_00.vma.lzo'
> >>>>> 123: 2020-12-03 03:29:01 INFO: started backup task
> >>>>> 'cc7cde4e-20e8-4e26-a89a-f6f1aa9e9612'
> >>>>> 123: 2020-12-03 03:29:01 INFO: resuming VM again
> >>>>> 123: 2020-12-03 03:29:04 INFO: 0% (284.0 MiB of 3.0 TiB) in 3s,
> >>>>> read:
> >>>>> 94.7 MiB/s, write: 51.7 MiB/s
> >>>>> [... ecc. ecc. ...]
> >>>>> 123: 2020-12-03 09:05:08 INFO: 36% (1.1 TiB of 3.0 TiB) in 5h
> >>>>> 36m 7s,
> >>>>> read: 57.3 MiB/s, write: 53.6 MiB/s
> >>>>> 123: 2020-12-03 09:22:57 ERROR: VM 123 qmp command 'query-
> >>>>> backup' failed
> >>>>>
> >>>>> -   got timeout
> >>>>>     123: 2020-12-03 09:22:57 INFO: aborting backup job
> >>>>>     123: 2020-12-03 09:32:57 ERROR: VM 123 qmp command 'backup-
> >>>>> cancel'
> >>>>>     failed - unable to connect to VM 123 qmp socket - timeout
> >>>>> after
> >>>>> 5981 retries
> >>>>>     123: 2020-12-03 09:32:57 ERROR: Backup of VM 123 failed -
> >>>>> VM 123 qmp
> >>>>>     command 'query-backup' failed - got timeout
> >>>>>
> >>>>>
> >>>>> The VM has some quite big vdisks (20G, 1T and 2T). All stored
> >>>>> in Ceph.
> >>>>> There is still plenty of space in Ceph.
> >>>>>
> >>>>> Can anyone give us some hint on how to investigate and debug
> >>>>> this
> >>>>> further?
> >>>>
> >>>> Because it is a write error, maybe we should look at the backup
> >>>> destination.
> >>>> Maybe it is a network connection issue? Maybe something wrong
> >>>> with the
> >>>> host? Maybe the disk is full?
> >>>> Which storage are you using for backup? Can you show us the
> >>>> corresponding entry in /etc/pve/storage.cfg?
> >>>
> >>> We are backing up to cephfs with still 8 TB or so free.
> >>>
> >>> /etc/pve/storage.cfg is
> >>> ------------
> >>> dir: local
> >>>          path /var/lib/vz
> >>>          content vztmpl,backup,iso
> >>>
> >>> dir: data
> >>>          path /data
> >>>          content snippets,images,backup,iso,rootdir,vztmpl
> >>>
> >>> cephfs: cephfs
> >>>          path /mnt/pve/cephfs
> >>>          content backup,vztmpl,iso
> >>>          maxfiles 5
> >>>
> >>> rbd: ceph-rbd
> >>>          content images,rootdir
> >>>          krbd 0
> >>>          pool pve-pool1
> >>> ------------
> >>>
> >>
> >> The problem has reached a new level of urgency, as since two days
> >> each
> >> time after a failed backup the VMm becomes unaccessible and has to be
> >> stopped and started manually from the PVE UI.
> >
> > I don't see anything wrong the configuration that you shared.
> > Was anything changed in the last few days since the last successful
> > backup? Any updates from Proxmox? Changes to the network?
> > I know very little about Ceph and clusters, sorry.
> > What makes this VM different, except for the size of the disks?
>
> On December 1st the Hypervisor has been updated to PVE 6.3-2 (I think
> from 6.1-3).  After that the error message slightly changed and - in
> hindsight - since then the VM stops being accessible after the failed
> backup.
>
> However: The VM never ever backed up successfully, not even before the
> PVE upgrade.  It's just that no one really took notice of it.
>
> The VM is not really special.  It's our only Debian VM (but I hope
> that's not an issue :-) and the VM has been migrated 1:1 from oVirt by
> migrating and importing the disk images.  But we have a few other such
> VMs and they run and back up just fine.
>
> No network changes. Basically nothing changed that I could think of.
>
> But to be clear: Our current main problem is the failing backup, not the
> crash.
>
>
> Cheers, Frank
>
>
>
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
> --
Sent from Gmail Mobile