[PVE-User] Backup Failing sporadically due to "PBS backups are not supported by the running QEMU version"

[PVE-User] Backup Failing sporadically due to "PBS backups are not supported by the running QEMU version"

[Ralf Storm]

Hello,


I have this Error first time ca. 2 weeks ago when backing up the biggest 
of our 70 vms : "PBS backups are not supported by the running QEMU 
version. Please make sure you've installed the latest version and the VM 
has been restarted"

We have a Subscription and are on the newest updates.

The only thing thats helps ist to shutdown the VM and reboot the whole 
Node on which the VM is runninng. Shutdown, waiting for the quemu to 
stop and then restart does not help, migrating to another Node does not 
help. Shutdown and migrate to another Node does not help.

And this happens onyto this one VM, the other more than 70 VMs are not 
affected.

The storage is ceph in Proxmox.


-- 
Ralf Storm
IT Administrator

Konzept Informationssysteme GmbH
...

Re: [PVE-User] Backup Failing sporadically due to "PBS backups are not supported by the running QEMU version"

[Fabian Grünbichler]

On February 5, 2021 10:43 am, Ralf Storm wrote:
> Hello,
> 
> 
> I have this Error first time ca. 2 weeks ago when backing up the biggest 
> of our 70 vms : "PBS backups are not supported by the running QEMU 
> version. Please make sure you've installed the latest version and the VM 
> has been restarted"
> 
> We have a Subscription and are on the newest updates.

that does sound rather strange. could you post the following information 
(feel free to censort parts you are not comfortable with sharing 
publicly)

pveversion -v
qm config XXX
qm status XXX --verbose

(where XXX is the VMID of the problematic VM)

journal output from around the attempted backup and the full backup task 
log would also be interesting.

> The only thing thats helps ist to shutdown the VM and reboot the whole 
> Node on which the VM is runninng. Shutdown, waiting for the quemu to 
> stop and then restart does not help, migrating to another Node does not 
> help. Shutdown and migrate to another Node does not help.

that sounds strange as well - a reboot of the node and a fresh start of 
the VM (via migration or shutdown and start) are equivalent w.r.t. what 
state of the old running VM is carried over..

> And this happens onyto this one VM, the other more than 70 VMs are not 
> affected.

what does biggest mean in the context of this node? biggest disk? 
highest memory? ... ?

thanks!

Re: [PVE-User] Backup Failing sporadically due to "PBS backups are not supported by the running QEMU version"

[Fabian Grünbichler]

are the package versions unchanged? could you try upgrading to pve-qemu-kvm 5.1.0-8?


> storm <ralf.storm@konzept-is.de> hat am 18.02.2021 09:20 geschrieben:
> 
>  
> Hello,
> 
> 
> so the Backup was working for a few days and then the error appaers again.
> 
> additionally I cannot live migrate the vm:
> 
> 2021-02-18 09:15:11 ERROR: migration aborted (duration 00:00:03): VM 
> 3091 qmp command 'query-machines' failed - unable to connect to VM 3091 
> qmp socket - timeout after 31 retries TASK ERROR: migration aborted
> 
> and the same for console
> 
> 
> best regards

Re: [PVE-User] Backup Failing sporadically due to "PBS backups are not supported by the running QEMU version"

[storm]

upgrade of the nodes is in progress, moved the affected vm to a node 
which is upgraded to 5.1.0-8, backup is runnging right now - will keep 
you informed if this happens again on this version



Am 18/02/2021 um 09:50 schrieb Fabian Grünbichler:
> are the package versions unchanged? could you try upgrading to pve-qemu-kvm 5.1.0-8?
>
>
>> storm <ralf.storm@konzept-is.de> hat am 18.02.2021 09:20 geschrieben:
>>
>>   
>> Hello,
>>
>>
>> so the Backup was working for a few days and then the error appaers again.
>>
>> additionally I cannot live migrate the vm:
>>
>> 2021-02-18 09:15:11 ERROR: migration aborted (duration 00:00:03): VM
>> 3091 qmp command 'query-machines' failed - unable to connect to VM 3091
>> qmp socket - timeout after 31 retries TASK ERROR: migration aborted
>>
>> and the same for console
>>
>>
>> best regards

Re: [PVE-User] Backup Failing sporadically due to "PBS backups are not supported by the running QEMU version"

[storm]

so the last backup during last night failed again with the same error on 
the same node with pve-qemu-kvm 5.1.0-8 on which the backup was working 
yesterday

any ideas?






Am 18/02/2021 um 11:08 schrieb storm:
> upgrade of the nodes is in progress, moved the affected vm to a node 
> which is upgraded to 5.1.0-8, backup is runnging right now - will keep 
> you informed if this happens again on this version
>
>
>
> Am 18/02/2021 um 09:50 schrieb Fabian Grünbichler:
>> are the package versions unchanged? could you try upgrading to 
>> pve-qemu-kvm 5.1.0-8?
>>
>>
>>> storm <ralf.storm@konzept-is.de> hat am 18.02.2021 09:20 geschrieben:
>>>
>>>   Hello,
>>>
>>>
>>> so the Backup was working for a few days and then the error appaers 
>>> again.
>>>
>>> additionally I cannot live migrate the vm:
>>>
>>> 2021-02-18 09:15:11 ERROR: migration aborted (duration 00:00:03): VM
>>> 3091 qmp command 'query-machines' failed - unable to connect to VM 3091
>>> qmp socket - timeout after 31 retries TASK ERROR: migration aborted
>>>
>>> and the same for console
>>>
>>>
>>> best regards
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Backup Failing sporadically due to "PBS backups are not supported by the running QEMU version"

[Fabian Grünbichler]

On February 19, 2021 7:20 am, storm wrote:
> so the last backup during last night failed again with the same error on 
> the same node with pve-qemu-kvm 5.1.0-8 on which the backup was working 
> yesterday
> 
> any ideas?

just to confirm, can you do the following right after a backup fails, 
and then again after you got the VM back into a working state? 123 
should be replaced with the correct VMID..

perl -e 'use strict; use warnings; use PVE::QemuServer::Monitor; use Data::Dumper; $Data::Dumper::Sortkeys=1; print Dumper(PVE::QemuServer::Monitor::qmp_cmd(123, { "execute" => "query-proxmox-support" })), "\n";'

it issues exactly the QMP command that queries the running VM for 
proxmox-related features. it should return right away with output like 
this:

$VAR1 = {
          'pbs-dirty-bitmap' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),
          'pbs-dirty-bitmap-migration' => $VAR1->{'pbs-dirty-bitmap'},
          'pbs-library-version' => '1.0.2 (0723f448fdad1fcf84ec7ae7d702baba11cb06d1)',
          'query-bitmap-info' => $VAR1->{'pbs-dirty-bitmap'}
        };

(the commit hash will be different)

or with some error message.