[PVE-User] Live VM migration 6.3 to 6.2 fails

[PVE-User] Live VM migration 6.3 to 6.2 fails

[Christian Balzer]

Hello,

yes I know that migrating back to an older version is not guaranteed but
I'm rather confused as to why this happens.

Firstly this, more details below:
---
2020-12-02 16:34:08 start migrate command to tcp:10.0.0.11:60000
2020-12-02 16:34:09 migration status error: failed
2020-12-02 16:34:09 ERROR: online migrate failure - aborting
2020-12-02 16:34:09 aborting phase 2 - cleanup resources
2020-12-02 16:34:09 migrate_cancel
2020-12-02 16:34:11 ERROR: migration finished with problems (duration 00:00:08)
TASK ERROR: migration problems
---

That's all I get, where would one find the exact issue?

Now for the details, in the previous minor upgrade we saw a similar issue,
but that only affected VMs which had been created on an upgraded node, old
ones were fine to move in both ways.
OK, that was because of new features int he VM definition not present on
the old version, understandable.

With this one on a (more frequently updated) test cluster 6.3 to 6.2
migrations also works, the test cluster old nodes are at 
pve-manager (6.2-12), the one with issues are pve-manager (6.2-6).
I however see nothing in the changelogs which would explain this difference.

In general being able to live migrate back to a node with the previous
version is a very desirable situation, since if there are issues/bugs (on
the VM level) with the new version there no longer is an impact free way
of reverting if this functionality is not present.

Regards,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi@gol.com   	Rakuten Mobile Inc.

Re: [PVE-User] Live VM migration 6.3 to 6.2 fails

[Daniel Berteaud]

----- Le 2 Déc 20, à 9:11, Christian Balzer chibi@gol.com a écrit :

> Hello,
> 
> yes I know that migrating back to an older version is not guaranteed but
> I'm rather confused as to why this happens.
> 
> Firstly this, more details below:
> ---
> 2020-12-02 16:34:08 start migrate command to tcp:10.0.0.11:60000
> 2020-12-02 16:34:09 migration status error: failed
> 2020-12-02 16:34:09 ERROR: online migrate failure - aborting
> 2020-12-02 16:34:09 aborting phase 2 - cleanup resources
> 2020-12-02 16:34:09 migrate_cancel
> 2020-12-02 16:34:11 ERROR: migration finished with problems (duration 00:00:08)
> TASK ERROR: migration problems
> ---
> 
> That's all I get, where would one find the exact issue?
> 
> Now for the details, in the previous minor upgrade we saw a similar issue,
> but that only affected VMs which had been created on an upgraded node, old
> ones were fine to move in both ways.
> OK, that was because of new features int he VM definition not present on
> the old version, understandable.
> 
> With this one on a (more frequently updated) test cluster 6.3 to 6.2
> migrations also works, the test cluster old nodes are at
> pve-manager (6.2-12), the one with issues are pve-manager (6.2-6).
> I however see nothing in the changelogs which would explain this difference.
> 
> In general being able to live migrate back to a node with the previous
> version is a very desirable situation, since if there are issues/bugs (on
> the VM level) with the new version there no longer is an impact free way
> of reverting if this functionality is not present.
> 


That's most likely the QEMU version 5.1 on the 6.3 node vs QEMU 5.0 on the 6.2 one.
There's no migration support from newer to older, only from older to newer.

Cheers,
Daniel

-- 
[ https://www.firewall-services.com/ ] 	
Daniel Berteaud 
FIREWALL-SERVICES SAS, La sécurité des réseaux 
Société de Services en Logiciels Libres 
Tél : +33.5 56 64 15 32 
Matrix: @dani:fws.fr 
[ https://www.firewall-services.com/ | https://www.firewall-services.com ]

Re: [PVE-User] Live VM migration 6.3 to 6.2 fails

[Christian Balzer]

On Wed, 2 Dec 2020 09:17:44 +0100 (CET) Daniel Berteaud wrote:

> ----- Le 2 Déc 20, à 9:11, Christian Balzer chibi@gol.com a écrit :
> 
> > Hello,
> > 
> > yes I know that migrating back to an older version is not guaranteed but
> > I'm rather confused as to why this happens.
> > 
> > Firstly this, more details below:
> > ---
> > 2020-12-02 16:34:08 start migrate command to tcp:10.0.0.11:60000
> > 2020-12-02 16:34:09 migration status error: failed
> > 2020-12-02 16:34:09 ERROR: online migrate failure - aborting
> > 2020-12-02 16:34:09 aborting phase 2 - cleanup resources
> > 2020-12-02 16:34:09 migrate_cancel
> > 2020-12-02 16:34:11 ERROR: migration finished with problems (duration 00:00:08)
> > TASK ERROR: migration problems
> > ---
> > 
> > That's all I get, where would one find the exact issue?
> > 
> > Now for the details, in the previous minor upgrade we saw a similar issue,
> > but that only affected VMs which had been created on an upgraded node, old
> > ones were fine to move in both ways.
> > OK, that was because of new features int he VM definition not present on
> > the old version, understandable.
> > 
> > With this one on a (more frequently updated) test cluster 6.3 to 6.2
> > migrations also works, the test cluster old nodes are at
> > pve-manager (6.2-12), the one with issues are pve-manager (6.2-6).
> > I however see nothing in the changelogs which would explain this difference.
> > 
> > In general being able to live migrate back to a node with the previous
> > version is a very desirable situation, since if there are issues/bugs (on
> > the VM level) with the new version there no longer is an impact free way
> > of reverting if this functionality is not present.
> >   
> 
> 
> That's most likely the QEMU version 5.1 on the 6.3 node vs QEMU 5.0 on the 6.2 one.
> There's no migration support from newer to older, only from older to newer.
> 
If that is really the case (qemu migrations between minor versions have
worked in the past, at least sometimes), then this is actually worse.

As I wrote, some 6.2 nodes don't have that issue and indeed they have qemu
5.1 which was brought in with 6.2-13. 
So no version upgrade flashing warning lights going off in your brain and
if one had not upgraded all nodes, the same would have happened in a
cluster that was nominally still 6.2?
That's rather... scary.

Christian

> Cheers,
> Daniel
> 
> -- 
> [ https://www.firewall-services.com/ ] 	
> Daniel Berteaud 
> FIREWALL-SERVICES SAS, La sécurité des réseaux 
> Société de Services en Logiciels Libres 
> Tél : +33.5 56 64 15 32 
> Matrix: @dani:fws.fr 
> [ https://www.firewall-services.com/ | https://www.firewall-services.com ]
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user


-- 
Christian Balzer        Network/Systems Engineer                
chibi@gol.com   	Rakuten Mobile Inc.

Re: [PVE-User] Live VM migration 6.3 to 6.2 fails

[Daniel Berteaud]

----- Le 2 Déc 20, à 9:47, Christian Balzer chibi@gol.com a écrit :

>> 
> If that is really the case (qemu migrations between minor versions have
> worked in the past, at least sometimes), then this is actually worse.
> 
> As I wrote, some 6.2 nodes don't have that issue and indeed they have qemu
> 5.1 which was brought in with 6.2-13.
> So no version upgrade flashing warning lights going off in your brain and
> if one had not upgraded all nodes, the same would have happened in a
> cluster that was nominally still 6.2?
> That's rather... scary.
> 

If you got QEMU 5.1 with pve 6.2, I guess you're using either the pvetest or no-subscription repo. In which case, it's up to you to check for changelog and important version changes, as those repo are kinda "rolling updates"
If you want peace of mind on this front, you should pay access to the enterprise repo, which have more stable updates

Regards,
Daniel

-- 
[ https://www.firewall-services.com/ ] 	
Daniel Berteaud 
FIREWALL-SERVICES SAS, La sécurité des réseaux 
Société de Services en Logiciels Libres 
Tél : +33.5 56 64 15 32 
Matrix: @dani:fws.fr 
[ https://www.firewall-services.com/ | https://www.firewall-services.com ]

Re: [PVE-User] Live VM migration 6.3 to 6.2 fails

[alexandre derumier]

On 02/12/2020 09:11, Christian Balzer wrote:
> Hello,
>
> yes I know that migrating back to an older version is not guaranteed but
> I'm rather confused as to why this happens.
>
> Firstly this, more details below:
> ---
> 2020-12-02 16:34:08 start migrate command to tcp:10.0.0.11:60000
> 2020-12-02 16:34:09 migration status error: failed
> 2020-12-02 16:34:09 ERROR: online migrate failure - aborting
> 2020-12-02 16:34:09 aborting phase 2 - cleanup resources
> 2020-12-02 16:34:09 migrate_cancel
> 2020-12-02 16:34:11 ERROR: migration finished with problems (duration 00:00:08)
> TASK ERROR: migration problems
> ---
>
> That's all I get, where would one find the exact issue?
>
> Now for the details, in the previous minor upgrade we saw a similar issue,
> but that only affected VMs which had been created on an upgraded node, old
> ones were fine to move in both ways.
> OK, that was because of new features int he VM definition not present on
> the old version, understandable.
>
> With this one on a (more frequently updated) test cluster 6.3 to 6.2
> migrations also works, the test cluster old nodes are at
> pve-manager (6.2-12), the one with issues are pve-manager (6.2-6).
> I however see nothing in the changelogs which would explain this difference.
>
> In general being able to live migrate back to a node with the previous
> version is a very desirable situation, since if there are issues/bugs (on
> the VM level) with the new version there no longer is an impact free way
> of reverting if this functionality is not present.
>
> Regards,
>
> Christian

you can't migrate from a newer qemu version to an older qemu version,

if the vm start has been started first on the newer qemu version.


if the vm was started first on an older qemu version,

you can live migrate it to a newer qemu version

and migrate it again to older qemu version.

(until you stop/start it again on a newer qemu version).


This is because, when the vm is started, it's keeping internally the 
qemu machine version

of the current qemu process version.


The only thing to keep in mind, try to keep all yours nodes at same version.

Proxmox is a rolling release, you can have qemu upgrade between 2 releases.