[PVE-User] Live migration fails with "Mismatched RAM page size ram-node0 (local) 2097152 != 1526773257204281392"

[PVE-User] Live migration fails with "Mismatched RAM page size ram-node0 (local) 2097152 != 1526773257204281392"

[Sergey Korobkov]

Hello,

Two exactly the same machines ( except ram size 380G and 1.5T ).

Upgraded on Debian 10.7 from:
pve-manager/6.1-5/9bf06119
Linux 5.3.13-1-pve #1 SMP PVE 5.3.13-1 (Thu, 05 Dec 2019 07:18:14 +0100)
QEMU emulator version 4.1.1 (pve-qemu-kvm_4.1.1)

to:
pve-manager/6.3-3/eee5f901
Linux 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100)
QEMU emulator version 5.1.0 (pve-qemu-kvm_5.1.0)

We had enabled hugepages for virtual machines( "hugepages: 2" specified 
in virtual machine description).

Live migration fails with errors like this:

Feb 02 16:26:13 QEMU[12090]: kvm7: load of migration failed: Invalid 
argument
Feb 02 16:26:13 QEMU[12090]: kvm7: error while loading state for
instance 0x0 of device 'ram'
Feb 02 16:26:13 QEMU[12090]: kvm7: Mismatched RAM page size ram-node0
(local) 2097152 != 1526773257204281392

We think it's some overflow issue.

------------------
Sincerely yours,
Sergey.

Re: [PVE-User] Live migration fails with "Mismatched RAM page size ram-node0 (local) 2097152 != 1526773257204281392"

[Stefan Reiter]

On 02/02/2021 15:50, Sergey Korobkov wrote:
> Hello,
> 
> Two exactly the same machines ( except ram size 380G and 1.5T ).
> 
> Upgraded on Debian 10.7 from:
> pve-manager/6.1-5/9bf06119
> Linux 5.3.13-1-pve #1 SMP PVE 5.3.13-1 (Thu, 05 Dec 2019 07:18:14 +0100)
> QEMU emulator version 4.1.1 (pve-qemu-kvm_4.1.1)
> 
> to:
> pve-manager/6.3-3/eee5f901
> Linux 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100)
> QEMU emulator version 5.1.0 (pve-qemu-kvm_5.1.0)
> 
> We had enabled hugepages for virtual machines( "hugepages: 2" specified 
> in virtual machine description).
> 
> Live migration fails with errors like this:
> 
> Feb 02 16:26:13 QEMU[12090]: kvm7: load of migration failed: Invalid 
> argument
> Feb 02 16:26:13 QEMU[12090]: kvm7: error while loading state for
> instance 0x0 of device 'ram'
> Feb 02 16:26:13 QEMU[12090]: kvm7: Mismatched RAM page size ram-node0
> (local) 2097152 != 1526773257204281392
> 
> We think it's some overflow issue.
> 

Hi!

After looking carefully I believe to have found the root cause of this 
issue in an upstream bug that we run into since pve-qemu-kvm 5.1.0-4, 
where we started migrating dirty bitmaps.

I have sent a potential fix to the upstream qemu-devel mailing list:
https://lists.nongnu.org/archive/html/qemu-devel/2021-02/msg01711.html

If the resident experts on there agree that this is indeed the solution, 
we will most likely ship it once we release our QEMU 5.2 build :)

Thanks for the report!