Re: [PVE-User] PBS backup/restore issue

Re: [PVE-User] PBS backup/restore issue

[Stefan Reiter]

On 2/23/21 8:46 AM, Eneko Lacunza wrote:
> Hi all,
> 
> Yesterday I hit a problem with PBS. My sequence of actions was:
> 
> 1.a Install debian 10 VM (updated to latest, only system utilities and 
> SSH server)
> 1.b Add SSH access key to root
> 1.c Shutdown VM
> 1.d Backup to PBS and to NFS vía regular backup.
> 1.e Start VM and install various packages - all OK.
> 1.f Stop VM
> 
> 2.a Restore VM from PBS backup
> 2.b Start VM and install various packages - apt complains about no 
> newline in linux-image-amd64 file manifest, can't install new packages.
> 2.c Stop VM
> 
> 3.a Restore VM from NFS regular backup
> 3.b Start VM and install various packages - all OK.
> 

Hi,

I tried to replicate your problem by following pretty much the exact 
steps you mention above, but to no avail.

Fresh Debian 10 VM, PBS backup + VMA to local disk, restored both and 
both work. I compared the disk images of the restored VMs with md5sum 
and the hashes matched, as expected. Tested with the latest package 
versions, though using QEMU 5.2.0, but the backup and restore 
functionality shouldn't have changed there.

Could you try to see what the difference is between the restored disks 
from the VMA and the PBS backups (if there is any)? I.e. restore them 
both to seperate VMIDs and then compare hashes and/or run (v)bindiff on 
the disk images (best restore to a directory storage in 'raw' mode) to 
see where they differ - of course, this only makes sense if the backups 
are created when the VM was shut down.

Potentially something about your NFS/PBS setup is corrupting data? You 
mentioned in another mail that they "are the same disks", how is that 
configured?

Also please post the exact error messages APT is giving you, and your VM 
config.

> I hit the issue 3 times, and tried creating a new VM and a new Debian 
> installation each time.
> 
> Any ideas? I have the "broken" PBS backups saved.
> 
> VM node:
> # pveversion -v [...]
> 
> PBS node:
> # pveversion -v [...]

FYI, the more useful command for PBS would be:
proxmox-backup-manager versions --verbose

Re: [PVE-User] PBS backup/restore issue

[Thomas Lamprecht]

Hi!

On 24.02.21 12:53, Eneko Lacunza wrote:
> El 24/2/21 a las 11:34, Stefan Reiter escribió:
>> On 2/23/21 8:46 AM, Eneko Lacunza wrote:
>>> Yesterday I hit a problem with PBS. My sequence of actions was:
>>>
>>> 1.a Install debian 10 VM (updated to latest, only system utilities and SSH server)
>>> 1.b Add SSH access key to root
>>> 1.c Shutdown VM
>>> 1.d Backup to PBS and to NFS vía regular backup.
>>> 1.e Start VM and install various packages - all OK.
>>> 1.f Stop VM
>>>
>>> 2.a Restore VM from PBS backup
>>> 2.b Start VM and install various packages - apt complains about no newline in linux-image-amd64 file manifest, can't install new packages.
>>> 2.c Stop VM
>>>
>>> 3.a Restore VM from NFS regular backup
>>> 3.b Start VM and install various packages - all OK.
>>>
[snip]

> My original restores were to a ceph storage (r2 on HDD with SSD journal/DB). So i have restored again both, to that ceph storage:
[snip]

> 
> Hope this info helps you to replicate the issue.

FYI: Stefan could replicate it and send a patch[0] which is included now.

It boiled down to caches not being flushed to disk for storage attached through
user-space libraries (vs. kernel) by QEMU in the PBS restore case.

In your case you probably do not have the "KRBD" checkbox ticked for the Ceph
storage entry in PVE, so the user-space librbd is used for accessing that storage
through QEMU, where this issue then hit on restore.

If you want to try the fix out, there's a new QEMU package available on pvetest:

http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/pve-qemu-kvm_5.2.0-2_amd64.deb
sha256sum: 6a086e0b669447f0b8a7899f9194a26785e6b0070f3dec1be4f95067bc707c7a

cheers,
Thomas

[0]: https://lists.proxmox.com/pipermail/pve-devel/2021-February/047131.html

Re: [PVE-User] PBS backup/restore issue

[Dietmar Maurer]

> I hit the issue 3 times, and tried creating a new VM and a new Debian 
> installation each time.
> 
> Any ideas? I have the "broken" PBS backups saved.

Already tried to verify the datastore? (Datastore Verify is OK?)

Re: [PVE-User] PBS backup/restore issue

[mj]

Hi Eneko,

Just to say: that sounds scary! Do you have any update on this..?

MJ

Re: [PVE-User] PBS backup/restore issue

[proxmox]

Any update on this? 

I had read an issue with pbs and restore of vms stored on ceph. Which is hopefully solved in actual verion of pbs and qemu...

Kind regards
Mehmet

Am 24. Februar 2021 11:06:14 MEZ schrieb Eneko Lacunza via pve-user <pve-user@lists.proxmox.com>:
>_______________________________________________
>pve-user mailing list
>pve-user@lists.proxmox.com
>https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user