public inbox for pve-user@lists.proxmox.com
 help / color / mirror / Atom feed
* Re: [PVE-User] PBS backup/restore issue
       [not found] <c390959d-e5b5-9a95-770c-ff6853357cbe@binovo.es>
@ 2021-02-24 10:34 ` Stefan Reiter
       [not found]   ` <6120d3f8-67c8-873f-164c-1913c2016a54@binovo.es>
  0 siblings, 1 reply; 5+ messages in thread
From: Stefan Reiter @ 2021-02-24 10:34 UTC (permalink / raw)
  To: Eneko Lacunza, pve-user

On 2/23/21 8:46 AM, Eneko Lacunza wrote:
> Hi all,
> 
> Yesterday I hit a problem with PBS. My sequence of actions was:
> 
> 1.a Install debian 10 VM (updated to latest, only system utilities and 
> SSH server)
> 1.b Add SSH access key to root
> 1.c Shutdown VM
> 1.d Backup to PBS and to NFS vía regular backup.
> 1.e Start VM and install various packages - all OK.
> 1.f Stop VM
> 
> 2.a Restore VM from PBS backup
> 2.b Start VM and install various packages - apt complains about no 
> newline in linux-image-amd64 file manifest, can't install new packages.
> 2.c Stop VM
> 
> 3.a Restore VM from NFS regular backup
> 3.b Start VM and install various packages - all OK.
> 

Hi,

I tried to replicate your problem by following pretty much the exact 
steps you mention above, but to no avail.

Fresh Debian 10 VM, PBS backup + VMA to local disk, restored both and 
both work. I compared the disk images of the restored VMs with md5sum 
and the hashes matched, as expected. Tested with the latest package 
versions, though using QEMU 5.2.0, but the backup and restore 
functionality shouldn't have changed there.

Could you try to see what the difference is between the restored disks 
from the VMA and the PBS backups (if there is any)? I.e. restore them 
both to seperate VMIDs and then compare hashes and/or run (v)bindiff on 
the disk images (best restore to a directory storage in 'raw' mode) to 
see where they differ - of course, this only makes sense if the backups 
are created when the VM was shut down.

Potentially something about your NFS/PBS setup is corrupting data? You 
mentioned in another mail that they "are the same disks", how is that 
configured?

Also please post the exact error messages APT is giving you, and your VM 
config.

> I hit the issue 3 times, and tried creating a new VM and a new Debian 
> installation each time.
> 
> Any ideas? I have the "broken" PBS backups saved.
> 
> VM node:
> # pveversion -v [...]
> 
> PBS node:
> # pveversion -v [...]

FYI, the more useful command for PBS would be:
proxmox-backup-manager versions --verbose




^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PVE-User] PBS backup/restore issue
       [not found]   ` <6120d3f8-67c8-873f-164c-1913c2016a54@binovo.es>
@ 2021-02-24 19:03     ` Thomas Lamprecht
  0 siblings, 0 replies; 5+ messages in thread
From: Thomas Lamprecht @ 2021-02-24 19:03 UTC (permalink / raw)
  To: Eneko Lacunza, Stefan Reiter, pve-user

Hi!

On 24.02.21 12:53, Eneko Lacunza wrote:
> El 24/2/21 a las 11:34, Stefan Reiter escribió:
>> On 2/23/21 8:46 AM, Eneko Lacunza wrote:
>>> Yesterday I hit a problem with PBS. My sequence of actions was:
>>>
>>> 1.a Install debian 10 VM (updated to latest, only system utilities and SSH server)
>>> 1.b Add SSH access key to root
>>> 1.c Shutdown VM
>>> 1.d Backup to PBS and to NFS vía regular backup.
>>> 1.e Start VM and install various packages - all OK.
>>> 1.f Stop VM
>>>
>>> 2.a Restore VM from PBS backup
>>> 2.b Start VM and install various packages - apt complains about no newline in linux-image-amd64 file manifest, can't install new packages.
>>> 2.c Stop VM
>>>
>>> 3.a Restore VM from NFS regular backup
>>> 3.b Start VM and install various packages - all OK.
>>>
[snip]

> My original restores were to a ceph storage (r2 on HDD with SSD journal/DB). So i have restored again both, to that ceph storage:
[snip]

> 
> Hope this info helps you to replicate the issue.

FYI: Stefan could replicate it and send a patch[0] which is included now.

It boiled down to caches not being flushed to disk for storage attached through
user-space libraries (vs. kernel) by QEMU in the PBS restore case.

In your case you probably do not have the "KRBD" checkbox ticked for the Ceph
storage entry in PVE, so the user-space librbd is used for accessing that storage
through QEMU, where this issue then hit on restore.

If you want to try the fix out, there's a new QEMU package available on pvetest:

http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/pve-qemu-kvm_5.2.0-2_amd64.deb
sha256sum: 6a086e0b669447f0b8a7899f9194a26785e6b0070f3dec1be4f95067bc707c7a

cheers,
Thomas

[0]: https://lists.proxmox.com/pipermail/pve-devel/2021-February/047131.html




^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PVE-User] PBS backup/restore issue
       [not found]   ` <mailman.44.1614161184.365.pve-user@lists.proxmox.com>
@ 2021-06-13 18:36     ` proxmox
  0 siblings, 0 replies; 5+ messages in thread
From: proxmox @ 2021-06-13 18:36 UTC (permalink / raw)
  To: Proxmox VE user list, Eneko Lacunza via pve-user, pve-user

Any update on this? 

I had read an issue with pbs and restore of vms stored on ceph. Which is hopefully solved in actual verion of pbs and qemu...

Kind regards
Mehmet

Am 24. Februar 2021 11:06:14 MEZ schrieb Eneko Lacunza via pve-user <pve-user@lists.proxmox.com>:
>_______________________________________________
>pve-user mailing list
>pve-user@lists.proxmox.com
>https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PVE-User] PBS backup/restore issue
       [not found] <mailman.28.1614066435.365.pve-user@lists.proxmox.com>
  2021-02-23  8:06 ` Dietmar Maurer
@ 2021-02-24  9:44 ` mj
       [not found]   ` <mailman.44.1614161184.365.pve-user@lists.proxmox.com>
  1 sibling, 1 reply; 5+ messages in thread
From: mj @ 2021-02-24  9:44 UTC (permalink / raw)
  To: pve-user

Hi Eneko,

Just to say: that sounds scary! Do you have any update on this..?

MJ



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PVE-User] PBS backup/restore issue
       [not found] <mailman.28.1614066435.365.pve-user@lists.proxmox.com>
@ 2021-02-23  8:06 ` Dietmar Maurer
  2021-02-24  9:44 ` mj
  1 sibling, 0 replies; 5+ messages in thread
From: Dietmar Maurer @ 2021-02-23  8:06 UTC (permalink / raw)
  To: Proxmox VE user list

> I hit the issue 3 times, and tried creating a new VM and a new Debian 
> installation each time.
> 
> Any ideas? I have the "broken" PBS backups saved.

Already tried to verify the datastore? (Datastore Verify is OK?)




^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-06-13 18:42 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <c390959d-e5b5-9a95-770c-ff6853357cbe@binovo.es>
2021-02-24 10:34 ` [PVE-User] PBS backup/restore issue Stefan Reiter
     [not found]   ` <6120d3f8-67c8-873f-164c-1913c2016a54@binovo.es>
2021-02-24 19:03     ` Thomas Lamprecht
     [not found] <mailman.28.1614066435.365.pve-user@lists.proxmox.com>
2021-02-23  8:06 ` Dietmar Maurer
2021-02-24  9:44 ` mj
     [not found]   ` <mailman.44.1614161184.365.pve-user@lists.proxmox.com>
2021-06-13 18:36     ` proxmox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal