[PVE-User] Revisited: External disk backup using PBS - Requesting Criticism/Advice?

[PVE-User] Revisited: External disk backup using PBS - Requesting Criticism/Advice?

[Lindsay Mathieson]

TTL;DR - Backup PC is a standalone Proxmox Server, running a PBS lxc 
container using its host filesystem for the Backup store. The PBS 
container is backed up using vzdump to an attached external hard drive 
which is rotated offsite.


    Source Data

  * 5 Node Proxmox Cluster
  * Ceph storage (size = 3)
  * 30 VM's


    Backup Destination

Standalone Proxmox Server in same server room as main cluster

  * 16 GM Ram
  * CPU: Intel i5
  * Boot - 2 SSD's in ZFS RAID1
  * Data - 2 4TB WD NAS Drives in ZFS RAID1
  * Bonded 1G * 3
  * PBS Container
      o Data Store - 4TB, passed through from Host


    Schedule

  * Proxmox backups up all VM's to PBS Container Weekly
      o Will revisit the schedule
  * Host proxmox server backs up PBS and its data to external hard drive
      o Approx 2 TB Data (we are just an SMB)
      o Drive is rotated offsite


    Recovery

  * VM's can easily be restored from the PBS server as needed (very rare
    occurrence - usually a user messed up their VM)


    Disaster Recovery

This is the real concern - Fire, Theft etc. All servers and data 
including the Standalone Host and PBS server are gone.

  * Recreate Proxmox Cluster
  * Recreate Proxmox PBS Host
      o Restore the PBS Container and data from an offsite backup disk
  * Restore the VM's to the Cluster from the PBS Container

Does all this seem practical and safe?


Thanks - Lindsay



-- 
Lindsay

Re: [PVE-User] Revisited: External disk backup using PBS - Requesting Criticism/Advice?

[Lindsay Mathieson]

On 10/04/2021 5:10 pm, Arjen via pve-user wrote:
> Don't expect to be able to backup the PBS container with 4TB to a 2TB external drive.

I only passed 2TB through and the actual backup data comes to 1.3TB

>   The Datastore of a PSB does not compress much further and Proxmox VE Backup will only backup virtual disks and not mountpoints or storages passed from host, if I understand correctly.


I wondered that. Will be testing.

> I suggest adding a virtual disk of 2TB to the PBS container (and format it with ext4) which can be backed up by the Proxmox VE Backup.


Certainly a possibility.


I also wondered if it was practical to attach an external disk to PBS as 
a Datastore, then detach it. A bit more manual, but doable.


> I would also run the PBS container (with virtual disk) on the cluster instead on separate hardware which is a single point of failure. The local PBS would be then just as reliable as your cluster.


I want to keep the storage separate from the cluster, in that regard the 
local storage is a single point of failure, hence the need for offsite 
storage as well :)


>
> Regarding safeness: I suggest doing a automated disaster recovery every week to make sure it works as expected. Or at least partially, like restoring the PBS from an external drive.


I'll definitely be testing restore options to check that it works.

> Regarding practicality: I would have a remote PBS sync with your local PBS instead of moving physical disks (but you mentioned before that that was not really possible).


Alas :(


Perhaps I could do a backup on site, then physically move it offsite and 
attach it to a offsite PBS server and then sync it remotely - 
incremental backups over the net would be doable.


nb. Our NAS died, hence my increased investigation of this :) Definitely 
want to go with a more open and targeted solution this time, the NAS was 
a good appliance, but inflexible.

Thanks!

-- 
Lindsay

Re: [PVE-User] Revisited: External disk backup using PBS - Requesting Criticism/Advice?

[Lindsay Mathieson]

On 10/04/2021 5:10 pm, Arjen via pve-user wrote:
> Don't expect to be able to backup the PBS container with 4TB to a 2TB external drive. The Datastore of a PSB does not compress much further and Proxmox VE Backup will only backup virtual disks and not mountpoints or storages passed from host, if I understand correctly.
> I suggest adding a virtual disk of 2TB to the PBS container (and format it with ext4) which can be backed up by the Proxmox VE Backup.
> I would also run the PBS container (with virtual disk) on the cluster instead on separate hardware which is a single point of failure. The local PBS would be then just as reliable as your cluster.
>
> Regarding safeness: I suggest doing a automated disaster recovery every week to make sure it works as expected. Or at least partially, like restoring the PBS from an external drive.
> Regarding practicality: I would have a remote PBS sync with your local PBS instead of moving physical disks (but you mentioned before that that was not really possible).


Did some testing over the weekend.

  * Setup a PBS Container with a 2TB root disk
  * Backed up 6 VM's to it.
  * Backed up the PBS Container to a external disk using vzdump (using
    the std proxmox gui)
      o Slooow process at 30MB/s :)
  * Deleted the PBS Container
  * Deleted the backup VM's
  * Restored the PBS container from the external hard disk
      o Much faster - averaged around 100MB/s
  * Restored PBS Container started fine and verified.
  * Restored the backed up VM's from the PBS Container
      o Worked as expected
      o All backup and running.

All in all, worked as I wanted and seems a viable option for full image 
offsite backups via external hard disks.

The process of backing up the PBS container to the external drive is 
*very* slow :( I estimate 11 hours for a full cluster backup copy. But 
since its on a independent node, it doesn't load the main cluster and 
can just happen over the weekend.



-- 
Lindsay