Re: [pve-devel] [PATCH storage v4 09/27] plugin: introduce new_backup_provider() method

From: Andreas Rogge <andreas.rogge@bareos.com>
To: Fiona Ebner <f.ebner@proxmox.com>,
	Proxmox VE development discussion <pve-devel@lists.proxmox.com>,
	Thomas Lamprecht <t.lamprecht@proxmox.com>
Cc: Philipp Storz <philipp.storz@bareos.com>
Subject: Re: [pve-devel] [PATCH storage v4 09/27] plugin: introduce new_backup_provider() method
Date: Thu, 3 Apr 2025 10:06:01 +0200	[thread overview]
Message-ID: <c3d3d434-cf68-422a-8a28-35ee5531eaca@bareos.com> (raw)
In-Reply-To: <324b156c-0b12-47e9-bffb-403a733c2b7d@proxmox.com>

Am 02.04.25 um 10:33 schrieb Fiona Ebner:
> Am 01.04.25 um 20:21 schrieb Thomas Lamprecht:
>> Am 01.04.25 um 18:02 schrieb Andreas Rogge:
>>> Um... that has nothing to do with what you provided when we take the
>>> backup. Is there a reason PVE cannot provide a writeable block device to
>>> restore to?
> 
> Good idea! I'll look into adding a second restore mechanism with this
> approach.

That would be really great!

> 
>>> For Bareos this requirement would imply that we need an unpleasantly
>>> large staging area on the PVE node to facilitate a restore: As we can
>>> only stream the data we'd have to create a temporary disk image that PVE
>>> can then read. This sounds pretty inefficient - especially when
>>> comparing with qmrestore's ability to just read read from stdin.
>>
>> Bareos could provide a NBD that is streamed directly from the backup
>> repository, this should be quite efficient w.r.t. space usage.
>> But as this was v4 and a few things changed, and I'm less involved as
>> the actual author it might make more sense for her to answer this.
> 
> It's still mostly the same in the current version. But yes, you can
> provide an NBD URI or path to a FUSE virtual file to avoid the need for
> a temporary image.
The main problem here is that Bareos cannot provide random access to the 
backed up data. When restoring, Bareos provides a linear data-stream 
containing data and metadata to the backup agent. In case of layered 
backups (i.e. 1 full + 1 differential + N incrementals) data will be 
restored out of order: first data from the full, then data from the 
differential and each incremental.
As a result Bareos would have to stall read access to regions that are 
not yet restored until the data was restored.

The worst-case scenario would be a a full backup and an incremental that 
contains only an updated first sector of the virual hard drive. In this 
scenario Bareos has to restore everything but the first sector from the 
full and will only then restore the first sector from the incremental.
As a result Bareos would have to stage the whole virtual disk in a 
temporary location before qemu could read the first sector of that 
virtual disk. This not only wastes space as the staging-area must be 
large enough for that worst-case (and we're not yet talking about 
running multiple restore jobs in parallel), but also wastes time and 
resources as we're moving the data twice (Bareos -> staging -> PVE's 
storage).

While this is clearly suboptimal when compared to what PBS does, it is 
the only decent way to support direct restore from layered tape backups.

Best Regards,
Andreas

-- 
Andreas Rogge                             andreas.rogge@bareos.com
   Bareos GmbH & Co. KG                      Phone: +49 221-630693-86
   http://www.bareos.com

   Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
   Komplementär: Bareos Verwaltungs-GmbH
   Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel