Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

From: Adam Kalisz <adam.kalisz@notnullmakers.com>
To: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>,
	pbs-devel@lists.proxmox.com
Cc: Proxmox VE development discussion <pve-devel@lists.proxmox.com>,
	Thomas Lamprecht <t.lamprecht@proxmox.com>
Subject: Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere
Date: Fri, 08 Aug 2025 15:08:26 +0200	[thread overview]
Message-ID: <010164753bbda5705109d2e192f9d2a361205049.camel@notnullmakers.com> (raw)
In-Reply-To: <1753708847.wueupzjnbf.astroid@yuna.none>

Hi,

any news about the chunk verification speed-up?

On Mon, 2025-07-28 at 15:22 +0200, Fabian Grünbichler wrote:
> On July 28, 2025 3:14 pm, Adam Kalisz wrote:
> > On Fri, 2025-07-25 at 14:46 +0200, Fabian Grünbichler wrote:
> > > On July 25, 2025 1:23 pm, Adam Kalisz wrote:
> > > > Similarly the sync performance between two Proxmox Backup
> > > > Servers and live-migration got mentioned in various places.
> > > 
> > > do you mean live-restore here? live migration has nothing to do
> > > with PBS..
> > 
> > I threw more things into a single basked as the e-mail was sent to
> > both lists and meant live migration that seems like it could go
> > faster in some cases. (But that might be code that originates in
> > the QEMU project, right?) We could of course have a look at live-
> > restore or even some things like migration from VMware/ ESXi too
> > (again, unrelated to PBS).
> 
> yes, live migration also has some performance optimizations that we
> want to check out, in particular the multi-FD feature:
> 
> https://bugzilla.proxmox.com/show_bug.cgi?id=5766
> 
> live restore uses an entirely different mechanism, and migration from
> ESXi uses yet another mechanism that is bottle-necked by the API on
> the other side at the moment.

The guys over at XCP-ng seem to have switched to VDDK. I just thought I
would point to it for reference:

"Our test VM used to take 25 minutes to migrate, now it's done in under
2 🤯 . Same VM, same setup, just a much better approach.

What changed? We stopped using the native ESXi API to pull disks and
switched to VDDK (VMware’s own library for accessing VM data), which is
way more efficient.

Speed is one thing, but it’s not the only win. We now transfer only the
used blocks, without reading the whole disk first, which cuts down
migration time significantly. VDDK also brings back warm migration
support (even on recent ESXi versions), and keeps the load on the ESXi
side barely noticeable (unlike before, where it could freeze the API
entirely)."

https://www.linkedin.com/feed/update/urn:li:activity:7359195462538604547/

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel