[pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Adam Kalisz]

Hi list,

I missed whether the chunk verification speedup when loading chunks got
applied or whether it was somehow included in the S3-like storage
option change set.

In
https://forum.proxmox.com/threads/abysmally-slow-restore-from-backup.133602/page-7
we have discussed some other opportunities for speedup using similar
patterns. People mentioned LXC container restore speed and  host-based
backup restore, which looking at the code for the latter seems like a
similar async loop pattern would bring some improvement without too
much trouble:

https://github.com/proxmox/proxmox-backup/blob/4940514b0f05d6cd6a5f711edfdd47c1fa41b537/proxmox-backup-client/src/main.rs#L1109

Similarly the sync performance between two Proxmox Backup Servers and
live-migration got mentioned in various places.

Btw. congrats on the 4.0 beta.

Regards
Adam

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Fabian Grünbichler]

On July 25, 2025 1:23 pm, Adam Kalisz wrote:
> Hi list,
> 
> I missed whether the chunk verification speedup when loading chunks got
> applied or whether it was somehow included in the S3-like storage
> option change set.

https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/

hasn't been applied (and probably needs a rebase post-S3 ;))

> In
> https://forum.proxmox.com/threads/abysmally-slow-restore-from-backup.133602/page-7
> we have discussed some other opportunities for speedup using similar
> patterns. People mentioned LXC container restore speed and  host-based
> backup restore, which looking at the code for the latter seems like a
> similar async loop pattern would bring some improvement without too
> much trouble:
> 
> https://github.com/proxmox/proxmox-backup/blob/4940514b0f05d6cd6a5f711edfdd47c1fa41b537/proxmox-backup-client/src/main.rs#L1109

that's not the code path for container/host backups, that's the code
path for fixed-size indices like vm backups, but just dumping to a file
instead of via qmrestore.. i.e., nothing that is used by PVE or users
usually, but can still be improved of course ;)

> Similarly the sync performance between two Proxmox Backup Servers and
> live-migration got mentioned in various places.

do you mean live-restore here? live migration has nothing to do with
PBS..


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pve-devel] [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Adam Kalisz via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 8405 bytes --]

From: Adam Kalisz <adam.kalisz@notnullmakers.com>
To: pbs-devel@lists.proxmox.com
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "Fabian Grünbichler" <f.gruenbichler@proxmox.com>, "Thomas Lamprecht" <t.lamprecht@proxmox.com>, "Dominik Csapak" <d.csapak@proxmox.com>
Subject: Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere
Date: Mon, 28 Jul 2025 15:14:47 +0200
Message-ID: <874cbfc3d659edb7c8ebdda71e77530e437065a2.camel@notnullmakers.com>

On Fri, 2025-07-25 at 14:46 +0200, Fabian Grünbichler wrote:
> On July 25, 2025 1:23 pm, Adam Kalisz wrote:
> > Hi list,
> > 
> > I missed whether the chunk verification speedup when loading chunks
> > got
> > applied or whether it was somehow included in the S3-like storage
> > option change set.
> 
> https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/
> 
> hasn't been applied (and probably needs a rebase post-S3 ;))

Yes and the opt-out knob that Thomas mentioned on Friday should be
added:

"So the mentioned series should IMO only be applied if it can be opted
out, or, better in the long term, a pacing/IO bandwidth limit can be
enforced. That is naturally a big amount of work, mostly in evaluating
different options and deciding for what can make sense, that's why I
mentioned that being able to opt-out could be enough, then admins
could at least keep the status quo."

> > In
> > https://forum.proxmox.com/threads/abysmally-slow-restore-from-backup.133602/page-7
> > we have discussed some other opportunities for speedup using
> > similar
> > patterns. People mentioned LXC container restore speed and  host-
> > based
> > backup restore, which looking at the code for the latter seems like
> > a
> > similar async loop pattern would bring some improvement without too
> > much trouble:
> > 
> > https://github.com/proxmox/proxmox-backup/blob/4940514b0f05d6cd6a5f711edfdd47c1fa41b537/proxmox-backup-client/src/main.rs#L1109
> 
> that's not the code path for container/host backups, that's the code
> path for fixed-size indices like vm backups, but just dumping to a
> file instead of via qmrestore.. i.e., nothing that is used by PVE or
> users usually, but can still be improved of course ;)

Ah ok, I just skimmed over it to find patterns that look like the
things Dominik and I improved before elsewhere.

> > Similarly the sync performance between two Proxmox Backup Servers
> > and
> > live-migration got mentioned in various places.
> 
> do you mean live-restore here? live migration has nothing to do with
> PBS..

I threw more things into a single basked as the e-mail was sent to both
lists and meant live migration that seems like it could go faster in
some cases. (But that might be code that originates in the QEMU
project, right?) We could of course have a look at live-restore or even
some things like migration from VMware/ ESXi too (again, unrelated to
PBS).

I just want to confirm if any of this makes sense and where to look
basically so we can have a list of areas for improvement and what code
path that is.

The speed improvements already caused qualitative improvements that
e.g. allow some companies to choose PVE+PBS instead of other options
even in more demanding cases.

Thank you for bringing me up to speed. Have a successful week!


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Adam Kalisz]

On Fri, 2025-07-25 at 14:46 +0200, Fabian Grünbichler wrote:
> On July 25, 2025 1:23 pm, Adam Kalisz wrote:
> > Hi list,
> > 
> > I missed whether the chunk verification speedup when loading chunks
> > got
> > applied or whether it was somehow included in the S3-like storage
> > option change set.
> 
> https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/
> 
> hasn't been applied (and probably needs a rebase post-S3 ;))

Yes and the opt-out knob that Thomas mentioned on Friday should be
added:

"So the mentioned series should IMO only be applied if it can be opted
out, or, better in the long term, a pacing/IO bandwidth limit can be
enforced. That is naturally a big amount of work, mostly in evaluating
different options and deciding for what can make sense, that's why I
mentioned that being able to opt-out could be enough, then admins
could at least keep the status quo."

> > In
> > https://forum.proxmox.com/threads/abysmally-slow-restore-from-backup.133602/page-7
> > we have discussed some other opportunities for speedup using
> > similar
> > patterns. People mentioned LXC container restore speed and  host-
> > based
> > backup restore, which looking at the code for the latter seems like
> > a
> > similar async loop pattern would bring some improvement without too
> > much trouble:
> > 
> > https://github.com/proxmox/proxmox-backup/blob/4940514b0f05d6cd6a5f711edfdd47c1fa41b537/proxmox-backup-client/src/main.rs#L1109
> 
> that's not the code path for container/host backups, that's the code
> path for fixed-size indices like vm backups, but just dumping to a
> file instead of via qmrestore.. i.e., nothing that is used by PVE or
> users usually, but can still be improved of course ;)

Ah ok, I just skimmed over it to find patterns that look like the
things Dominik and I improved before elsewhere.

> > Similarly the sync performance between two Proxmox Backup Servers
> > and
> > live-migration got mentioned in various places.
> 
> do you mean live-restore here? live migration has nothing to do with
> PBS..

I threw more things into a single basked as the e-mail was sent to both
lists and meant live migration that seems like it could go faster in
some cases. (But that might be code that originates in the QEMU
project, right?) We could of course have a look at live-restore or even
some things like migration from VMware/ ESXi too (again, unrelated to
PBS).

I just want to confirm if any of this makes sense and where to look
basically so we can have a list of areas for improvement and what code
path that is.

The speed improvements already caused qualitative improvements that
e.g. allow some companies to choose PVE+PBS instead of other options
even in more demanding cases.

Thank you for bringing me up to speed. Have a successful week!

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Fabian Grünbichler]

On July 28, 2025 3:14 pm, Adam Kalisz wrote:
> On Fri, 2025-07-25 at 14:46 +0200, Fabian Grünbichler wrote:
>> On July 25, 2025 1:23 pm, Adam Kalisz wrote:
>> > Similarly the sync performance between two Proxmox Backup Servers
>> > and
>> > live-migration got mentioned in various places.
>> 
>> do you mean live-restore here? live migration has nothing to do with
>> PBS..
> 
> I threw more things into a single basked as the e-mail was sent to both
> lists and meant live migration that seems like it could go faster in
> some cases. (But that might be code that originates in the QEMU
> project, right?) We could of course have a look at live-restore or even
> some things like migration from VMware/ ESXi too (again, unrelated to
> PBS).

yes, live migration also has some performance optimizations that we want
to check out, in particular the multi-FD feature:

https://bugzilla.proxmox.com/show_bug.cgi?id=5766

live restore uses an entirely different mechanism, and migration from
ESXi uses yet another mechanism that is bottle-necked by the API on the
other side at the moment.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pve-devel] [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Fabian Grünbichler]

On July 28, 2025 3:14 pm, Adam Kalisz wrote:
> On Fri, 2025-07-25 at 14:46 +0200, Fabian Grünbichler wrote:
>> On July 25, 2025 1:23 pm, Adam Kalisz wrote:
>> > Similarly the sync performance between two Proxmox Backup Servers
>> > and
>> > live-migration got mentioned in various places.
>> 
>> do you mean live-restore here? live migration has nothing to do with
>> PBS..
> 
> I threw more things into a single basked as the e-mail was sent to both
> lists and meant live migration that seems like it could go faster in
> some cases. (But that might be code that originates in the QEMU
> project, right?) We could of course have a look at live-restore or even
> some things like migration from VMware/ ESXi too (again, unrelated to
> PBS).

yes, live migration also has some performance optimizations that we want
to check out, in particular the multi-FD feature:

https://bugzilla.proxmox.com/show_bug.cgi?id=5766

live restore uses an entirely different mechanism, and migration from
ESXi uses yet another mechanism that is bottle-necked by the API on the
other side at the moment.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Adam Kalisz via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 8244 bytes --]

From: Adam Kalisz <adam.kalisz@notnullmakers.com>
To: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>, pbs-devel@lists.proxmox.com
Cc: Dominik Csapak <d.csapak@proxmox.com>, Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Thomas Lamprecht <t.lamprecht@proxmox.com>
Subject: Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere
Date: Fri, 08 Aug 2025 15:08:26 +0200
Message-ID: <010164753bbda5705109d2e192f9d2a361205049.camel@notnullmakers.com>

Hi,

any news about the chunk verification speed-up?

On Mon, 2025-07-28 at 15:22 +0200, Fabian Grünbichler wrote:
> On July 28, 2025 3:14 pm, Adam Kalisz wrote:
> > On Fri, 2025-07-25 at 14:46 +0200, Fabian Grünbichler wrote:
> > > On July 25, 2025 1:23 pm, Adam Kalisz wrote:
> > > > Similarly the sync performance between two Proxmox Backup
> > > > Servers and live-migration got mentioned in various places.
> > > 
> > > do you mean live-restore here? live migration has nothing to do
> > > with PBS..
> > 
> > I threw more things into a single basked as the e-mail was sent to
> > both lists and meant live migration that seems like it could go
> > faster in some cases. (But that might be code that originates in
> > the QEMU project, right?) We could of course have a look at live-
> > restore or even some things like migration from VMware/ ESXi too
> > (again, unrelated to PBS).
> 
> yes, live migration also has some performance optimizations that we
> want to check out, in particular the multi-FD feature:
> 
> https://bugzilla.proxmox.com/show_bug.cgi?id=5766
> 
> live restore uses an entirely different mechanism, and migration from
> ESXi uses yet another mechanism that is bottle-necked by the API on
> the other side at the moment.

The guys over at XCP-ng seem to have switched to VDDK. I just thought I
would point to it for reference:

"Our test VM used to take 25 minutes to migrate, now it's done in under
2 🤯 . Same VM, same setup, just a much better approach.

What changed? We stopped using the native ESXi API to pull disks and
switched to VDDK (VMware’s own library for accessing VM data), which is
way more efficient.

Speed is one thing, but it’s not the only win. We now transfer only the
used blocks, without reading the whole disk first, which cuts down
migration time significantly. VDDK also brings back warm migration
support (even on recent ESXi versions), and keeps the load on the ESXi
side barely noticeable (unlike before, where it could freeze the API
entirely)."

https://www.linkedin.com/feed/update/urn:li:activity:7359195462538604547/


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pbs-devel] Speedups get detailed publicity

[Adam Kalisz]

Hello lists,

the recent backup restore speedups got some detailed publicity. Jan
Sedlák of Lupa.cz asked me about all the little details and it got into
the article almost without editing. Please excuse the title, it's not
my doing, this was a global effort with people helping with testing
even as far as the USA.

https://www.lupa.cz/clanky/cesi-vyrazne-zrychlili-obnovu-dat-u-rostouci-konkurence-vmwaru-aktualizaci-dali-zdarma-vsem/

Here is a link to the Google Translate translation into English:

https://www-lupa-cz.translate.goog/clanky/cesi-vyrazne-zrychlili-obnovu-dat-u-rostouci-konkurence-vmwaru-aktualizaci-dali-zdarma-vsem/?_x_tr_sl=cs&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp

Thank you Dominik and everybody else for the work on this and
developing Proxmox into a platform, where even smaller changes have a
large impact.

Adam

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pve-devel] Speedups get detailed publicity

[Adam Kalisz via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 6724 bytes --]

From: Adam Kalisz <adam.kalisz@notnullmakers.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>,  Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com>
Cc: Thomas Lamprecht <t.lamprecht@proxmox.com>, Dominik Csapak <d.csapak@proxmox.com>
Subject: Speedups get detailed publicity
Date: Tue, 12 Aug 2025 12:57:07 +0200
Message-ID: <fc78e14f7e3b7a8e265a6e8ee883892a2d50b3ee.camel@notnullmakers.com>

Hello lists,

the recent backup restore speedups got some detailed publicity. Jan
Sedlák of Lupa.cz asked me about all the little details and it got into
the article almost without editing. Please excuse the title, it's not
my doing, this was a global effort with people helping with testing
even as far as the USA.

https://www.lupa.cz/clanky/cesi-vyrazne-zrychlili-obnovu-dat-u-rostouci-konkurence-vmwaru-aktualizaci-dali-zdarma-vsem/

Here is a link to the Google Translate translation into English:

https://www-lupa-cz.translate.goog/clanky/cesi-vyrazne-zrychlili-obnovu-dat-u-rostouci-konkurence-vmwaru-aktualizaci-dali-zdarma-vsem/?_x_tr_sl=cs&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp

Thank you Dominik and everybody else for the work on this and
developing Proxmox into a platform, where even smaller changes have a
large impact.

Adam


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Adam Kalisz]

Hi,

any news about the chunk verification speed-up?

On Mon, 2025-07-28 at 15:22 +0200, Fabian Grünbichler wrote:
> On July 28, 2025 3:14 pm, Adam Kalisz wrote:
> > On Fri, 2025-07-25 at 14:46 +0200, Fabian Grünbichler wrote:
> > > On July 25, 2025 1:23 pm, Adam Kalisz wrote:
> > > > Similarly the sync performance between two Proxmox Backup
> > > > Servers and live-migration got mentioned in various places.
> > > 
> > > do you mean live-restore here? live migration has nothing to do
> > > with PBS..
> > 
> > I threw more things into a single basked as the e-mail was sent to
> > both lists and meant live migration that seems like it could go
> > faster in some cases. (But that might be code that originates in
> > the QEMU project, right?) We could of course have a look at live-
> > restore or even some things like migration from VMware/ ESXi too
> > (again, unrelated to PBS).
> 
> yes, live migration also has some performance optimizations that we
> want to check out, in particular the multi-FD feature:
> 
> https://bugzilla.proxmox.com/show_bug.cgi?id=5766
> 
> live restore uses an entirely different mechanism, and migration from
> ESXi uses yet another mechanism that is bottle-necked by the API on
> the other side at the moment.

The guys over at XCP-ng seem to have switched to VDDK. I just thought I
would point to it for reference:

"Our test VM used to take 25 minutes to migrate, now it's done in under
2 🤯 . Same VM, same setup, just a much better approach.

What changed? We stopped using the native ESXi API to pull disks and
switched to VDDK (VMware’s own library for accessing VM data), which is
way more efficient.

Speed is one thing, but it’s not the only win. We now transfer only the
used blocks, without reading the whole disk first, which cuts down
migration time significantly. VDDK also brings back warm migration
support (even on recent ESXi versions), and keeps the load on the ESXi
side barely noticeable (unlike before, where it could freeze the API
entirely)."

https://www.linkedin.com/feed/update/urn:li:activity:7359195462538604547/

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Thomas Lamprecht]

Am 25.07.25 um 13:24 schrieb Adam Kalisz:
> I missed whether the chunk verification speedup when loading chunks got
> applied or whether it was somehow included in the S3-like storage
> option change set.

btw. had a quick talk with Dominik about this topic, while having the
possibility for this would be nice, server side parallelism is a different
thing than client-side, as being able to starve the rest of the system
of all available IO is much easier possible in the server-side case, as
there is currently no knob to enforce some kind of pacing or maximum
bandwidth like the bandwidth limits that client can be enforced too.

So the mentioned series should IMO only be applied if it can be opted
out, or, better in the long term, a pacing/IO bandwidth limit can be
enforced. That is naturally a big amount of work, mostly in evaluating
different options and deciding for what can make sense, that's why I
mentioned that being able to opt-out could be enough, then admins
could at least keep the status quo.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pve-devel] [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Thomas Lamprecht]

Am 25.07.25 um 13:24 schrieb Adam Kalisz:
> I missed whether the chunk verification speedup when loading chunks got
> applied or whether it was somehow included in the S3-like storage
> option change set.

btw. had a quick talk with Dominik about this topic, while having the
possibility for this would be nice, server side parallelism is a different
thing than client-side, as being able to starve the rest of the system
of all available IO is much easier possible in the server-side case, as
there is currently no knob to enforce some kind of pacing or maximum
bandwidth like the bandwidth limits that client can be enforced too.

So the mentioned series should IMO only be applied if it can be opted
out, or, better in the long term, a pacing/IO bandwidth limit can be
enforced. That is naturally a big amount of work, mostly in evaluating
different options and deciding for what can make sense, that's why I
mentioned that being able to opt-out could be enough, then admins
could at least keep the status quo.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel