[pve-devel] qemu 6 : does proxmox backup handle parallel async chunck backup like the new backup code from qemu 6.0 ?

[pve-devel] qemu 6 : does proxmox backup handle parallel async chunck backup like the new backup code from qemu 6.0 ?

[aderumier]

Hi,

I was looking for qemu 6.0 new features,
and it seem that they have implement parallel async chunks backup (and
I think for other block operations, through a new block-copy feature)

https://github.com/qemu/qemu/commit/71eed4cebed487a4f3c9f97aba83c611bbe22f8d

https://github.com/qemu/qemu/commit/de4641b46b020c5b332175f80e8bfe3d352888e8#diff-b33323044f2699244c126c6eae6c4083c3c99a16f4840030ac13238b1f569dc0

https://github.com/qemu/qemu/commit/26be9d62dd5f5268b814da24fd8e8b5c5b999ebe

Is it alrealy implemented or on the roadmap for the proxmox backup code
?
(To be honest, I didn't follow proxmox backup patches since a long
time, so I really don't known how much they are sharing with qemu
backup code)


Regards,

Alexandre

Re: [pve-devel] qemu 6 : does proxmox backup handle parallel async chunck backup like the new backup code from qemu 6.0 ?

[Thomas Lamprecht]

Hi,

On 02.06.21 08:39, aderumier@odiso.com wrote:
> I was looking for qemu 6.0 new features,
> and it seem that they have implement parallel async chunks backup (and
> I think for other block operations, through a new block-copy feature)
> 
> https://github.com/qemu/qemu/commit/71eed4cebed487a4f3c9f97aba83c611bbe22f8d
> 
> https://github.com/qemu/qemu/commit/de4641b46b020c5b332175f80e8bfe3d352888e8#diff-b33323044f2699244c126c6eae6c4083c3c99a16f4840030ac13238b1f569dc0
> 
> https://github.com/qemu/qemu/commit/26be9d62dd5f5268b814da24fd8e8b5c5b999ebe
> 
> Is it alrealy implemented or on the roadmap for the proxmox backup code
> ?
> (To be honest, I didn't follow proxmox backup patches since a long
> time, so I really don't known how much they are sharing with qemu
> backup code)

the parallel stuff works with our implementation, Stefan tried it and with their
default the backup is much faster, so fast that the guest cannot do anything during
that anymore :D So, IIRC, Stefan turned the default number of parallel operations down
for a better balance of backup speed and guest impact.

FYI, we have basic packages for bullseye as base ready here, and possible sync to
pvetest in the next days/weeks - in those repos there's already QEMU 6.0.

Something else we can use with Bullseye/PVE 7 is io_uring instead of AIO, looks like
a much saner (and actually useful) design and there should be some, at least slight,
improvements due to that too.

https://kernel.dk/io_uring.pdf

cheers,
Thomas

Re: [pve-devel] qemu 6 : does proxmox backup handle parallel async chunck backup like the new backup code from qemu 6.0 ?

[aderumier]

Le mercredi 02 juin 2021 à 08:58 +0200, Thomas Lamprecht a écrit :
> Hi,
> 
> On 02.06.21 08:39, aderumier@odiso.com wrote:
> > I was looking for qemu 6.0 new features,
> > and it seem that they have implement parallel async chunks backup
> > (and
> > I think for other block operations, through a new block-copy
> > feature)
> > 
> > https://github.com/qemu/qemu/commit/71eed4cebed487a4f3c9f97aba83c611bbe22f8d
> > 
> > https://github.com/qemu/qemu/commit/de4641b46b020c5b332175f80e8bfe3d352888e8#diff-b33323044f2699244c126c6eae6c4083c3c99a16f4840030ac13238b1f569dc0
> > 
> > https://github.com/qemu/qemu/commit/26be9d62dd5f5268b814da24fd8e8b5c5b999ebe
> > 
> > Is it alrealy implemented or on the roadmap for the proxmox backup
> > code
> > ?
> > (To be honest, I didn't follow proxmox backup patches since a long
> > time, so I really don't known how much they are sharing with qemu
> > backup code)
> 
> the parallel stuff works with our implementation, Stefan tried it and
> with their
> default the backup is much faster, so fast that the guest cannot do
> anything during
> that anymore :D So, IIRC, Stefan turned the default number of
> parallel operations down
> for a better balance of backup speed and guest impact.
> 
ok thanks ! (I'll still don't use pbs, I'll try to test with my rbd
cluster soon to compare speed vs rbd export/import)

> FYI, we have basic packages for bullseye as base ready here, and
> possible sync to
> pvetest in the next days/weeks - in those repos there's already QEMU
> 6.0.
> 
yes, I see that.I'll try to do test soon

> Something else we can use with Bullseye/PVE 7 is io_uring instead of
> AIO, looks like
> a much saner (and actually useful) design and there should be some,
> at least slight,
> improvements due to that too.
> 
> https://kernel.dk/io_uring.pdf
> 
yes, If I remember Wolfgang had seen some bug with kernel 5.4, but I
think it's should works fine with 5.11.
I'll try it too soon :)


> cheers,
> Thomas
>