[pbs-devel] Poor client backup performance [Bug#3811]

[pbs-devel] Poor client backup performance [Bug#3811]

[Mark Schouten]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

Hi!

I switched a few servers to Proxmox Backup Server this week (a slow week :)), but am somewhat disappointed in the performance of the client. I created bug 3811 for it, and would like to help and assist in debugging this issue.

The mentioned server in the bug should backup about three times per day. Currently, the machine is working on it almost fulltime, because the backup takes more than 7 hours.

I think the client reads all files I every run, and I would expect stat’ing the files should be enough to determine whether a file should be re-read, but I’ve not spent a single second reading the code, so I do not have any more than suspicions.

As said, I’d like to help improving this situation, please let me know what you need from us.

https://bugzilla.proxmox.com/show_bug.cgi?id=3811 <https://bugzilla.proxmox.com/show_bug.cgi?id=3811>

Kind regards,

— 
Mark Schouten, CTO
Tuxis B.V.
mark@tuxis.nl <mailto:mark@tuxis.nl>

[-- Attachment #2: Type: text/html, Size: 1609 bytes --]

Re: [pbs-devel] Poor client backup performance [Bug#3811]

[Lubomir Apostolov]

[-- Attachment #1: Type: text/plain, Size: 1771 bytes --]

Hello,

One year ago I talked about this here without success.

Proxmox Backup has no differential read algorithm and no plans to do it. It handles only changes during current backup, but has to read every time all backup data. If you have 20Tb to backup every 3 hours, your storage must be able to read at at least 2Gb/s 24/7 and even more if you dare use your VMs at the same time. Complete design failure except for some dev/lab setups.

It could use snapshots from the underlying storage like ZFS or Ceph, but no, devs seem happy without it. A simple script for differential snapshot copy does a better job.

Best regards,
Lubomir


-------- Message d'origine --------
De : Mark Schouten <mark@tuxis.nl>
Date : 30/12/2021 10:07 (GMT+01:00)
À : pbs-devel <pbs-devel@lists.proxmox.com>
Objet : [pbs-devel] Poor client backup performance [Bug#3811]

Hi!

I switched a few servers to Proxmox Backup Server this week (a slow week :)), but am somewhat disappointed in the performance of the client. I created bug 3811 for it, and would like to help and assist in debugging this issue.

The mentioned server in the bug should backup about three times per day. Currently, the machine is working on it almost fulltime, because the backup takes more than 7 hours.

I think the client reads all files I every run, and I would expect stat’ing the files should be enough to determine whether a file should be re-read, but I’ve not spent a single second reading the code, so I do not have any more than suspicions.

As said, I’d like to help improving this situation, please let me know what you need from us.

https://bugzilla.proxmox.com/show_bug.cgi?id=3811

Kind regards,

—
Mark Schouten, CTO
Tuxis B.V.
mark@tuxis.nl<mailto:mark@tuxis.nl>

[-- Attachment #2: Type: text/html, Size: 2887 bytes --]

Re: [pbs-devel] Poor client backup performance [Bug#3811]

[Mark Schouten]

[-- Attachment #1: Type: text/plain, Size: 2716 bytes --]

Hi Lubomir,

I think it’s a bit more nuanced, but indeed, the current situation is quite unusable. I’ll have to move my file-based backups back to my previous situation.

My bug has rightfully been closed as a duplicate, but maybe if you have any ideas you could share them here: https://bugzilla.proxmox.com/show_bug.cgi?id=3174 <https://bugzilla.proxmox.com/show_bug.cgi?id=3174>

I’d like to see this fixed too, and maybe with all brains combined, we actually can :)
 
— 
Mark Schouten, CTO
Tuxis B.V.
mark@tuxis.nl



> On 30 Dec 2021, at 11:49, Lubomir Apostolov <Lubomir.Apostolov@directique.com> wrote:
> 
> Hello, 
> 
> One year ago I talked about this here without success. 
> 
> Proxmox Backup has no differential read algorithm and no plans to do it. It handles only changes during current backup, but has to read every time all backup data. If you have 20Tb to backup every 3 hours, your storage must be able to read at at least 2Gb/s 24/7 and even more if you dare use your VMs at the same time. Complete design failure except for some dev/lab setups.
> 
> It could use snapshots from the underlying storage like ZFS or Ceph, but no, devs seem happy without it. A simple script for differential snapshot copy does a better job.
> 
> Best regards,
> Lubomir
> 
> 
> -------- Message d'origine --------
> De : Mark Schouten <mark@tuxis.nl>
> Date : 30/12/2021 10:07 (GMT+01:00)
> À : pbs-devel <pbs-devel@lists.proxmox.com>
> Objet : [pbs-devel] Poor client backup performance [Bug#3811]
> 
> Hi!
> 
> I switched a few servers to Proxmox Backup Server this week (a slow week :)), but am somewhat disappointed in the performance of the client. I created bug 3811 for it, and would like to help and assist in debugging this issue.
> 
> The mentioned server in the bug should backup about three times per day. Currently, the machine is working on it almost fulltime, because the backup takes more than 7 hours.
> 
> I think the client reads all files I every run, and I would expect stat’ing the files should be enough to determine whether a file should be re-read, but I’ve not spent a single second reading the code, so I do not have any more than suspicions.
> 
> As said, I’d like to help improving this situation, please let me know what you need from us.
> 
> https://bugzilla.proxmox.com/show_bug.cgi?id=3811 <https://bugzilla.proxmox.com/show_bug.cgi?id=3811>
> 
> Kind regards,
> 
> — 
> Mark Schouten, CTO
> Tuxis B.V.
> mark@tuxis.nl <mailto:mark@tuxis.nl>_______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


[-- Attachment #2: Type: text/html, Size: 4938 bytes --]

Re: [pbs-devel] Poor client backup performance [Bug#3811]

[Maximilian Hill]

Hi,

we ourselves don't have this problem, because we mostly use PBS to back
up QEMU VMs, but whenever we use containers, this becomes a problem for
us as well.

For containers this could be solved using rbd/zfs diffs of snapshots and
QEMU backups would benefit as well, but this problem would stay a problem
for host backups.

It's way too easy to push Proxmox Bakup (Server) to it's limits right now.


On Thu, Dec 30, 2021 at 05:30:24PM +0100, Mark Schouten wrote:
> Hi Lubomir,
> 
> I think it’s a bit more nuanced, but indeed, the current situation is quite unusable. I’ll have to move my file-based backups back to my previous situation.
> 
> My bug has rightfully been closed as a duplicate, but maybe if you have any ideas you could share them here: https://bugzilla.proxmox.com/show_bug.cgi?id=3174 <https://bugzilla.proxmox.com/show_bug.cgi?id=3174>
> 
> I’d like to see this fixed too, and maybe with all brains combined, we actually can :)
>  
> — 
> Mark Schouten, CTO
> Tuxis B.V.
> mark@tuxis.nl
> 
> 
> 
> 
inett GmbH
>> Ihr IT Systemhaus in  Saarbrücken

Maximilian Hill


t: 0681-410993-0
e: mhill@inett.de
www.inett.de

Technische Fragen:
Support E-Mail: support@inett.de
Support Hotline: 0681-410993-42


inett GmbH | Geschäftsführung: Marco Gabriel | Amtsgericht Saarbrücken HRB 16588
> On 30 Dec 2021, at 11:49, Lubomir Apostolov <Lubomir.Apostolov@directique.com> wrote:
> > 
> > Hello, 
> > 
> > One year ago I talked about this here without success. 
> > 
> > Proxmox Backup has no differential read algorithm and no plans to do it. It handles only changes during current backup, but has to read every time all backup data. If you have 20Tb to backup every 3 hours, your storage must be able to read at at least 2Gb/s 24/7 and even more if you dare use your VMs at the same time. Complete design failure except for some dev/lab setups.
> > 
> > It could use snapshots from the underlying storage like ZFS or Ceph, but no, devs seem happy without it. A simple script for differential snapshot copy does a better job.
> > 
> > Best regards,
> > Lubomir
> > 
> > 
> > -------- Message d'origine --------
> > De : Mark Schouten <mark@tuxis.nl>
> > Date : 30/12/2021 10:07 (GMT+01:00)
> > À : pbs-devel <pbs-devel@lists.proxmox.com>
> > Objet : [pbs-devel] Poor client backup performance [Bug#3811]
> > 
> > Hi!
> > 
> > I switched a few servers to Proxmox Backup Server this week (a slow week :)), but am somewhat disappointed in the performance of the client. I created bug 3811 for it, and would like to help and assist in debugging this issue.
> > 
> > The mentioned server in the bug should backup about three times per day. Currently, the machine is working on it almost fulltime, because the backup takes more than 7 hours.
> > 
> > I think the client reads all files I every run, and I would expect stat’ing the files should be enough to determine whether a file should be re-read, but I’ve not spent a single second reading the code, so I do not have any more than suspicions.
> > 
> > As said, I’d like to help improving this situation, please let me know what you need from us.
> > 
> > https://bugzilla.proxmox.com/show_bug.cgi?id=3811 <https://bugzilla.proxmox.com/show_bug.cgi?id=3811>
> > 
> > Kind regards,
> > 
> > — 
> > Mark Schouten, CTO
> > Tuxis B.V.
> > mark@tuxis.nl <mailto:mark@tuxis.nl>_______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 

> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel