From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id BFE5F87317 for ; Thu, 30 Dec 2021 17:30:39 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B26081C716 for ; Thu, 30 Dec 2021 17:30:39 +0100 (CET) Received: from kerio.tuxis.nl (alrami.saas.tuxis.net [31.3.111.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id CAB871C700 for ; Thu, 30 Dec 2021 17:30:37 +0100 (CET) X-Footer: dHV4aXMubmw= Received: from smtpclient.apple ([2a03:7900:64::1001]) (authenticated user mark@tuxis.nl) by kerio.tuxis.nl (Kerio Connect 9.3.1 patch 2) with ESMTPSA (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256 bits)) for pbs-devel@lists.proxmox.com; Thu, 30 Dec 2021 17:30:36 +0100 From: Mark Schouten Content-Type: multipart/alternative; boundary="Apple-Mail=_4F2DD42D-5CF6-4627-952B-2FCB8F95841A" Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Date: Thu, 30 Dec 2021 17:30:24 +0100 References: <17A444C0-1F31-42AA-81FE-A95CE4B3291E@tuxis.nl> <718D0AF11703FA4C85B0535448A056100381A40015@SCOM4.directique.net> To: pbs-devel In-Reply-To: <718D0AF11703FA4C85B0535448A056100381A40015@SCOM4.directique.net> Message-Id: X-Mailer: Apple Mail (2.3693.40.0.1.81) X-SPAM-LEVEL: Spam detection results: 0 AWL 0.210 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% HTML_MESSAGE 0.001 HTML included in message KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] Poor client backup performance [Bug#3811] X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Dec 2021 16:30:39 -0000 --Apple-Mail=_4F2DD42D-5CF6-4627-952B-2FCB8F95841A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Lubomir, I think it=E2=80=99s a bit more nuanced, but indeed, the current = situation is quite unusable. I=E2=80=99ll have to move my file-based = backups back to my previous situation. My bug has rightfully been closed as a duplicate, but maybe if you have = any ideas you could share them here: = https://bugzilla.proxmox.com/show_bug.cgi?id=3D3174 = I=E2=80=99d like to see this fixed too, and maybe with all brains = combined, we actually can :) =20 =E2=80=94=20 Mark Schouten, CTO Tuxis B.V. mark@tuxis.nl > On 30 Dec 2021, at 11:49, Lubomir Apostolov = wrote: >=20 > Hello,=20 >=20 > One year ago I talked about this here without success.=20 >=20 > Proxmox Backup has no differential read algorithm and no plans to do = it. It handles only changes during current backup, but has to read every = time all backup data. If you have 20Tb to backup every 3 hours, your = storage must be able to read at at least 2Gb/s 24/7 and even more if you = dare use your VMs at the same time. Complete design failure except for = some dev/lab setups. >=20 > It could use snapshots from the underlying storage like ZFS or Ceph, = but no, devs seem happy without it. A simple script for differential = snapshot copy does a better job. >=20 > Best regards, > Lubomir >=20 >=20 > -------- Message d'origine -------- > De : Mark Schouten > Date : 30/12/2021 10:07 (GMT+01:00) > =C3=80 : pbs-devel > Objet : [pbs-devel] Poor client backup performance [Bug#3811] >=20 > Hi! >=20 > I switched a few servers to Proxmox Backup Server this week (a slow = week :)), but am somewhat disappointed in the performance of the client. = I created bug 3811 for it, and would like to help and assist in = debugging this issue. >=20 > The mentioned server in the bug should backup about three times per = day. Currently, the machine is working on it almost fulltime, because = the backup takes more than 7 hours. >=20 > I think the client reads all files I every run, and I would expect = stat=E2=80=99ing the files should be enough to determine whether a file = should be re-read, but I=E2=80=99ve not spent a single second reading = the code, so I do not have any more than suspicions. >=20 > As said, I=E2=80=99d like to help improving this situation, please let = me know what you need from us. >=20 > https://bugzilla.proxmox.com/show_bug.cgi?id=3D3811 = >=20 > Kind regards, >=20 > =E2=80=94=20 > Mark Schouten, CTO > Tuxis B.V. > mark@tuxis.nl = _______________________________________________ > pbs-devel mailing list > pbs-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel --Apple-Mail=_4F2DD42D-5CF6-4627-952B-2FCB8F95841A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Lubomir,

I think = it=E2=80=99s a bit more nuanced, but indeed, the current situation is = quite unusable. I=E2=80=99ll have to move my file-based backups back to = my previous situation.

My bug has rightfully been closed as a duplicate, but maybe = if you have any ideas you could share them here: https://bugzilla.proxmox.com/show_bug.cgi?id=3D3174

I=E2=80=99d like to = see this fixed too, and maybe with all brains combined, we actually can = :)
 
=E2=80=94 
Mark = Schouten, CTO
Tuxis B.V.



On 30 Dec 2021, at 11:49, Lubomir Apostolov <Lubomir.Apostolov@directique.com> wrote:

Hello, 

One year ago I talked about this here = without success. 

Proxmox Backup has no differential read = algorithm and no plans to do it. It handles only changes during current = backup, but has to read every time all backup data. If you have 20Tb to = backup every 3 hours, your storage must be able to read at at least 2Gb/s 24/7 and even more if you dare use your VMs at the same = time. Complete design failure except for some dev/lab setups.

It could use snapshots from the underlying = storage like ZFS or Ceph, but no, devs seem happy without it. A simple = script for differential snapshot copy does a better job.

Best regards,
Lubomir


-------- Message d'origine --------
De : Mark Schouten <mark@tuxis.nl>
Date : 30/12/2021 10:07 (GMT+01:00)
=C3=80 : pbs-devel <pbs-devel@lists.proxmox.com>
Objet : [pbs-devel] Poor client backup performance = [Bug#3811]

Hi!

I switched a few servers to Proxmox Backup Server this = week (a slow week :)), but am somewhat disappointed in the performance = of the client. I created bug 3811 for it, and would like to help and = assist in debugging this issue.

The mentioned server in the bug should backup about = three times per day. Currently, the machine is working on it almost = fulltime, because the backup takes more than 7 hours.

I think the client reads all files I every run, and I = would expect stat=E2=80=99ing the files should be enough to determine = whether a file should be re-read, but I=E2=80=99ve not spent a single = second reading the code, so I do not have any more than = suspicions.

As said, I=E2=80=99d like to help improving this = situation, please let me know what you need from us.

https://bugzilla.proxmox.com/show_bug.cgi?id=3D3811

Kind regards,

=E2=80=94 
Mark Schouten, CTO
Tuxis B.V.
mark@tuxis.nl
_______________________________________________
pbs-devel = mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

= --Apple-Mail=_4F2DD42D-5CF6-4627-952B-2FCB8F95841A--