From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D70F492266 for ; Fri, 5 Apr 2024 13:22:54 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B3C751242F for ; Fri, 5 Apr 2024 13:22:54 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 5 Apr 2024 13:22:54 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id AFD1746464 for ; Fri, 5 Apr 2024 13:22:53 +0200 (CEST) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable In-Reply-To: <20240328123707.336951-57-c.ebner@proxmox.com> References: <20240328123707.336951-1-c.ebner@proxmox.com> <20240328123707.336951-57-c.ebner@proxmox.com> From: Fabian =?utf-8?q?Gr=C3=BCnbichler?= To: Christian Ebner , pbs-devel@lists.proxmox.com Date: Fri, 05 Apr 2024 13:22:47 +0200 Message-ID: <171231616753.1926770.10738045721029938736@yuna.proxmox.com> User-Agent: alot/0.10 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.058 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pbs-devel] [PATCH v3 proxmox-backup 56/58] docs: add section describing change detection mode X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2024 11:22:54 -0000 Quoting Christian Ebner (2024-03-28 13:37:05) > Describe the motivation and basic principle of the clients change > detection mode and show an example invocation. >=20 > Signed-off-by: Christian Ebner > --- > changes since version 2: > - not present in previous version >=20 > docs/backup-client.rst | 33 +++++++++++++++++++++++++++++++++ > 1 file changed, 33 insertions(+) >=20 > diff --git a/docs/backup-client.rst b/docs/backup-client.rst > index 00a1abbb3..e54a52bf4 100644 > --- a/docs/backup-client.rst > +++ b/docs/backup-client.rst > @@ -280,6 +280,39 @@ Multiple paths can be excluded like this: > =20 > # proxmox-backup-client backup.pxar:./linux --exclude=3D/usr --exclu= de=3D/rust > =20 > +.. _client_change_detection_mode: > + > +Change detection mode > +~~~~~~~~~~~~~~~~~~~~~ > + > +Backing up filesystems with large contents can take a long time, as the = default > +behaviour for the Proxmox Backup Client is to read all data and re-encod= e it > +into chunks. For some usecases, where files do not change frequently thi= s is not > +feasible and undesired. Maybe File-based backups containing a lot of data can and maybe also add something along the lines of "before being able to decide whether those chunks need uploading or are already available on the server." > + > +In order to instruct the client to not re-encode files with unchanged me= tadata, re-read (as that is what gives us the performance bonus ;)) > +the `change-detection-mode` can be set from the default `data` to `metad= ata`. > +By this, regular file payloads for files with unchanged metadata are loo= ked up s/By this/In this mode do we want to explain "payload" here? or just use "contents" or "data"? > +and re-used from the previous backup runs snapshot when possible. For th= is to s/runs// > +be feasible, the pxar archives for backup runs using this mode are split= into s/ runs/s/ > +two separate files, the `mpxar` containing the archives metadata and the= `ppxar` > +containing a concatenation of the file payloads. > + > +During backup, the current file metadata is compared to the one looked u= p in the During the backup=20 or maybe When creating the backup archives,=20 > +previous `mpxar` archive, and if unchanged, the payload of the file is i= ncluded > +in the current backup by referencing the indices of the previous snaphsh= ot. The by reusing the chunks containing that payload in the previous snapshot. > +increase in backup speed comes at the cost of a possible increase of use= d space, > +as chunks might only be partially reused, containing unneeded padding. T= his is > +however minimized by selectively re-encoding files where the padding ove= rhead > +does not justify a re-use. nit: reused is not hyphenated consistenly > + > +The following shows an example for the client invocation with the `metad= ata` > +mode: > + > +.. code-block:: console > + > + # proxmox-backup-client backup.pxar:./linux --change-detection-mode= =3Dmetadata > + > .. _client_encryption: > =20 > Encryption > --=20 > 2.39.2 >=20 >=20 >=20 > _______________________________________________ > pbs-devel mailing list > pbs-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel >=20 >