From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 4BCA7BC0B9 for ; Thu, 28 Mar 2024 13:38:14 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 76382A104 for ; Thu, 28 Mar 2024 13:37:51 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 28 Mar 2024 13:37:50 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id D9173429B0 for ; Thu, 28 Mar 2024 13:37:49 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Thu, 28 Mar 2024 13:37:05 +0100 Message-Id: <20240328123707.336951-57-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240328123707.336951-1-c.ebner@proxmox.com> References: <20240328123707.336951-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.030 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH v3 proxmox-backup 56/58] docs: add section describing change detection mode X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2024 12:38:14 -0000 Describe the motivation and basic principle of the clients change detection mode and show an example invocation. Signed-off-by: Christian Ebner --- changes since version 2: - not present in previous version docs/backup-client.rst | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/docs/backup-client.rst b/docs/backup-client.rst index 00a1abbb3..e54a52bf4 100644 --- a/docs/backup-client.rst +++ b/docs/backup-client.rst @@ -280,6 +280,39 @@ Multiple paths can be excluded like this: # proxmox-backup-client backup.pxar:./linux --exclude=/usr --exclude=/rust +.. _client_change_detection_mode: + +Change detection mode +~~~~~~~~~~~~~~~~~~~~~ + +Backing up filesystems with large contents can take a long time, as the default +behaviour for the Proxmox Backup Client is to read all data and re-encode it +into chunks. For some usecases, where files do not change frequently this is not +feasible and undesired. + +In order to instruct the client to not re-encode files with unchanged metadata, +the `change-detection-mode` can be set from the default `data` to `metadata`. +By this, regular file payloads for files with unchanged metadata are looked up +and re-used from the previous backup runs snapshot when possible. For this to +be feasible, the pxar archives for backup runs using this mode are split into +two separate files, the `mpxar` containing the archives metadata and the `ppxar` +containing a concatenation of the file payloads. + +During backup, the current file metadata is compared to the one looked up in the +previous `mpxar` archive, and if unchanged, the payload of the file is included +in the current backup by referencing the indices of the previous snaphshot. The +increase in backup speed comes at the cost of a possible increase of used space, +as chunks might only be partially reused, containing unneeded padding. This is +however minimized by selectively re-encoding files where the padding overhead +does not justify a re-use. + +The following shows an example for the client invocation with the `metadata` +mode: + +.. code-block:: console + + # proxmox-backup-client backup.pxar:./linux --change-detection-mode=metadata + .. _client_encryption: Encryption -- 2.39.2