From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 6F2AC1FF38E for ; Tue, 14 May 2024 12:35:59 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 798EF4B76; Tue, 14 May 2024 12:35:39 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Tue, 14 May 2024 12:34:10 +0200 Message-Id: <20240514103421.289431-55-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240514103421.289431-1-c.ebner@proxmox.com> References: <20240514103421.289431-1-c.ebner@proxmox.com> MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.025 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH v6 proxmox-backup 54/65] docs: add section describing change detection mode X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Describe the motivation and basic principle of the clients change detection mode and show an example invocation. Signed-off-by: Christian Ebner --- docs/backup-client.rst | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/backup-client.rst b/docs/backup-client.rst index 00a1abbb3..e48b5dd60 100644 --- a/docs/backup-client.rst +++ b/docs/backup-client.rst @@ -280,6 +280,47 @@ Multiple paths can be excluded like this: # proxmox-backup-client backup.pxar:./linux --exclude=/usr --exclude=/rust +.. _client_change_detection_mode: + +Change detection mode +~~~~~~~~~~~~~~~~~~~~~ + +File-based backups containing a lot of data can take a long time, as the default +behavior for the Proxmox backup client is to read all data and re-encode it. +The encoded stream is split into variable sized chunks for efficient +deduplication and based on the chunk digest a decision can be made whether a +given chunk needs to be uploaded or can be indexed without upload as it is +already available on the server (and therefore deduplicated). For some +use-cases, where files do not change frequently the full re-reading is not +feasible and undesired. + +The backup clients `change-detection-mode` can be switched from default to +`metadata` based detection to reduce limitations as described above, instructing +the client to avoid re-reading files with unchanged metadata whenever possible. +When using this mode, instead of the regular pxar archive, the backup snapshot +is stored into two separate files: the `mpxar` containing the archives metadata +and the `ppxar` containing a concatenation of the file contents. This splitting +allows for metadata lookups without the overhead of the file contents. +Using the `change-detection-mode` set to `data` allows to create the same split +archive as when using the `metadata` mode, but without using a previous +reference and therefore reencoding all file payloads. + +When creating the backup archives, the current file metadata is compared to the +one looked up in the previous `mpxar` archive, and if unchanged the entry cached +for possible re-use of content chunks without re-reading, by indexing the +already present chunks containing the contents from the previous backup +snapshot. Since the file might only partially re-use chunks (thereby introducing +wasted space in the form of padding), the decision whether to re-use or +re-encode the currently cached entries is delegated to when enough information +is available, comparing the possible padding a threshold value. + +The following shows an example for the client invocation with the `metadata` +mode: + +.. code-block:: console + + # proxmox-backup-client backup.pxar:./linux --change-detection-mode=metadata + .. _client_encryption: Encryption -- 2.39.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel