From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id D7D3B1FF168 for ; Tue, 26 Nov 2024 09:55:50 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 9C93B2A372; Tue, 26 Nov 2024 09:55:50 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Tue, 26 Nov 2024 09:55:02 +0100 Message-Id: <20241126085502.77438-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.029 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox-backup 1/1] docs: explain some further caveats of the change detection modes X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Explain that the change detection mode data makes sure that no files are considered reusable, even if their metadata might match and that the use of ctime and inode number is not possible for detection of unchanged files if the filesystem was synced to a temporary location, therefore the mtime and size are used for detection. Also note the reduced deduplication when storing snaphshots with mixed archive formats on the same datastore. Further, mention the backwards compatibility to older version of the Proxmox Backup Server. Suggested-by: Thomas Lamprecht Signed-off-by: Christian Ebner --- docs/technical-overview.rst | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/docs/technical-overview.rst b/docs/technical-overview.rst index 21793c5c5..ac42da3a2 100644 --- a/docs/technical-overview.rst +++ b/docs/technical-overview.rst @@ -141,6 +141,25 @@ The change detection mode controls how to detect and act for files which did not change in-between subsequent backup runs as well as the archive file format used to encode the directory entries. +There are 3 modes available, the current default ``legacy`` mode, as well as the +``data`` and ``metadata`` mode. While the ``legacy`` mode encodes all contents +in a single ``pxar`` archive, the latter two modes split data and metadata into +``ppxar`` and ``mpxar`` archives. This is done to allow for fast comparison of +metadata with the previous snapshot, used by the ``metadata`` mode to detect +reusable files. The ``data`` mode refrains from reusing unchanged files by +rechunking the file uncoditionally. This mode therefore assures that no file +changes are missed even if the metadata are unchanged. + +.. NOTE:: ``pxar`` and ``mpxar``/``ppxar`` file formats are different and cannot + be deduplicated as efficiently if a datastore stores archive snapshots of + both types. + +As the change detection modes are client side changes, they are backwards +compatible with older versions of Proxmox Backup Server. Exploring the backup +contents for the new archive format via the web interface requires however a +Proxmox Backup Server with version 3.2.5 or higher. Upgrading to the latest +version is recommended for full feature compatibility. + .. _change-detection-mode-legacy: Legacy Mode @@ -182,6 +201,11 @@ chunks. This is used for example for entry lookups to list the archive contents or to navigate the mounted filesystem via the FUSE implementation. No dedicated catalog is therefore created for archives encoded using this mode. +By not comparing metadata to the previous backup snapshot, no files will be +considered reusable by this mode, in contrast to the ``metadata`` mode. +Latter can reuse files which have changed, but file size and mtime did not +change because restored after changing the files contents. + .. _change-detection-mode-metadata: Metadata Mode @@ -191,9 +215,15 @@ The ``metadata`` mode detects files whose file metadata did not change in-between subsequent backup runs. The metadata comparison includes file size, file type, ownership and permission information, as well as acls and attributes and most importantly the file's mtime, for details see the -:ref:`pxar metadata archive format `. This mode will avoid -reading and rechunking the file contents whenever possible by reusing the file -content chunks of unchanged files from the previous backup snapshot. +:ref:`pxar metadata archive format `. Files ctime and inode +number are not stored and used for comparison, since some tools (e.g. +``vzdump``) might sync the contents of the filesystem to a temporary location +before actually performing the backup via the Proxmox backup client. For these +cases, ctime and inode number will always change. + +This mode will avoid reading and rechunking the file contents whenever possible +by reusing the file content chunks of unchanged files from the previous backup +snapshot. To compare the metadata, the previous snapshots ``mpxar`` metadata archive is downloaded at the start of the backup run and used as a reference. Further, the -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel