From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 815511FF15C for ; Wed, 5 Mar 2025 16:15:43 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5D90119342; Wed, 5 Mar 2025 16:15:39 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Wed, 5 Mar 2025 16:14:45 +0100 Message-Id: <20250305151453.388817-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.031 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [datastore.rs, proxmox.com] Subject: [pbs-devel] [PATCH v4 proxmox-backup 0/8] fix #5982: check atime update is honored X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" These patches add a check to phase 1 of garbage collection and datastore creation in order to detect when the filesystem backing the chunk store does not honor atime updates. This avoids possible data loss for situations where garbage collection could otherwise delete chunks still referenced by a backup snaphost's index file. The check is performed on a fixed size 4 MiB unencrypted and compressed chunk of all-zeros, inserted if not present yet. The Linux kernel timestamp granularity is taken into account by sleeping for 1 second to avoid discarded atime update attempts by utimensat calls. The test is enabled by default, but an opt-out option can be set via the datastore tuning parameters for backwards compatibility. Further, add a datastore tuning parameter to reduce the wait period for chunk removal in phase 2 of garbage collection. Make this conditional on the atime update check being enabled and successful, to avoid possible data loss. Most notable changes sice version 3 (thanks Fabian for feedback): - Drop check for relatime like behaviour, as this is not supported and does not show up in any of the tests performed on btrfs, cephfs, ext4, NFS3, NFS4, ntfs, SMB3_11, xfs or ZFS. - Additionally check chunk inode to detect possible but very unlikely file changes, perform check once again in that case. - Move atime cutoff selection and min_atime calculation to the same location, as they are logically related. Most notable changes sice version 2 (thanks Fabian and Thomas for comments and suggestions): - Take into account Linux timestamp granularity, do not set timestamp to the past, as that introduces other error paths such as lack of permissions or fs limitations. - Check relatime behavior, if atime behaviour is not honored. Fallback to original cutoff in that case. - Adapt tuning parameter names. Most notable changes sice version 1 (thanks Fabian and Thomas for comments and suggestions): - Optimize check by using the all zero chunk - Enable the check by default and fail GC job if not honored, but allow to opt-out - Add GC wait period tuning option Link to the issue in the bugtracker: https://bugzilla.proxmox.com/show_bug.cgi?id=5982 proxmox: Christian Ebner (2): pbs api types: add garbage collection atime safety check flag pbs api types: add option to set GC chunk cleanup atime cutoff pbs-api-types/src/datastore.rs | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) proxmox-backup: Christian Ebner (6): fix #5982: garbage collection: check atime updates are honored ui: expose GC atime safety check flag in datastore tuning options docs: mention GC atime update check for tuning options datastore: conditionally use custom GC atime cutoff if set ui: expose GC atime cutoff in datastore tuning option docs: mention gc-atime-cutoff as datastore tuning option docs/storage.rst | 19 ++++++- pbs-datastore/src/chunk_store.rs | 97 +++++++++++++++++++++++++++----- pbs-datastore/src/datastore.rs | 37 +++++++++++- src/api2/config/datastore.rs | 1 + www/Utils.js | 9 +++ www/datastore/OptionView.js | 17 ++++++ 6 files changed, 162 insertions(+), 18 deletions(-) -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel