From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id E6F9B1FF0B7 for ; Fri, 30 Oct 2026 12:31:33 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5B9102005A; Fri, 30 Oct 2026 12:31:33 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup] GC: fail on abnormal system time drifted between phase 1 and 2 Date: Fri, 30 Oct 2026 12:31:13 +0100 Message-ID: <20261030113113.5877-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793359889543 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.242 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_NUMSUBJECT 0.5 Subject ends in numbers excluding current years RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: QMIYFRAHCBMWNGUD4X7RSVI2DKOXNZAI X-Message-ID-Hash: QMIYFRAHCBMWNGUD4X7RSVI2DKOXNZAI X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The garbage collection cutoff time for chunks being considered as no longer in-use and therefore removed is calculated at the start of phase 1, taking into account the system time. Chunks access timestamp is then updated using utimensat() calls with the current system time. If the system time changed into the past between phase 1 and phase 2, e.g. due to unexpected manual intervention or NTP time update, in-use chunks might be incorrectly assumed as no longer referenced. Under normal operation this should not happen, and the cutoff time already acts as limited safeguard to protect from this. The default cutoff being 24h 5m, the minimum cutoff currently possible to select as datastore tuning parameter is 1 minute, allowing for out of bound faster cleanup of chunks. To fully protect from this it would be require to keep track of the minimum atime set on any still referenced chunk, which would however require expensive continuous system time monitoring, or moving away from libc::UTIME_NOW for utimensat() calls, the latter however having potential for regressions due to unexpected implementation of filesystems. Therefore, only protect from drifts into the past between start of phase 1 to start of phase 2 with 1 second time resolution, eventhough this cannot protect against multiple system time drifts happening within the runtime span of phase 1. If detected, the garbage collection task fails now with, for example: ``` TASK ERROR: Unexpected system time drift detected, current time 2026-10-30T10:22:29Z before start of phase 1 2026-10-30T23:00:00Z ``` A hard failure of garbage collection is preferred over adjustments to the cutoff time and guards against chunk removal and results in a notable task failure and notification (if set up), for sysadmins to investigate and act upon. Signed-off-by: Christian Ebner --- pbs-datastore/src/datastore.rs | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs index 665333f52..f7cf82d62 100644 --- a/pbs-datastore/src/datastore.rs +++ b/pbs-datastore/src/datastore.rs @@ -2469,6 +2469,15 @@ impl DataStore { ) .context("marking used chunks failed")?; + let phase2_start_time = proxmox_time::epoch_i64(); + if phase2_start_time < phase1_start_time { + bail!( + "Unexpected system time drift detected, current time {} before start of phase 1 {}", + proxmox_time::epoch_to_rfc3339_utc(phase2_start_time)?, + proxmox_time::epoch_to_rfc3339_utc(phase1_start_time)?, + ); + } + info!("Start GC phase2 (sweep unused chunks)"); if let Some(ref s3_client) = s3_client { -- 2.47.3