From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [PATCH proxmox-backup] GC: fail on abnormal system time drifted between phase 1 and 2
Date: Fri, 30 Oct 2026 12:31:13 +0100 [thread overview]
Message-ID: <20261030113113.5877-1-c.ebner@proxmox.com> (raw)
The garbage collection cutoff time for chunks being considered as no
longer in-use and therefore removed is calculated at the start of
phase 1, taking into account the system time. Chunks access timestamp
is then updated using utimensat() calls with the current system time.
If the system time changed into the past between phase 1 and phase 2,
e.g. due to unexpected manual intervention or NTP time update, in-use
chunks might be incorrectly assumed as no longer referenced.
Under normal operation this should not happen, and the cutoff time
already acts as limited safeguard to protect from this. The default
cutoff being 24h 5m, the minimum cutoff currently possible to select
as datastore tuning parameter is 1 minute, allowing for out of bound
faster cleanup of chunks.
To fully protect from this it would be require to keep track of the
minimum atime set on any still referenced chunk, which would however
require expensive continuous system time monitoring, or moving away
from libc::UTIME_NOW for utimensat() calls, the latter however having
potential for regressions due to unexpected implementation of
filesystems.
Therefore, only protect from drifts into the past between start of
phase 1 to start of phase 2 with 1 second time resolution, eventhough
this cannot protect against multiple system time drifts happening
within the runtime span of phase 1.
If detected, the garbage collection task fails now with, for example:
```
TASK ERROR: Unexpected system time drift detected, current time 2026-10-30T10:22:29Z before start of phase 1 2026-10-30T23:00:00Z
```
A hard failure of garbage collection is preferred over adjustments to
the cutoff time and guards against chunk removal and results in a
notable task failure and notification (if set up), for sysadmins to
investigate and act upon.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
pbs-datastore/src/datastore.rs | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 665333f52..f7cf82d62 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -2469,6 +2469,15 @@ impl DataStore {
)
.context("marking used chunks failed")?;
+ let phase2_start_time = proxmox_time::epoch_i64();
+ if phase2_start_time < phase1_start_time {
+ bail!(
+ "Unexpected system time drift detected, current time {} before start of phase 1 {}",
+ proxmox_time::epoch_to_rfc3339_utc(phase2_start_time)?,
+ proxmox_time::epoch_to_rfc3339_utc(phase1_start_time)?,
+ );
+ }
+
info!("Start GC phase2 (sweep unused chunks)");
if let Some(ref s3_client) = s3_client {
--
2.47.3
reply other threads:[~2026-10-30 11:31 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261030113113.5877-1-c.ebner@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox