From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 84F861FF0E1 for ; Mon, 10 Aug 2026 07:02:48 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4952A215E3; Mon, 10 Aug 2026 07:02:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786338158; x=1786942958; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=7FtcB1ABnwHcMgaOvMbv4Oadelv0+3myH6K0jTq62z8=; b=CZeKn2Vub7B90AKPC2Hu7Q41VnYY4cJde8nNHu+7PzqL061sS4l7w/vMyW+1LUmmOW YMcxPpiq7o24clOM6ERq5siDJ/jt7pNK7BKIjY4pj2RSg6IkcYf/Sr7e5373EPloAreZ AYugO+TwIhlk3cM8PZf15CkgAC6YWP6znCwLIjixMCPfSPD0+mT19Da02jEk9sp0JI8L Oyp2baNg5GYtHGZmfzy7vNfQiHcHuxnGauOJpuI+fCoAujBniu0UJQ8KsPK+QREwfOiX N8eURtU5631cyZKYbvPt1hM1TX1X+9yaQes2ot5g5mdTW4Dj2HfyCywzUKN1oMCGEf0z TtKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786338158; x=1786942958; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=7FtcB1ABnwHcMgaOvMbv4Oadelv0+3myH6K0jTq62z8=; b=KpQY04hKj2WaSj+lekMk7hLg0PlIxDjtPMnMGIqY/2b5s4fl0zoFdbVNxv5BPtoIl8 kV7lkIF6cZgTJucXWEVD5MM4Ik1DzU04dkK2/FMSZ3rLcaR1Ogsl+qPviXZ6cFiyGk4H Iv2bwtWTJYtgi3cxhrQ1QDnba9k+JpVh5K8pH//G7PFdrC53uZ8D84424OS0fUdm6Qzu Kg9KN+hbVekXB5eyq3Es6RAwNxSSEEXsC4lyo5pW7/GEYitDOa0/DS1QFeBH4iHL86GZ TdKj/8FduZ9eeQXiqb70LYNsWfItbzteqrCpEsx2govy6hbjcuLzXGc/2MTjxytrXG7t naqA== X-Gm-Message-State: AOJu0Yz6uHjDyy8Jf95gfM826sN2DUX+2wWpaJ6JJHuyxy1JdFb8986D tV39GI4N7VO0sAFwudcGhe30GQaPcMzrFeo0p0Dz1rW8UUrjhDa4/mE6ofYJPvHC X-Gm-Gg: AR+sD11dr8S7b1Fm/zL8creaMei3+3JQAUwVZ21jRxKL9KVd/p1oPvHhsfndAfpRM6u bALsrdt3cEE7ZQcAjRjI6wBJRuH/knX0iQv1sdt8MxN7ry69JW5cbY0cpB7gyKMErddg7ZmAtbR sZuVDRn9GSFyY/DI8SKVr5jEJCLWHFpSQdTS52TtdVOGDuH6FMo6/4jzH+TRtPbGLU1FoiRlSSN 5ZpFOwUcDh9qiKpjsdnS3QPb9cyVSvYoP8T1F/Nt3TwugsP4J0Ru8UyMLLHy9WsgQB4pRJ4mmsK d597XSXLqYDANsmhhLKaTFUQaYghpnfNc+Dxs9pRyNftnMNO8QZ02yI9BI3lT6qWkRZV8Z1CfPs oRTZihkhLQSXfC2ZaSzCuTtAU/tBJN6mG2AYAssQBp3RECOlOdeWLp8Uh7PM9VvMclSrWmR+7Sf xAj9XG1z6PpAummtTqKZvpvKaaoOfw/YZm65fMvJZvhTeHpgK6PYFfS448csjAjzAVG+vjBDNrd 1DsUSjwPXeucVet3e2kafzFDrjXMoQVv/v3AYCIT2ig1cZM55Gno/HXAdvC6h2p X-Received: by 2002:a5d:64ce:0:b0:47f:e4e3:d85f with SMTP id ffacd0b85a97d-47ffd9180e2mr40933279f8f.17.1786338158353; Sun, 09 Aug 2026 22:02:38 -0700 (PDT) From: Enrico Plantulli To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup 2/3] datastore: gc: optionally prefetch chunk metadata before phase 1 Date: Mon, 10 Aug 2026 07:01:59 +0200 Message-ID: <20260810050201.347124-3-plantulli@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260810050201.347124-1-plantulli@gmail.com> References: <20260810050201.347124-1-plantulli@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.375 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FORGED_GMAIL_RCVD 1 'From' gmail.com does not match 'Received' headers FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider KAM_NUMSUBJECT 0.5 Subject ends in numbers excluding current years RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: QRQ7DAVAQD66H6ACBQSJJGFJHA2SZDHB X-Message-ID-Hash: QRQ7DAVAQD66H6ACBQSJJGFJHA2SZDHB X-MailFrom: plantulli@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Enrico Plantulli X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Phase 1 of garbage collection resolves each chunk from its digest and calls utimensat() on it directly. It never iterates the chunk directories, so on a cold store every chunk costs an independent, serialized metadata read, issued in digest order - which is uncorrelated with the on-disk layout of the inodes by construction, since the digest picks the directory while the inode number follows creation order. On file systems that prefetch inode metadata while iterating a directory this can be avoided almost entirely: iterating the chunk directories once brings in that metadata in one go, and phase 1 then finds it cached. Measured on a ZFS datastore on spinning disks (7x raidz1 of 4 HDDs, ZFS 2.4.3, cold cache), over 32768 of its 65536 chunk directories, holding 5231473 chunks: readdir over those 32768 directories: 270.9 s, 869726 physical reads lstat() over the same 5231473 entries: 80.1 s, 0 physical reads That is 0.166 physical reads per chunk brought in. 94.7% of the resulting ARC misses were prefetch misses, and zfs_readdir() calls dmu_prefetch_dnode() for the entries it returns, which is where the win comes from - getdents itself needs only the directory entry, not the inode. The pass only helps as long as that metadata is still cached when phase 1 uses it. Note that the pass removes the read stalls only: phase 1 still dirties every chunk's dnode via utimensat(), so once the reads are cached the marking rate is expected to cap at the pool's metadata write-back rate, not at readdir/lstat speed. The pass is not free and it does not help everywhere, so it is opt-in via the new gc-chunk-metadata-prefetch tuning option, and it is skipped for S3 backed datastores: there the local chunk store only holds the cached chunks and the in-use markers, not the full chunk set, so a full pass over the local chunk directories is not what phase 1 has to wait for. A failing prefetch is logged and ignored, except for abort and shutdown requests, which are re-checked in the error path: the pass is an optimization and must never become a new way for garbage collection to fail, but it must not swallow a cancellation either. Signed-off-by: Enrico Plantulli --- docs/storage.rst | 12 ++++++++++ pbs-datastore/src/chunk_store.rs | 38 ++++++++++++++++++++++++++++++++ pbs-datastore/src/datastore.rs | 23 +++++++++++++++++++ 3 files changed, 73 insertions(+) diff --git a/docs/storage.rst b/docs/storage.rst index 2c6ba0a..575ece9 100644 --- a/docs/storage.rst +++ b/docs/storage.rst @@ -686,6 +686,18 @@ There are some tuning related options for the datastore that are more advanced: cache slots, 1048576 (= 1024 * 1024) being the default, 8388608 (= 8192 * 1024) the maximum value. +* ``gc-chunk-metadata-prefetch``: Prefetch chunk metadata before GC phase 1: + Phase 1 reaches each chunk directly by digest and never iterates the chunk + directories. On file systems that prefetch inode metadata while iterating a + directory, such as ZFS, a single pass over the chunk directories before + phase 1 brings in that metadata in one go, instead of leaving phase 1 to + fault it in one chunk at a time. This can reduce phase 1 runtime + substantially on large datastores on rotational disks, at the cost of one + extra pass. The pass only pays off if the metadata is still cached once + phase 1 uses it: on ZFS that is roughly 512 bytes of dnode per chunk, so a + datastore whose dnodes do not fit into the ARC will not benefit. It is + disabled by default and has no effect on S3 backed datastores. + * ``default-verification-workers`` and ``default-verification-readers``: Define the default number of threads used for verification and reading of chunks, respectively. By default, 4 threads are used for verification and 1 thread is used diff --git a/pbs-datastore/src/chunk_store.rs b/pbs-datastore/src/chunk_store.rs index 6d2ffdb..b595d15 100644 --- a/pbs-datastore/src/chunk_store.rs +++ b/pbs-datastore/src/chunk_store.rs @@ -302,6 +302,44 @@ impl ChunkStore { Ok(is_bad) } + /// Iterate all chunk directories once, discarding the entries. + /// + /// No chunk file is opened, read, written or stat'ed, so no chunk access time is touched. + /// The readdir does update the access time of the up to 65536 `.chunks/` subdirectories, + /// which garbage collection never looks at: phase 2 only ever unlinks regular files, and + /// the atime cutoff applies to chunks, not to directories. + /// + /// The only purpose is to let the file system fault in the chunk inode metadata in bulk, so + /// that garbage collection phase 1 finds it cached instead of faulting it in one chunk at a + /// time in digest order, which is uncorrelated with the on-disk layout of the inodes. + /// + /// Returns the number of chunk directory entries seen (chunks, bad chunks and markers). + pub fn prefetch_chunk_metadata(&self, worker: &dyn WorkerTaskContext) -> Result { + let mut last_percentage = 0; + let mut chunk_count = 0; + + for (entry, percentage, _chunk_ext) in self.get_chunk_store_iterator()? { + if last_percentage != percentage { + last_percentage = percentage; + info!("prefetched {percentage}% ({chunk_count} entries)"); + } + + worker.check_abort()?; + worker.fail_on_shutdown()?; + + if let Err(err) = entry { + bail!( + "chunk iterator on chunk store '{}' failed - {err}", + self.name, + ); + } + + chunk_count += 1; + } + + Ok(chunk_count) + } + fn get_chunk_store_iterator( &self, ) -> Result< diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs index c3db522..da77134 100644 --- a/pbs-datastore/src/datastore.rs +++ b/pbs-datastore/src/datastore.rs @@ -2459,6 +2459,29 @@ impl DataStore { 1024 * 1024 }; + if tuning.gc_chunk_metadata_prefetch.unwrap_or(false) { + if s3_client.is_some() { + info!("Chunk metadata prefetch not supported for S3 backed datastore, skipping."); + } else { + info!("Start GC chunk metadata prefetch"); + let start = std::time::Instant::now(); + // Best effort only: this is an optimization, it must never fail the collection. + match self.inner.chunk_store.prefetch_chunk_metadata(worker) { + Ok(chunk_count) => info!( + "Chunk metadata prefetch done, {chunk_count} entries in {}", + TimeSpan::from(start.elapsed()), + ), + Err(err) => { + // an abort or shutdown request must not be swallowed by the + // best effort handling below + worker.check_abort()?; + worker.fail_on_shutdown()?; + warn!("Chunk metadata prefetch failed, continuing without it - {err}"); + } + } + } + } + info!("Start GC phase1 (mark used chunks)"); self.mark_used_chunks( base-commit: 5d95fc20eeae6df1936216b1ab1ab6952472fdcf -- 2.47.3