From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 642541FF0E1 for ; Mon, 10 Aug 2026 07:02:43 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 2B290215E6; Mon, 10 Aug 2026 07:02:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786338154; x=1786942954; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=G3rdaPk+Na+PLm+uRt9r0/1ANgIVtYoM97x8aS4/Wqw=; b=dArKbtHX++1c7Zo2z2kyFuGTwUreKpqCxe4l0k2nMopxcd+fpFOBa7T6zd4xq2RgtZ rcYhp5KfKKpN5hHEVNTQmuH8dw2Ui6bO5qeczY5kxr2Am1obZAmacO4Ubm2VmXTtH+VO 4UhzPg9SmpSZMvMae2UvdpB9aMsrsI5otTBGf+6HOKINsMnvFnNoGRM8++cBiKRA17sf UClkv+cEDGs6EBCVXKpqXL993bTlOr0UqJ2spAnK3oWQB429aoBmUKNo8oN4TGNkAf80 btdFeeJ6KWVx/gOsb1CTr629BIXf1/gQJ2feMIEl5Qfv31B3KE8kHwEoplWDuC538rzD CR/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786338154; x=1786942954; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=G3rdaPk+Na+PLm+uRt9r0/1ANgIVtYoM97x8aS4/Wqw=; b=O7O/4Jsk7rqwpPfbQyojC4J8FLPJG0unoaiLwBxvGgK+Dhw/DmU7J9om73zZyH8iRY xQEgWY+r+lhDgyc4VqqBkZ7x2uyq8xAUBJRpJchzHmWCy2EiGtlD5SlZrfzih+0+ZF8V k5sD8ZGkr6FeYBl2wgjDjUkiHPdpU2ouYQJcoliaNGH3NifKgwh6znPS+klEdpsnIVvo KG/20WsyrMrXmNgjLWglYBjvSAP3SV/K/AI9fj/kFsiIi/pNoxesHG6hWYAcWGvKEDId Lho68NK/xgU/K1ex/lByatB3UGrabTJ5qVRJ8tYgu0+6/1stzyD35T/i+s5tTmsoQzCX vIBQ== X-Gm-Message-State: AOJu0YxjW3mp3kbeCuScWFkd+HVo3bKPoNa10q1wcJmNwa7OCMyl4/+Y zw2/uTCQCWr3zGnq4VU9q62h5H8GMIH6BC+dBzJ0NacMvyJ9NZBxDAMsrTansyhl X-Gm-Gg: AR+sD12/5n0BWuo/CNMldcpiwuI7qQUp41mVoar2tALf3BsmpKfko2zQnIiJhNR0Btb XtlcC88uxE/eY0RRNjF+JrnF1a7fxFTywtHTm/1sFzJLd6IdhYyq0+RRXLBOevoQOT1AjndP2YO zUTIvvHX1PZCTgOGK/0AOYwgN+iv13WcBkClkAavKiIIGlMdOwR26/Iegl6NHdfit8oGJ2RNnqU jtRfvJeluRR3hRvmh3taaCSpNho30DK77c2z4gf7XZLczzMeaTCZ9PbmYnsCyPGumdKiBzhzdJT v3XI+BowdndQ75QOFVxPZJ8sFNhho6WgYwqgXlTiTo3f5dxyCxFoZsKY0HBJiN/2uFF7DsJ97O7 ztzbmd6y9zWieBIAHNclopcCSXKVq2DXqrbn4N2Rz82oeCgxg0Ygir+w+A2cLMRdbIK4dPqxU79 Bl/qmgDRSh7Eq9mT6Of/BKThlwk+pKSmqrqg7NScNHZhqamqbvbyhSie8RAnhWMO3l4Juatqrk9 eH26+tBRsJtiJzoe92cZSjbV5gQAcjLrzh/CKc66WY66VrkLTYmUXDnsRiHp6zoDw== X-Received: by 2002:a05:600c:242:b0:493:c845:bc20 with SMTP id 5b1f17b1804b1-49959e0174cmr263714255e9.4.1786338154256; Sun, 09 Aug 2026 22:02:34 -0700 (PDT) From: Enrico Plantulli To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox 1/3] pbs-api-types: add gc chunk metadata prefetch tuning option Date: Mon, 10 Aug 2026 07:01:58 +0200 Message-ID: <20260810050201.347124-2-plantulli@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260810050201.347124-1-plantulli@gmail.com> References: <20260810050201.347124-1-plantulli@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.750 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 7BBUKHH2UGQSKR4CXV6JYUZNPMNTCLAY X-Message-ID-Hash: 7BBUKHH2UGQSKR4CXV6JYUZNPMNTCLAY X-MailFrom: plantulli@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Enrico Plantulli X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Garbage collection phase 1 reaches each chunk by building its path from the digest, so it never iterates the chunk directories. On file systems that prefetch inode metadata while iterating a directory, that leaves a large win unused: a single readdir pass over .chunks/ brings in the metadata in bulk, while phase 1 on its own faults it in one chunk at a time, in an order uncorrelated with the on-disk layout. Add an opt-in tuning option so that garbage collection can do that pass before phase 1. It defaults to false because the pass costs time on file systems that do not prefetch inodes on readdir, and on datastores whose metadata already fits in the cache. Signed-off-by: Enrico Plantulli --- pbs-api-types/src/datastore.rs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs index 93ccaf0..76ecc5b 100644 --- a/pbs-api-types/src/datastore.rs +++ b/pbs-api-types/src/datastore.rs @@ -278,6 +278,14 @@ pub const GC_CACHE_CAPACITY_SCHEMA: Schema = schema: GC_CACHE_CAPACITY_SCHEMA, optional: true, }, + "gc-chunk-metadata-prefetch": { + description: + "Iterate the chunk directories once before garbage collection phase 1, so the \ + file system can fault in chunk metadata in bulk instead of one chunk at a time", + optional: true, + default: false, + type: bool, + }, "default-verification-workers": { schema: VERIFY_JOB_VERIFY_THREADS_SCHEMA, optional: true, @@ -304,6 +312,8 @@ pub struct DatastoreTuning { #[serde(skip_serializing_if = "Option::is_none")] pub gc_cache_capacity: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub gc_chunk_metadata_prefetch: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub default_verification_workers: Option, #[serde(skip_serializing_if = "Option::is_none")] pub default_verification_readers: Option, base-commit: e3e3ff11b9b92fe1ace89b84c1e15c150e2db660 -- 2.47.3