From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 7B6B41FF0E4 for ; Tue, 11 Aug 2026 11:37:57 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 22F2D21480; Tue, 11 Aug 2026 11:37:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786441065; x=1787045865; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=NhA0dEBUUSWQ+gN2l+4KCoa/qs5Y4Nnuq6kSd2mT2UE=; b=a2NvX3S2Mp5hW7iDedx5GrP0tBQ3aH/xwqtXGxT9OR+qJJXlUznO7fmdCPjzFEogss F4LzRQvU4RiKIIXiwb+IrwjRu/RT7NZwkN93R02Owipt/O8ERPzgZT1K+02sowapthyg ey/oaL0xiYlbgw4zu4EOC569v0UvtN5OgJHpKDDfGloQlyyd5QBhgXeusrhLiW7w82+P RMi3xUI7M9xasW70LuU+e8Cj19sJ29XDsh3uosBpJSmXhpPusJt6fuVAiWApGUiU3Bc+ NeIivf7Dv+lP3jC1/l8cHNgqxhx5NGFiaPnOhjFZkLnfbQcON5Q/ocs2TvKH7jUIxNNe LCZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786441065; x=1787045865; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=NhA0dEBUUSWQ+gN2l+4KCoa/qs5Y4Nnuq6kSd2mT2UE=; b=bs3rJvwha2Fy3ix10BfftflhYyFUgBv/w/MgbSO19aIRz6vPfgOi1WcZhKtacI1bbo GaAvNBjcwixyTmeeaJ1HDRk6FaXljcW2x9tQ2Fuf/ioPvoj7cyolorALq+QEzqRpvvao ZdRJySPL9ZfZ0QfKVTDiNcwtYGk8kPRdE9D3h9xMpqnYV99+0SCCpyp5LUr657QxKwsq 956xkKEvWDYOKTGC3w2keJzrdsFIexTTiPoqxAziTPUJ24o/yTCZ4HCgfwLZlyozPZJX 8BLJSQ82oDlDYwdbNtma1ulURouAVFgNd8fkzrtT3+VW1Z8F4A+OgMDMPYv8Zv/bPd8q Asdw== X-Gm-Message-State: AOJu0YwQFFKkH6h6/u5igz0HZ59M0rL9FxzFueScoNI4sbkUaujORVEe nkiv3mlIlslAx3UYeObdcXTiYTbkmfysTorU7WVNEU9ZJRn/aOMBYgDMWAvkvA== X-Gm-Gg: AR+sD12tnf6KL7In7mOui3B1thcLLMP6IfXBigrvBMhqzF9fjNY0RI5nayefqJyMzEl 1CP4NHEEDjY8XxFLF1IKQiqTM7XU4uu0+CmdmaU+0+qRf+RNRZ8xRIGfKBbPwa2r/aheZrztdu1 TJGFnPOaS+wBjp6USaRKL9eVwauka3bELu+csQ8/wPR1/Rv3AACBFenHkT9I6PJm6VU4YSa/uGB q83FL7WGSDkqdqB/oKvRInGLlNFFYSx9K8DAQDa+32pFy2YxbwCMlaiHmpPYx2sGmBtLbldFfuX qivKvrWmxD6bULaIFNe2b8vKLyYcPF3t0W+TE16RUAX+lQrEUjub4d0VmN1Py0xywakZOKq26R8 6RbcaFtbi7u5j2pnE45PyYs0qlOOs4rGyAf5DlwQhckhwHKjLKiNX0wC8PRkbLEw0IeMFTDhSlI mdrNcDpr7KLhQdBXEnSZjUNFVS1sRctL972GhvR/UXcrMq3MYgjzwwgyFnyjdsDflODq0+EwfX+ ufF/un80wiQ7fzj5HbEoGzX3Ox1TIb234Rhe4O/orE/Bw8IE14c7alZzxpCBm+n X-Received: by 2002:a05:600c:1d20:b0:499:621a:2ec2 with SMTP id 5b1f17b1804b1-49978430814mr28139795e9.3.1786441064409; Tue, 11 Aug 2026 02:37:44 -0700 (PDT) From: Enrico Plantulli To: pbs-devel@lists.proxmox.com Subject: [PATCH v2 proxmox proxmox-backup 0/3] datastore: gc: defer and batch chunk atime updates Date: Tue, 11 Aug 2026 11:37:18 +0200 Message-ID: <20260811093722.735290-1-plantulli@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260810050201.347124-1-plantulli@gmail.com> References: <20260810050201.347124-1-plantulli@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.375 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: Y3PSWRGLA5IEJUYWG27EDGFX4VHBNI6S X-Message-ID-Hash: Y3PSWRGLA5IEJUYWG27EDGFX4VHBNI6S X-MailFrom: plantulli@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Enrico Plantulli X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, this is v2 of the chunk metadata prefetch series, reworked along the lines suggested in the review of v1: instead of one whole-store readdir pass before phase 1, the chunk atime updates themselves are now deferred into a bounded list and flushed sorted by digest, opening and iterating each chunk directory right before its chunks are touched, with the utimensat() calls going through the open directory file descriptor. Changes since v1: * dropped the whole-store prefetch pass and the prefetch_chunk_metadata() helper entirely; * added deferred, batched atime updates behind the same opt-in gc-chunk-metadata-prefetch tuning option (kept opt-in as requested: backends that do not prefetch on readdir, e.g. network attached storage, may regress); * added a separate gc-prefetch-batch-size tuning option (default 1048576, range 1024 - 16M) as an independent knob, as suggested, to make benchmarking on different backends easy; * the batch is used for filesystem backed datastores only; S3 keeps the immediate path, since its in-use markers need per-chunk handling anyway; * a chunk found missing at flush time is handled as before (touch the .bad companions, warn), but the warning can only name the digest, not the referencing index file - noted in patch 2; * GUI patch now covers both options. Why the rework was needed, empirically: I ran the v1 one-pass version on the 72M chunk production datastore (A in the v1 thread). The pass itself was fast (81.8M directory entries in 59m32s), but the warmed metadata did not survive until a phase 1 that runs for hours reached it: the GC worker was observed blocked in zio_wait <- dbuf_read <- zap_get_leaf_byblk (and dnode_hold_impl), re-reading from disk the very blocks the pass had loaded hours earlier, with the ARC sitting at its adaptive target far below c_max. External re-walks recovered the rate only modestly (~+30%). Warming right before use removes that window entirely, which is what this v2 does. The batch size bound also controls the readdir amplification: a flush of N sorted entries spans up to min(N, 65536) directories, so about N/65536 chunks are served per directory read. The default of 1M gives ~16 used chunks per directory read on a full store; benchmarking larger values is exactly what the separate option is for. Testing: compile-tested (cargo build, clippy with no new warnings, fmt, cargo test) against current master of both repositories. I have not yet run this exact v2 end-to-end in production; I will follow up with figures from datastore A and B once it has. NAS-backed figures would be very welcome from anyone with such a setup, as discussed. CLA: signed and sent to office@proxmox.com on Aug 10; office has been in touch. proxmox: Enrico Plantulli (1): pbs-api-types: add gc chunk metadata prefetch tuning options pbs-api-types/src/datastore.rs | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) proxmox-backup: Enrico Plantulli (2): datastore: gc: optionally defer and batch chunk atime updates ui: tuning: add GC chunk metadata prefetch options docs/storage.rst | 20 ++++++++ pbs-datastore/src/chunk_store.rs | 102 +++++++++++++++++++++++++++++++++++++++ pbs-datastore/src/datastore.rs | 93 +++++++++++++++++++++++++++++++++----- www/Utils.js | 12 +++++ www/datastore/OptionView.js | 27 +++++++++++ 5 files changed, 245 insertions(+), 9 deletions(-) Thanks, Enrico Plantulli