From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 012E61FF09B for ; Mon, 14 Sep 2026 11:22:11 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id B323121548; Mon, 14 Sep 2026 11:22:10 +0200 (CEST) Message-ID: Date: Mon, 14 Sep 2026 11:22:05 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 proxmox proxmox-backup 0/3] datastore: gc: defer and batch chunk atime updates To: plantulli@gmail.com, pbs-devel@lists.proxmox.com References: <20260811093722.735290-1-plantulli@gmail.com> <20260905223209.097331@pbsiri.virtualsolution.net> Content-Language: en-US, de-DE From: Christian Ebner In-Reply-To: <20260905223209.097331@pbsiri.virtualsolution.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1789377713173 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.669 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: DXYSTBX7UYDCJ6P75ISJ4VMCI66VKBOH X-Message-ID-Hash: DXYSTBX7UYDCJ6P75ISJ4VMCI66VKBOH X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 9/5/26 10:32 PM, plantulli@gmail.com wrote: > Hi, > > as promised in my last mail: the first full end-to-end GC cycle with the > v2 (deferred, batched atime updates) design has now completed on datastore > A. Headline number first: > > start: 2026-08-18 18:52:33 > end: 2026-08-22 18:11:54 (TASK OK) > total: 3 days 23h19m > > phase 1 (mark, deferred batch, size 1,048,576): 59.3h > phase 2 (sweep): 36.0h > > removed garbage: 9.818 TiB > removed chunks: 5,242,712 > on-disk after: 121.78 TiB / 64,562,152 chunks > dedup factor: 10.55 > chunk cache hit ratio: 80.95% > > For the same store, that makes the series so far: > > historical (six cycles, unpatched): 21.25 - 65.81 days > v1 (whole-store prefetch): 7.33 days > v2 (deferred batch): 3.97 days > > The binaries were the v2 series as posted, plus the per-run warmed- > directory tracker I described in my previous mail (skips the redundant > readdir when a later flush revisits an already-warmed directory). > > Where the gain actually came from, with the honest caveats: > > * Phase 2 is the big win: 36.0h vs 121.4h for v1. Normalized per swept > chunk that is ~193 -> ~538 chunks/s (~2.8x) on a comparable sweep size > (84.2M vs 69.8M chunks). One caveat: this cycle had ~3.5x fewer chunks > to actually unlink (5.2M vs 18.2M), and unlinks are the expensive part > of the sweep, so the per-chunk rate comparison flatters v2 somewhat. > Even discounting that, the sweep clearly benefits from metadata still > being warm from the flush readdirs, where in v1 the upfront prefetch > had been evicted long before phase 2 started. > > * Phase 1 was comparable to v1, not faster: 59.3h vs v1's 53.5h plus a > 1h upfront prefetch. Consistent with what I reported earlier about the > first flush on a cold store at the default batch size: warming up to 1M > chunks spread over nearly all 65,536 directories in one synchronous > flush is a lot of work in one place. This is more support for making > the batch size independently tunable for benchmarking, as you > suggested; I briefly started a run at batch size 131,072 but aborted it > for unrelated operational reasons, so I don't have a clean comparison > yet. > > * This run did not start fully cold: the v1 cycle had finished about a > day earlier, so some metadata was still cached. The next scheduled run > on this store (Sep 16) will be a cold start; I'll report back if the > picture changes materially. > > Also worth noting: on our smaller datastore B (~10M chunks), daily GC > with v2 is now routine at ~40-50 minutes total, with phase 1 around 15 > minutes - down from over 17 hours for phase 1 alone before this work > started. > > Thanks, > Enrico Thank you for your additional information, there still is room for further optimization as I tried to emphasize in my review of version 2 of the patch series, please take that into consideration as well. Thanks!