From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id C54981FF09E for ; Tue, 18 Aug 2026 20:38:52 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 323182132C; Tue, 18 Aug 2026 20:38:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787078315; x=1787683115; darn=lists.proxmox.com; h=references:in-reply-to:message-id:subject:from:to:date:from:to:cc :subject:date:message-id:reply-to:content-type; bh=T+Z3OafoXCLDvMrbifk+43gTx1+l8Ss2E2htFQ9ST3o=; b=pmMOw04MsXyABiIXnhjDmGhiaW6X4i1JPzjQA0NgbMDdidRmiUnSqw8LOTi54KW3a1 COymfN/5Mq5FCsE3nkVYP7sQYmV5paKuSl++TxMrXnDvHQscivf9KqoT56J1OgvcOnsh ++JptTJUKyJvvyeKCal2AZLRCNGDioelIcuVgmp5ARc/iDYFOggdUEZs5vqVjxZ5Fk7L 10b3fZMg0JZbEtmNrBXy+PRhw+BE67sOrRjVHOjIoJdwypVMpv/KhtEuny8BUPmbltp8 TUXDSLwDz9ZxWNPZoGDkqnFXCqO5uixvXAZHxM95QDZ9MkmhpDaaZAmW26mH4O5bdk/S XFpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787078315; x=1787683115; h=references:in-reply-to:message-id:subject:from:to:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=T+Z3OafoXCLDvMrbifk+43gTx1+l8Ss2E2htFQ9ST3o=; b=FZ7q4PEC2pVdijVRu+0R/J6yxAL60Mm8FtaBHJyZtbVRdC2Vc8R5bQGpEKCtrMkdPf 4BxlFwrGsJQB6AYMz85npYW8zvvRrrqn7EuCqIeH7gsSxjuDpX5/6j5gcXEGPn5Ya1mi HGSIPO/G/FWGkDhO4zd3qJLdvJSmxTJiUiUZZbFEybgMZhKKaJInTq+krcZwB9iH/vE0 Yj1Skro9GQ7EPVMRSZnXy7BRrhWD4wfYAOlfcjxIV6TNeYLpmEOJWAXMFQZduhegHgYm R/vjttDR+4MNGsnLQ4q735lYhO3nEgIohJWm6WNi1NnMz2Hgc1jo1ZzYEaNPZOrF1xtj A0oQ== X-Gm-Message-State: AOJu0YyoM2ma1Heidzc1npaxmlL2mcAdbJg76p5M0RLMBDDlKm7A9YZY MQZjDvZHZkyBqysifgfuYPoDWiRZ2THzuxq5A/2o3x9dJjLSFweetBoIL0bS4w== X-Gm-Gg: AR+sD11YqdRapgjjWtDADopTkuLRLARxucidF+HD4jQDb63w3RgwnXB/caTL4b73u3y peewx6i9MEmAidnfcIvKCizGR2YM4HFZpnZJvuPCzq1uKMq7RfCQRzIZo2N9Oj7Wiex65jBZRp9 0DlR21z+UHqnuLusSleUPvJXqFO02pGYafE91fcf9JfLqqga7XVIOJj6FpW0Dx2tO+g0cGYlDa8 xAFLvhiUJs5a8TD2mgljg45kMDSXdRfkH57d69Msvhis1ooZ/LBKxDZw1jhf+7T7uPXia5bBLXp ZomY+MxMVgeCpuiatBDJtiZ3kRVpbMHmwOanuw1RteIIv/JIaXE2Bxj4Porh57KP6ubwlvGGodg kaOgApgZnmNMJCBFh5lYQekMPK10jVu9l0K6CIxWw/4+SWxnc111dNbju3Vt2OverEgP8BtLpoD V6mXOtXChPdAXNrxrVyjKr7xsvv/3x7zLARgqCWlVfKbs1r959qX5QtFLRg8Gqwieo3U3CFPQ5Y nIAzP5yTimA9zsqNp3ZZ38fqhzUyy1ezFswCFRs7K90uNqtakYb3MvQmFlYvUKo X-Received: by 2002:a05:600c:3f0e:b0:499:a277:e8b5 with SMTP id 5b1f17b1804b1-499a8f71ecbmr1790615e9.3.1787078315355; Tue, 18 Aug 2026 11:38:35 -0700 (PDT) Date: Tue, 18 Aug 2026 20:38:25 +0200 To: pbs-devel@lists.proxmox.com From: plantulli@gmail.com Subject: Re: [PATCH v2 proxmox proxmox-backup 0/3] datastore: gc: defer and batch chunk atime updates Message-Id: <20260818203825.3076486@pbsiri.virtualsolution.net> X-Mailer: swaks v20240103.0 jetmore.org/john/code/swaks/ In-Reply-To: <20260811093722.735290-1-plantulli@gmail.com> References: <20260811093722.735290-1-plantulli@gmail.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.188 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: GWSTFFKXYA6D56PO7TSPSPGKF4HSJNKH X-Message-ID-Hash: GWSTFFKXYA6D56PO7TSPSPGKF4HSJNKH X-MailFrom: plantulli@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, follow-up with the real-world numbers I promised for datastore A, now that its first full GC cycle with the v1 (whole-store prefetch) series has completed: store: ~82M chunks on disk before the run (81,778,438 entries counted by the initial whole-store prefetch below), 1.269 PiB logical on a 7x raidz1 (28 HDDs) ZFS pool start: 2026-08-10 09:01:09 end: 2026-08-17 16:51:06 (TASK OK) total: 7 days 7h50m prefetch (whole-store): 81,778,438 entries in 59m33s phase 1 (mark): 2d 5h28m (53.5h) phase 2 (sweep): 5d 1h23m (121.4h) removed garbage: 33.788 TiB removed chunks: 18,186,013 on-disk after: 124.64 TiB / 65,997,624 chunks dedup factor: 10.43 chunk cache hit ratio: 81.08% Historical baseline for this store, six completed cycles before the patch: 21.25 / 23.50 / 39.99 / 43.64 / 47.24 / 65.81 days. This run beat the best of those by close to 3x, and the worst by close to 9x. Two things worth reporting honestly, since they bear directly on the v2 design you proposed: * Phase 1 was fast (53.5h) but showed the survival problem in practice. On the dense stretches (multi-TB VM index groups) the GC worker was repeatedly observed blocked in zio_wait <- dbuf_read <- dnode_hold_impl, re-reading metadata the whole-store prefetch had already loaded hours earlier - the ARC was sitting at its adaptive target, not at c_max. I ran an external re-warm (same directory walk, done again mid-run) as a workaround, and it recovered the rate by only about +30%, not fully. This is exactly the failure mode your deferred-batch design (v2) is built to avoid, and I think this run is a fairly clean empirical argument for it: one upfront pass does not survive a phase 1 that runs for two days on a store this size. * Phase 2 (121.4h) turned out to be the larger share of the total time, more than I expected. Its rate varied widely - from ~50-80 chunks/s during dense stretches and heavy overnight contention (up to 10 concurrent backup jobs observed at once, including one writing directly into this same datastore) to ~300 chunks/s when the pool was otherwise idle. One ~4h stretch stayed pinned in dmu_tx_hold_free the whole time (batch chunk deletion), then resolved on its own. dmu_tx_dirty_delay only moved once in the entire week-long run, so the ZFS write throttle was not the dominant limit here - contention for disk time between GC, ingest and other datastores' GCs was. Since sending the v1 numbers above, I've started exercising the v2 (deferred, batched atime update) design you proposed, on this same store. First finding: at this store's chunk density (~1000 chunks per chunk directory on average), a single flush at the default batch size (1,048,576) statistically covers close to all 65,536 chunk directories - so without further care, the whole-store readdir cost effectively repeats on every flush instead of being paid once. I've added a per-run tracker that records which directories were already warmed and skips the redundant readdir on later flushes revisiting them; that's in place now, but I don't yet have a full end-to-end v2 cycle completed on this store to report a headline number for it. None of this changes the headline number: from a worst case of 66 days to 7.3 is already a result I'm happy to ship as-is. But it does suggest v2's warm-right-before-use approach should help more on this store's phase 1 than the numbers from datastore B alone implied, and that phase 2 might benefit from a similar treatment if you'd ever consider it - happy to discuss if useful, otherwise treat that as a side note, not a request to scope-creep this series. CLA: signed and sent to office@proxmox.com on Aug 10; I understand it's being processed. I'll follow up separately once v2 has run end-to-end on this store. Thanks, Enrico