From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 2A2A31FF0A8 for ; Sat, 05 Sep 2026 22:32:46 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 20EFC21492; Sat, 05 Sep 2026 22:32:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788640348; x=1789245148; darn=lists.proxmox.com; h=references:in-reply-to:message-id:subject:from:to:date:from:to:cc :subject:date:message-id:reply-to:content-type; bh=LW/8dD97Zg9ampmZCicfp45itUDnEtJw/oPnAdtPXIk=; b=nmw5vXoc6jZVoH6evM8I3BLvURziyTmE1TpUAnXPke1caNXdewU7LmJX1yTWJSayge 4R+j0krwS2ffChc4bC14SnQesogvIfRRjs388NkpsKCx2u5eCnVzF3S7IeC3QKyrnlBQ bZRuy5BJor71F1rC0AHtxdiD9WX551FFI4Igl4hOFdhWtA1XCfJ2sHNTQySiH9PDVnDK NnCzCOvCWoj53ud0Q1r5NDwthybrXyDUgp2v83SI4uOk/9mk8iDy8NOWaARmfWl3Ht5f 5Y6pkvU+UKgiohjq95SgmGvlEuzFktdKO+iqAkkwWyRz7PImwRNjIzK9JAs6eYlhdCUm gDjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788640348; x=1789245148; h=references:in-reply-to:message-id:subject:from:to:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=LW/8dD97Zg9ampmZCicfp45itUDnEtJw/oPnAdtPXIk=; b=eGlwLOTRV6017IoOZiR+m9sQ2eNjDNaXfDQ4q+pfu9UGgxZ6dtWr2vMKPTlfYHT4vx Pki/DOjM38JsxB2okPTTc6Z6dcolwxmVpNYcI5LSaLNE8JuXKM4aSB1U9wvI9av2BoY+ AWvXpul1UECh9ufz9iq3qCZzYXA4v+4VXrwZ/NdYwbqRRssbX4x3pG39d5Hp6puV1KQN jpfkV+qFmQZfh7zZB32UVMtdbqRnthxt60SqYRrRQ/6aSH2ZgTmf6zBNBHE6UoPIZcpR komazWLtvugboytZFhVScPcyzPErGg9Xv8jCxi9/07xWPv7vMHCOtC1uNyG/rIvxH7CO HtTg== X-Gm-Message-State: AFuF++koYVm+dzIYy2EvZ1IgjoQa7rTSQJ+94c3edh9irSRR8JWJWobi akLGJxeGzMUNTudtuM9Ii6DpfU6+nLZ3ZmRToG/vR9DVSphr7TGM6kHGEOtODg== X-Gm-Gg: AYBFou1xlMgYZcLfC7SjO6luS2wfNsE036jidoE3lBPXUFf+/P8jrCFv75+Q1hfLZLx daSNyXSsMGTga6IDYrx6fQOb8kbLCfx9I+G6lmnuHwGcs0jgVTKwl8gLPNWQeEz1O/MNEDc1xxZ bd/lUyKyJmJtMvztaJo5Fn26huvze8TihLsltsYZu+TrJ1VPekmZ0W+kIXGRSbOphLpmiGUbywB YkVi4ZixZCoIu7SYuAL5Wy22RhG7u/9+B9avbnwtYfmPx4PiYtcsOD/vRA3f19t4kZVDe1JUCBN cinb0gPvttU1ZNF4iKs2P4bt3rxyP2odR5SN53XksjAzsX8avF34AC2nvbFTcfNHysy6hT9XUft SqW92/kMx9kaatCDSKlGeGi3Ubh+gCOMeIVU69wfI0fEP2A8jd2kLBNWqkqvCxdAw5k7O4dJLqm Hon062Z0YZcdpU3tBI5rKxlJDwLGl45+lZeKFu9nfCvengKJm8J3cTIPExHlbJJNXjNEGsUE71x MohIXUnaoF5ny/560AIs2thGzGTrBiXkxwlKnZpuLl+9vqUpUTuIxWNeUmjm3c= X-Received: by 2002:a05:600c:1da8:b0:49c:cf18:494e with SMTP id 5b1f17b1804b1-49cf825a530mr166430305e9.12.1788640348032; Sat, 05 Sep 2026 13:32:28 -0700 (PDT) Date: Sat, 05 Sep 2026 22:32:09 +0200 To: pbs-devel@lists.proxmox.com From: plantulli@gmail.com Subject: Re: [PATCH v2 proxmox proxmox-backup 0/3] datastore: gc: defer and batch chunk atime updates Message-Id: <20260905223209.097331@pbsiri.virtualsolution.net> X-Mailer: swaks v20240103.0 jetmore.org/john/code/swaks/ In-Reply-To: <20260811093722.735290-1-plantulli@gmail.com> References: <20260811093722.735290-1-plantulli@gmail.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.167 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: HKIGFCFE2XX7YXMDFG2FUA5OXXUSJMM3 X-Message-ID-Hash: HKIGFCFE2XX7YXMDFG2FUA5OXXUSJMM3 X-MailFrom: plantulli@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, as promised in my last mail: the first full end-to-end GC cycle with the v2 (deferred, batched atime updates) design has now completed on datastore A. Headline number first: start: 2026-08-18 18:52:33 end: 2026-08-22 18:11:54 (TASK OK) total: 3 days 23h19m phase 1 (mark, deferred batch, size 1,048,576): 59.3h phase 2 (sweep): 36.0h removed garbage: 9.818 TiB removed chunks: 5,242,712 on-disk after: 121.78 TiB / 64,562,152 chunks dedup factor: 10.55 chunk cache hit ratio: 80.95% For the same store, that makes the series so far: historical (six cycles, unpatched): 21.25 - 65.81 days v1 (whole-store prefetch): 7.33 days v2 (deferred batch): 3.97 days The binaries were the v2 series as posted, plus the per-run warmed- directory tracker I described in my previous mail (skips the redundant readdir when a later flush revisits an already-warmed directory). Where the gain actually came from, with the honest caveats: * Phase 2 is the big win: 36.0h vs 121.4h for v1. Normalized per swept chunk that is ~193 -> ~538 chunks/s (~2.8x) on a comparable sweep size (84.2M vs 69.8M chunks). One caveat: this cycle had ~3.5x fewer chunks to actually unlink (5.2M vs 18.2M), and unlinks are the expensive part of the sweep, so the per-chunk rate comparison flatters v2 somewhat. Even discounting that, the sweep clearly benefits from metadata still being warm from the flush readdirs, where in v1 the upfront prefetch had been evicted long before phase 2 started. * Phase 1 was comparable to v1, not faster: 59.3h vs v1's 53.5h plus a 1h upfront prefetch. Consistent with what I reported earlier about the first flush on a cold store at the default batch size: warming up to 1M chunks spread over nearly all 65,536 directories in one synchronous flush is a lot of work in one place. This is more support for making the batch size independently tunable for benchmarking, as you suggested; I briefly started a run at batch size 131,072 but aborted it for unrelated operational reasons, so I don't have a clean comparison yet. * This run did not start fully cold: the v1 cycle had finished about a day earlier, so some metadata was still cached. The next scheduled run on this store (Sep 16) will be a cold start; I'll report back if the picture changes materially. Also worth noting: on our smaller datastore B (~10M chunks), daily GC with v2 is now routine at ~40-50 minutes total, with phase 1 around 15 minutes - down from over 17 hours for phase 1 alone before this work started. Thanks, Enrico