From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 8849A1FF09C for ; Mon, 19 Oct 2026 18:04:19 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 09089215F8; Mon, 19 Oct 2026 18:04:17 +0200 (CEST) From: Shan Shaji To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Date: Mon, 19 Oct 2026 18:03:50 +0200 Message-ID: <20261019160359.20851-1-s.shaji@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792425851295 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.428 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: J4KILPHOQWUJUVL7MOZGRRINKMA423W3 X-Message-ID-Hash: J4KILPHOQWUJUVL7MOZGRRINKMA423W3 X-MailFrom: s.shaji@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This series depends on the following series: [1] Reusing an S3-backed datastore currently restores the backup metadata to the local cache without creating local chunk markers. A garbage collection run is then required to recreate markers for chunks referenced by the backup indexes. To fix this, add an optional 'rebuild-chunk-markers' option during datastore creation to recreate the markers. If the number of chunks is large, datastore creation together with marker file creation could take a significant amount of time. Therefore, the option is disabled by default. When I tested the changes on cloudflare R2 storage, it took 3m 27.1s to create 352670 markers. Changes since RFC [2]: Thanks @chris for the feedback and offlist clarifications. - Remove rebuild-chunk-markers option from the s3-refresh command. - The rebuild-chunk-markers option is now exclusive to datastore creation. Exposed the option in the UI and backup-manager. - Use 'impl Trait' instead of generics. - Made the chunkstore dependency on the pagination helper explicit. - Add a comment explaining why holding the chunk store is acceptable. - Add progress logs with throttling. - Move the LogThrottle implementation to the proxmox-log crate. - [1] https://lore.proxmox.com/pbs-devel/20261014071715.88590-1-c.ebner@proxmox.com/T/ - [2] https://lore.proxmox.com/pbs-devel/2d665615-ed2e-4b82-b166-85d0f5eb41d5@proxmox.com/T/#t proxmox-backup: Shan Shaji (8): refactor: remove local `LogThrottle` in favor of proxmox-log datastore: factor out s3 chunk objects pagination into a helper pbs-datastore: inherit proxmox-log crate from workspace datastore: add method to create zero-byte chunk markers fix #7908: datastore: api: add option to create zero-byte chunk markers fix #7908: datastore: ui: add option to create zero-byte chunk markers fix #7908: backup-manager: add option to create zero-byte chunk markers docs: add explanation regarding ``rebuild-chunk-markers`` option docs/storage.rst | 7 +- pbs-client/src/lib.rs | 3 - pbs-client/src/log_throttle.rs | 36 ----- pbs-datastore/Cargo.toml | 1 + pbs-datastore/src/datastore.rs | 143 +++++++++++++++----- proxmox-backup-client/src/main.rs | 3 +- src/api2/config/datastore.rs | 15 +- src/api2/node/disks/directory.rs | 1 + src/api2/node/disks/zfs.rs | 1 + src/bin/proxmox_backup_manager/datastore.rs | 6 + www/window/DataStoreEdit.js | 20 +++ 11 files changed, 158 insertions(+), 78 deletions(-) delete mode 100644 pbs-client/src/log_throttle.rs proxmox: Shan Shaji (1): proxmox-log: add LogThrottle implementation proxmox-log/src/lib.rs | 1 + proxmox-log/src/log_throttle.rs | 39 +++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) create mode 100644 proxmox-log/src/log_throttle.rs Summary over all repositories: 13 files changed, 198 insertions(+), 78 deletions(-) -- Generated by murpp 0.10.0