From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 2C5FC1FF0AE for ; Tue, 15 Sep 2026 18:00:57 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 647AD215B6; Tue, 15 Sep 2026 18:00:56 +0200 (CEST) From: Shan Shaji To: pbs-devel@lists.proxmox.com Subject: [RFC PATCH proxmox-backup 0/4] fix #7903: rebuild local S3 chunk markers Date: Tue, 15 Sep 2026 17:59:31 +0200 Message-ID: <20260915155935.135460-1-s.shaji@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1789488037904 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.472 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: YKZMYCS7S4TWLTJV6GXE5S4R5AKSAV5F X-Message-ID-Hash: YKZMYCS7S4TWLTJV6GXE5S4R5AKSAV5F X-MailFrom: s.shaji@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Sending this as an RFC to get an initial feedback. I'd really appreciate any suggestions or pointers to anything I may have missed. Reusing an S3-backed datastore currently restores the backup metadata to the local cache without creating local chunk markers. A garbage collection run is then needed to recreate markers for chunks referenced by the backup indexes. This series addresses #7908 by listing the chunk objects in S3 and creating zero-byte local markers when a datastore is recreated with --reuse-datastore. It also adds an optional rebuild-chunk-markers parameter to the s3-refresh API and exposes it through proxmox-backup-manager: proxmox-backup-manager datastore s3-refresh --rebuild-chunk-markers true The option is disabled by default for explicit refreshes. When enabled, it creates markers for all chunk objects with valid chunk keys, including chunks not referenced by the restored backup indexes. Existing cached chunk files are replaced with empty markers. Shan Shaji (4): datastore: factor out s3 chunk objects pagination into a helper fix #7908: datastore: create zero-byte markers when re-using datastore datastore: api: add option to rebuild S3 chunk markers datastore: backup-manager: expose option to rebuild s3 chunk markers pbs-datastore/src/datastore.rs | 120 +++++++++++++++----- src/api2/admin/datastore.rs | 23 +++- src/api2/config/datastore.rs | 6 +- src/bin/proxmox_backup_manager/datastore.rs | 7 ++ 4 files changed, 121 insertions(+), 35 deletions(-) -- 2.47.3