From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id B7F981FF0B2 for ; Fri, 25 Sep 2026 14:12:02 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5302F216A2; Fri, 25 Sep 2026 14:12:02 +0200 (CEST) Message-ID: <2d665615-ed2e-4b82-b166-85d0f5eb41d5@proxmox.com> Date: Fri, 25 Sep 2026 14:11:58 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Christian Ebner Subject: Re: [RFC PATCH proxmox-backup 0/4] fix #7903: rebuild local S3 chunk markers To: Shan Shaji , pbs-devel@lists.proxmox.com References: <20260915155935.135460-1-s.shaji@proxmox.com> Content-Language: en-US, de-DE In-Reply-To: <20260915155935.135460-1-s.shaji@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1790338319112 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.617 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: P45R25XKMQ75WQBH5QICCHI2WIL73MLY X-Message-ID-Hash: P45R25XKMQ75WQBH5QICCHI2WIL73MLY X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Thanks for the patches, left a few suggestions and major concerns on individual patches. It is especially crucial to take the possible runtime of this into account! What is currently complete missing is any form of exposing this optional behavior to the user in the UI. It might make sense to have such a flag in the datastore create window, which is the main use-case for this. I see less use for normal s3 refresh operations. On 9/15/26 6:00 PM, Shan Shaji wrote: > Sending this as an RFC to get an initial feedback. I'd really appreciate > any suggestions or pointers to anything I may have missed. > > Reusing an S3-backed datastore currently restores the backup metadata to > the local cache without creating local chunk markers. A garbage collection > run is then needed to recreate markers for chunks referenced by the backup > indexes. > > This series addresses #7908 by listing the chunk objects in S3 and creating > zero-byte local markers when a datastore is recreated with > --reuse-datastore. It also adds an optional rebuild-chunk-markers parameter > to the s3-refresh API and exposes it through proxmox-backup-manager: > > proxmox-backup-manager datastore s3-refresh --rebuild-chunk-markers true > > The option is disabled by default for explicit refreshes. When enabled, > it creates markers for all chunk objects with valid chunk keys, including > chunks not referenced by the restored backup indexes. Existing cached > chunk files are replaced with empty markers. > > Shan Shaji (4): > datastore: factor out s3 chunk objects pagination into a helper > fix #7908: datastore: create zero-byte markers when re-using datastore > datastore: api: add option to rebuild S3 chunk markers > datastore: backup-manager: expose option to rebuild s3 chunk markers > > pbs-datastore/src/datastore.rs | 120 +++++++++++++++----- > src/api2/admin/datastore.rs | 23 +++- > src/api2/config/datastore.rs | 6 +- > src/bin/proxmox_backup_manager/datastore.rs | 7 ++ > 4 files changed, 121 insertions(+), 35 deletions(-) > > -- > 2.47.3 > > > >