From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 75F671FF0AF for ; Fri, 06 Nov 2026 18:23:46 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id BDEDB21413; Fri, 06 Nov 2026 18:23:17 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup v2 17/43] datastore: s3: fix namespace level bulk remove not honoring group locks Date: Fri, 6 Nov 2026 18:22:09 +0100 Message-ID: <20261106172235.890542-18-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261106172235.890542-1-c.ebner@proxmox.com> References: <20261106172235.890542-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793985775922 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.466 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: NLHTRPNYVYHKDZ6NVQSWFYSKONJYA7AW X-Message-ID-Hash: NLHTRPNYVYHKDZ6NVQSWFYSKONJYA7AW X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: For datastores backed by S3 recursive namespace removal was optimized by bulk removing all object keys, excluding these which correspond to a protected snapshot within the group. This reduced the number of potentially more cost intensive POST calls for DeleteObjects, as compared to the regular per-group cleanup. However, this did not take into account the snapshot/group locking and in consequence can lead to potential inconsistencies between the state in the local cache and the state on the S3 backend. To fix this, completely drop this optimization based on incomplete assumptions and do the cleanup on snapshot/group level. Fixes: 6ff078a5a ("datastore: prune groups/snapshots from s3 object store backend") Signed-off-by: Christian Ebner --- pbs-datastore/src/datastore.rs | 23 ----------------------- 1 file changed, 23 deletions(-) diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs index e6766a3b9..d5608d4da 100644 --- a/pbs-datastore/src/datastore.rs +++ b/pbs-datastore/src/datastore.rs @@ -1168,29 +1168,6 @@ impl DataStore { // Process deepest first so a parent's rmdir sees an empty directory. children.sort_by_key(|b| std::cmp::Reverse(b.depth())); - // When destroying the full subtree, a bulk S3 delete under the root prefix catches - // any orphaned group-level metadata (owner/notes) before per-level cleanup iterates. - if delete_groups { - if let DatastoreBackend::S3(s3_client) = &self.backend()? { - let ns_dir = ns.path(); - let ns_prefix = ns_dir - .to_str() - .ok_or_else(|| format_err!("invalid namespace path prefix"))?; - let prefix = format!("{S3_CONTENT_PREFIX}/{ns_prefix}"); - let delete_objects_error = proxmox_async::runtime::block_on( - s3_client.delete_objects_by_prefix_with_suffix_filter( - &S3PathPrefix::Some(prefix), - PROTECTED_MARKER_FILENAME, - &[GROUP_OWNER_FILE_NAME, GROUP_NOTES_FILE_NAME], - ), - )?; - if !delete_objects_error.is_empty() { - crate::s3::log_s3_delete_objects_errors(&delete_objects_error); - bail!("deleting objects failed"); - } - } - } - let mut removed_all_requested = true; let mut stats = BackupGroupDeleteStats::default(); for child in &children { -- 2.47.3