From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id BC51A1FF185 for ; Mon, 21 Jul 2025 15:51:23 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 1159A10760; Mon, 21 Jul 2025 15:52:34 +0200 (CEST) Mime-Version: 1.0 Date: Mon, 21 Jul 2025 15:52:00 +0200 Message-Id: Cc: "pbs-devel" From: "Hannes Laimer" To: "Proxmox Backup Server development discussion" X-Mailer: aerc 0.20.1-112-gd31995f1e20b References: <20250719125035.9926-1-c.ebner@proxmox.com> <20250719125035.9926-22-c.ebner@proxmox.com> In-Reply-To: <20250719125035.9926-22-c.ebner@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1753105913417 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.025 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH proxmox-backup v9 18/46] datastore: create namespace marker in s3 backend X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" On Sat Jul 19, 2025 at 2:50 PM CEST, Christian Ebner wrote: > The S3 object store only allows to store objects, referenced by their > key. For backup namespaces datastores however use directories, so > they cannot be represented as one to one mapping. > > Instead, create an empty marker file for each namespace and operate > based on that. > > Signed-off-by: Christian Ebner > --- > changes since version 8: > - use upload_on_replace_with_retry > > pbs-datastore/src/datastore.rs | 10 ++++++++++ > 1 file changed, 10 insertions(+) > > diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs > index bc829c5b8..5bb4e1777 100644 > --- a/pbs-datastore/src/datastore.rs > +++ b/pbs-datastore/src/datastore.rs > @@ -44,6 +44,7 @@ static DATASTORE_MAP: LazyLock>>> = > LazyLock::new(|| Mutex::new(HashMap::new())); > > const GROUP_NOTES_FILE_NAME: &str = "notes"; > +const NAMESPACE_MARKER_FILENAME: &str = ".namespace"; > > /// checks if auth_id is owner, or, if owner is a token, if > /// auth_id is the user of the token > @@ -613,6 +614,15 @@ impl DataStore { > // construct ns before mkdir to enforce max-depth and name validity > let ns = BackupNamespace::from_parent_ns(parent, name)?; > > + if let DatastoreBackend::S3(s3_client) = self.backend()? { > + let object_key = crate::s3::object_key_from_path(&ns.path(), NAMESPACE_MARKER_FILENAME) > + .context("invalid namespace marker object key")?; hmmm, I didn't test this, but with our NS max depth of 7 (IIRC) and the linux max filename length of ~250 this "could" be larger than the 1024 bytes an object key can be. Maybe we should check for that, even though I doubt this would ever be a problem... > + let _is_duplicate = proxmox_async::runtime::block_on( > + s3_client.upload_no_replace_with_retry(object_key, hyper::body::Bytes::from("")), > + ) > + .context("failed to create namespace on s3 backend")?; > + } > + > let mut ns_full_path = self.base_path(); > ns_full_path.push(ns.path()); > _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel