From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id CBD9C1FF16F for ; Tue, 8 Jul 2025 19:01:01 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 837871C378; Tue, 8 Jul 2025 19:01:38 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Tue, 8 Jul 2025 19:00:31 +0200 Message-ID: <20250708170114.1556057-4-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250708170114.1556057-1-c.ebner@proxmox.com> References: <20250708170114.1556057-1-c.ebner@proxmox.com> MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.040 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [lib.rs] Subject: [pbs-devel] [PATCH proxmox v6 3/9] s3 client: add dedicated type for s3 object keys X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" S3 objects are uniquely identified within a bucket by their object key [0]. Implements conversion and utility traits to easily convert and encode a string as corresponding object key for the S3 storage backend. Keys might either be full or relative. Full keys are not further expanded when performing api requests, while relative keys are prefixed by the common prefix as configured in the client. This allows for easy key grouping based on client configuration and is used for PBS datastore separation within the same bucket. Further, this adds type checking for s3 client operations requiring an object key and assures keys are properly encoded. [0] https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html Signed-off-by: Christian Ebner --- proxmox-s3-client/src/lib.rs | 4 ++ proxmox-s3-client/src/object_key.rs | 92 +++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 proxmox-s3-client/src/object_key.rs diff --git a/proxmox-s3-client/src/lib.rs b/proxmox-s3-client/src/lib.rs index f65d123f..7286cbd1 100644 --- a/proxmox-s3-client/src/lib.rs +++ b/proxmox-s3-client/src/lib.rs @@ -14,3 +14,7 @@ pub use aws_sign_v4::uri_decode; mod client; #[cfg(feature = "impl")] pub use client::{S3Client, S3ClientOptions}; +#[cfg(feature = "impl")] +mod object_key; +#[cfg(feature = "impl")] +pub use object_key::S3ObjectKey; diff --git a/proxmox-s3-client/src/object_key.rs b/proxmox-s3-client/src/object_key.rs new file mode 100644 index 00000000..cb562c30 --- /dev/null +++ b/proxmox-s3-client/src/object_key.rs @@ -0,0 +1,92 @@ +use anyhow::Error; + +use crate::aws_sign_v4::aws_sign_v4_uri_encode; + +#[derive(Clone, Debug)] +/// S3 Object Key +pub enum S3ObjectKey { + /// Object key which will not be prefixed any further by the client + Full(String), + /// Object key which will be expanded by the client with its configured common prefix + Relative(String), +} + +impl core::convert::From<&str> for S3ObjectKey { + fn from(s: &str) -> Self { + let encoded_key = aws_sign_v4_uri_encode(s, true); + if let Some(encoded_key) = encoded_key.strip_prefix("/") { + Self::Full(encoded_key.to_string()) + } else { + Self::Relative(encoded_key) + } + } +} +impl S3ObjectKey { + /// Convert the given object key to a full key by extending it via given prefix + /// If the object key is already a full key, the prefix is ignored. + pub(crate) fn to_full_key(&self, prefix: &str) -> Self { + match self { + Self::Full(ref key) => Self::Full(key.to_string()), + Self::Relative(ref key) => { + let prefix = aws_sign_v4_uri_encode(prefix, true); + let prefix = prefix.strip_prefix("/").unwrap_or(&prefix); + Self::Full(format!("{prefix}/{key}")) + } + } + } +} + +impl std::ops::Deref for S3ObjectKey { + type Target = str; + + fn deref(&self) -> &Self::Target { + match self { + Self::Full(key) => key, + Self::Relative(key) => key, + } + } +} + +impl std::fmt::Display for S3ObjectKey { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Full(key) => write!(f, "{key}"), + Self::Relative(key) => write!(f, "{key}"), + } + } +} + +impl std::str::FromStr for S3ObjectKey { + type Err = Error; + + fn from_str(s: &str) -> Result { + Ok(Self::from(s)) + } +} + +// Do not mangle with prefixes when de-serializing +impl<'de> serde::Deserialize<'de> for S3ObjectKey { + fn deserialize(deserializer: D) -> Result + where + D: serde::Deserializer<'de>, + { + let object_key = std::borrow::Cow::<'de, str>::deserialize(deserializer)?; + let encoded_key = aws_sign_v4_uri_encode(&object_key, true); + Ok(Self::Full(encoded_key)) + } +} + +impl S3ObjectKey { + /// Generate source key for copy object operations given the source bucket. + /// Extends relative object key variants also by the given prefix. + pub fn to_copy_source_key(&self, source_bucket: &str, prefix: &str) -> Self { + match self { + Self::Full(key) => Self::Full(format!("{source_bucket}{key}")), + Self::Relative(key) => { + let prefix = aws_sign_v4_uri_encode(prefix, true); + let prefix = prefix.strip_prefix("/").unwrap_or(&prefix); + Self::Full(format!("{source_bucket}/{prefix}/{key}")) + } + } + } +} -- 2.47.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel