* [pbs-devel] [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend
@ 2025-07-22 16:36 Christian Ebner
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox 1/1] s3 client: split config api type into 3 config structs Christian Ebner
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Christian Ebner @ 2025-07-22 16:36 UTC (permalink / raw)
To: pbs-devel
These are followup patches to be applied on top of the patch series
found here:
https://lore.proxmox.com/pbs-devel/20250722101106.526438-1-c.ebner@proxmox.com/T/
Most notably, they fix a regression introduced in version v11, as
discovered by Lukas. The api now did incorrectly return also the secret
key for s3 endpoint configurations.
Further, there is a patch which checks bucket access during datastore
creation already in the api, before spawning the datastore create task.
By this, if the bucket cannot be accessed, the user is notified right
away and does not need to re-enter all the datastore configurations.
proxmox:
Christian Ebner (1):
s3 client: split config api type into 3 config structs
proxmox-s3-client/src/api_types.rs | 62 ++++++++++++++++++++++++------
proxmox-s3-client/src/client.rs | 3 +-
2 files changed, 52 insertions(+), 13 deletions(-)
proxmox-backup:
Christian Ebner (2):
config: s3: adapt to new config struct layouts
datastore: check s3 bucket access before create datastore task
pbs-config/src/s3.rs | 7 ++--
pbs-datastore/src/datastore.rs | 11 +++--
src/api2/admin/s3.rs | 7 ++--
src/api2/config/datastore.rs | 76 ++++++++++++++++++++++++----------
src/api2/config/s3.rs | 65 ++++++++++++++++++++---------
5 files changed, 114 insertions(+), 52 deletions(-)
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* [pbs-devel] [PATCH proxmox 1/1] s3 client: split config api type into 3 config structs
2025-07-22 16:36 [pbs-devel] [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend Christian Ebner
@ 2025-07-22 16:36 ` Christian Ebner
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox-backup 1/2] config: s3: adapt to new config struct layouts Christian Ebner
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Christian Ebner @ 2025-07-22 16:36 UTC (permalink / raw)
To: pbs-devel
Splitting the config into 3 structs allows to use them once with
the full config, once without the password and once to only
serialize/deserialize when writing the config.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-s3-client/src/api_types.rs | 62 ++++++++++++++++++++++++------
proxmox-s3-client/src/client.rs | 3 +-
2 files changed, 52 insertions(+), 13 deletions(-)
diff --git a/proxmox-s3-client/src/api_types.rs b/proxmox-s3-client/src/api_types.rs
index 51f71d84..e05ad0f9 100644
--- a/proxmox-s3-client/src/api_types.rs
+++ b/proxmox-s3-client/src/api_types.rs
@@ -82,9 +82,6 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
#[api(
properties: {
- id: {
- schema: S3_CLIENT_ID_SCHEMA,
- },
endpoint: {
schema: S3_ENDPOINT_SCHEMA,
},
@@ -103,9 +100,6 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
"access-key": {
type: String,
},
- "secret-key": {
- type: String,
- },
"path-style": {
type: bool,
optional: true,
@@ -115,15 +109,12 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
type: u64,
optional: true,
},
- }
+ },
)]
#[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
#[serde(rename_all = "kebab-case")]
/// S3 client configuration properties.
pub struct S3ClientConfig {
- /// ID to identify s3 client config.
- #[updater(skip)]
- pub id: String,
/// Endpoint to access S3 object store.
pub endpoint: String,
/// Port to access S3 object store.
@@ -137,8 +128,6 @@ pub struct S3ClientConfig {
pub fingerprint: Option<String>,
/// Access key for S3 object store.
pub access_key: String,
- /// Secret key for S3 object store.
- pub secret_key: String,
/// Use path style bucket addressing over vhost style.
#[serde(skip_serializing_if = "Option::is_none")]
pub path_style: Option<bool>,
@@ -154,3 +143,52 @@ impl S3ClientConfig {
Vec::new()
}
}
+
+#[api(
+ properties: {
+ id: {
+ schema: S3_CLIENT_ID_SCHEMA,
+ },
+ config: {
+ type: S3ClientConfig,
+ },
+ "secret-key": {
+ type: String,
+ },
+ },
+)]
+#[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// S3 client configuration.
+pub struct S3ClientConf {
+ /// ID to identify s3 client config.
+ #[updater(skip)]
+ pub id: String,
+ /// S3 client config.
+ #[serde(flatten)]
+ pub config: S3ClientConfig,
+ /// Secret key for S3 object store.
+ pub secret_key: String,
+}
+
+
+#[api(
+ properties: {
+ id: {
+ schema: S3_CLIENT_ID_SCHEMA,
+ },
+ config: {
+ type: S3ClientConfig,
+ },
+ },
+)]
+#[derive(Serialize, Deserialize, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// S3 client configuration properties without secret.
+pub struct S3ClientConfigWithoutSecret {
+ /// ID to identify s3 client config.
+ pub id: String,
+ /// S3 client config.
+ #[serde(flatten)]
+ pub config: S3ClientConfig,
+}
diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index f418ee39..eb5fc7d9 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -75,6 +75,7 @@ impl S3ClientOptions {
/// Construct options for the S3 client give the provided configuration parameters.
pub fn from_config(
config: S3ClientConfig,
+ secret_key: String,
bucket: String,
common_prefix: String,
) -> Self {
@@ -87,7 +88,7 @@ impl S3ClientOptions {
region: config.region.unwrap_or("us-west-1".to_string()),
fingerprint: config.fingerprint,
access_key: config.access_key,
- secret_key: config.secret_key,
+ secret_key,
put_rate_limit: config.put_rate_limit,
}
}
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 1/2] config: s3: adapt to new config struct layouts
2025-07-22 16:36 [pbs-devel] [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend Christian Ebner
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox 1/1] s3 client: split config api type into 3 config structs Christian Ebner
@ 2025-07-22 16:36 ` Christian Ebner
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox-backup 2/2] datastore: check s3 bucket access before create datastore task Christian Ebner
2025-07-22 20:25 ` [pbs-devel] applied: [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend Thomas Lamprecht
3 siblings, 0 replies; 5+ messages in thread
From: Christian Ebner @ 2025-07-22 16:36 UTC (permalink / raw)
To: pbs-devel
In order to not return the secret key as part of the s3 endpoint
config, split the config into different struct depending on the
usecase. Either use the plain config without id and secret_key,
the struct with id and plain config or the combined variant with
all 3 fields present.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
pbs-config/src/s3.rs | 7 ++--
pbs-datastore/src/datastore.rs | 11 ++++--
src/api2/admin/s3.rs | 7 ++--
src/api2/config/datastore.rs | 12 ++++---
src/api2/config/s3.rs | 65 +++++++++++++++++++++++-----------
5 files changed, 68 insertions(+), 34 deletions(-)
diff --git a/pbs-config/src/s3.rs b/pbs-config/src/s3.rs
index 428f097bb..88c582bcb 100644
--- a/pbs-config/src/s3.rs
+++ b/pbs-config/src/s3.rs
@@ -3,11 +3,10 @@ use std::sync::LazyLock;
use anyhow::Error;
-use proxmox_s3_client::S3ClientConfig;
+use proxmox_s3_client::{S3ClientConf, S3_CLIENT_ID_SCHEMA};
use proxmox_schema::*;
use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
-use pbs_api_types::JOB_ID_SCHEMA;
use pbs_buildcfg::configdir;
@@ -16,10 +15,10 @@ use crate::{open_backup_lockfile, replace_backup_config, BackupLockGuard};
pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
fn init() -> SectionConfig {
- let obj_schema = S3ClientConfig::API_SCHEMA.unwrap_object_schema();
+ let obj_schema = S3ClientConf::API_SCHEMA.unwrap_all_of_schema();
let plugin =
SectionConfigPlugin::new("s3client".to_string(), Some(String::from("id")), obj_schema);
- let mut config = SectionConfig::new(&JOB_ID_SCHEMA);
+ let mut config = SectionConfig::new(&S3_CLIENT_ID_SCHEMA);
config.register_plugin(plugin);
config
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index a051f0556..e8be576f7 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -14,7 +14,7 @@ use tokio::io::AsyncWriteExt;
use tracing::{info, warn};
use proxmox_human_byte::HumanByte;
-use proxmox_s3_client::{S3Client, S3ClientConfig, S3ClientOptions, S3ObjectKey, S3PathPrefix};
+use proxmox_s3_client::{S3Client, S3ClientConf, S3ClientOptions, S3ObjectKey, S3PathPrefix};
use proxmox_schema::ApiType;
use proxmox_sys::error::SysError;
@@ -251,9 +251,14 @@ impl DataStore {
.ok_or_else(|| format_err!("missing bucket for s3 backend"))?;
let (config, _config_digest) = pbs_config::s3::config()?;
- let config: S3ClientConfig = config.lookup(S3_CFG_TYPE_ID, s3_client_id)?;
+ let config: S3ClientConf = config.lookup(S3_CFG_TYPE_ID, s3_client_id)?;
- let options = S3ClientOptions::from_config(config, bucket, self.name().to_owned());
+ let options = S3ClientOptions::from_config(
+ config.config,
+ config.secret_key,
+ bucket,
+ self.name().to_owned(),
+ );
let s3_client = S3Client::new(options)?;
DatastoreBackend::S3(Arc::new(s3_client))
}
diff --git a/src/api2/admin/s3.rs b/src/api2/admin/s3.rs
index 113ebf6fa..9a7b08c81 100644
--- a/src/api2/admin/s3.rs
+++ b/src/api2/admin/s3.rs
@@ -6,7 +6,7 @@ use serde_json::Value;
use proxmox_http::Body;
use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap};
use proxmox_s3_client::{
- S3Client, S3ClientConfig, S3ClientOptions, S3ObjectKey, S3_BUCKET_NAME_SCHEMA,
+ S3Client, S3ClientConf, S3ClientOptions, S3ObjectKey, S3_BUCKET_NAME_SCHEMA,
S3_CLIENT_ID_SCHEMA,
};
use proxmox_schema::*;
@@ -43,11 +43,12 @@ pub async fn check(
_rpcenv: &mut dyn RpcEnvironment,
) -> Result<Value, Error> {
let (config, _digest) = pbs_config::s3::config()?;
- let config: S3ClientConfig = config
+ let config: S3ClientConf = config
.lookup(S3_CFG_TYPE_ID, &s3_client_id)
.context("config lookup failed")?;
- let options = S3ClientOptions::from_config(config, bucket, store_prefix);
+ let options =
+ S3ClientOptions::from_config(config.config, config.secret_key, bucket, store_prefix);
let test_object_key =
S3ObjectKey::try_from(".s3-client-test").context("failed to generate s3 object key")?;
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index 8abed7e32..2702c7db3 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -9,7 +9,7 @@ use serde_json::Value;
use tracing::{info, warn};
use proxmox_router::{http_bail, Permission, Router, RpcEnvironment, RpcEnvironmentType};
-use proxmox_s3_client::{S3Client, S3ClientConfig, S3ClientOptions};
+use proxmox_s3_client::{S3Client, S3ClientConf, S3ClientOptions};
use proxmox_schema::{api, param_bail, ApiType};
use proxmox_section_config::SectionConfigData;
use proxmox_uuid::Uuid;
@@ -147,11 +147,15 @@ pub(crate) fn do_create_datastore(
.ok_or_else(|| format_err!("missing required bucket"))?;
let (config, _config_digest) =
pbs_config::s3::config().context("failed to get s3 config")?;
- let config: S3ClientConfig = config
+ let config: S3ClientConf = config
.lookup(S3_CFG_TYPE_ID, s3_client_id)
.with_context(|| format!("no '{s3_client_id}' in config"))?;
- let options =
- S3ClientOptions::from_config(config, bucket, datastore.name.to_owned());
+ let options = S3ClientOptions::from_config(
+ config.config,
+ config.secret_key,
+ bucket,
+ datastore.name.to_owned(),
+ );
let s3_client = S3Client::new(options).context("failed to create s3 client")?;
// Fine to block since this runs in worker task
proxmox_async::runtime::block_on(s3_client.head_bucket())
diff --git a/src/api2/config/s3.rs b/src/api2/config/s3.rs
index 891c017c7..04b801028 100644
--- a/src/api2/config/s3.rs
+++ b/src/api2/config/s3.rs
@@ -4,7 +4,10 @@ use hex::FromHex;
use serde_json::Value;
use proxmox_router::{http_bail, Permission, Router, RpcEnvironment};
-use proxmox_s3_client::{S3ClientConfig, S3ClientConfigUpdater};
+use proxmox_s3_client::{
+ S3ClientConf, S3ClientConfig, S3ClientConfigUpdater, S3ClientConfigWithoutSecret,
+ S3_CLIENT_ID_SCHEMA,
+};
use proxmox_schema::{api, param_bail, ApiType};
use pbs_api_types::{
@@ -20,7 +23,7 @@ use pbs_config::s3::{self, S3_CFG_TYPE_ID};
returns: {
description: "List configured s3 clients.",
type: Array,
- items: { type: S3ClientConfig },
+ items: { type: S3ClientConfigWithoutSecret },
},
access: {
permission: &Permission::Privilege(&[], PRIV_SYS_AUDIT, false),
@@ -30,7 +33,7 @@ use pbs_config::s3::{self, S3_CFG_TYPE_ID};
pub fn list_s3_client_config(
_param: Value,
rpcenv: &mut dyn RpcEnvironment,
-) -> Result<Vec<S3ClientConfig>, Error> {
+) -> Result<Vec<S3ClientConfigWithoutSecret>, Error> {
let (config, digest) = s3::config()?;
let list = config.convert_to_typed_array(S3_CFG_TYPE_ID)?;
rpcenv["digest"] = hex::encode(digest).into();
@@ -42,10 +45,17 @@ pub fn list_s3_client_config(
protected: true,
input: {
properties: {
+ id: {
+ schema: S3_CLIENT_ID_SCHEMA,
+ },
config: {
type: S3ClientConfig,
flatten: true,
},
+ "secret-key": {
+ type: String,
+ description: "S3 secret key",
+ },
},
},
access: {
@@ -54,15 +64,23 @@ pub fn list_s3_client_config(
)]
/// Create a new s3 client configuration.
pub fn create_s3_client_config(
+ id: String,
config: S3ClientConfig,
+ secret_key: String,
_rpcenv: &mut dyn RpcEnvironment,
) -> Result<(), Error> {
let _lock = s3::lock_config()?;
let (mut section_config, _digest) = s3::config()?;
- if section_config.sections.contains_key(&config.id) {
- param_bail!("id", "s3 client config '{}' already exists.", config.id);
+ if section_config.sections.contains_key(&id) {
+ param_bail!("id", "s3 client config '{}' already exists.", &id);
}
+ let config = S3ClientConf {
+ id: id.clone(),
+ config,
+ secret_key,
+ };
+
section_config.set_data(&config.id, S3_CFG_TYPE_ID, &config)?;
s3::save_config(§ion_config)?;
@@ -77,7 +95,7 @@ pub fn create_s3_client_config(
},
},
},
- returns: { type: S3ClientConfig },
+ returns: { type: S3ClientConfigWithoutSecret },
access: {
permission: &Permission::Privilege(&[], PRIV_SYS_AUDIT, false),
},
@@ -86,9 +104,9 @@ pub fn create_s3_client_config(
pub fn read_s3_client_config(
id: String,
rpcenv: &mut dyn RpcEnvironment,
-) -> Result<S3ClientConfig, Error> {
+) -> Result<S3ClientConfigWithoutSecret, Error> {
let (config, digest) = s3::config()?;
- let s3_client_config: S3ClientConfig = config.lookup(S3_CFG_TYPE_ID, &id)?;
+ let s3_client_config: S3ClientConfigWithoutSecret = config.lookup(S3_CFG_TYPE_ID, &id)?;
rpcenv["digest"] = hex::encode(digest).into();
Ok(s3_client_config)
@@ -120,6 +138,11 @@ pub enum DeletableProperty {
type: S3ClientConfigUpdater,
flatten: true,
},
+ "secret-key": {
+ type: String,
+ description: "S3 client secret key.",
+ optional: true,
+ },
delete: {
description: "List of properties to delete.",
type: Array,
@@ -143,6 +166,7 @@ pub enum DeletableProperty {
pub fn update_s3_client_config(
id: String,
update: S3ClientConfigUpdater,
+ secret_key: Option<String>,
delete: Option<Vec<DeletableProperty>>,
digest: Option<String>,
_rpcenv: &mut dyn RpcEnvironment,
@@ -156,46 +180,47 @@ pub fn update_s3_client_config(
crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
}
- let mut data: S3ClientConfig = config.lookup(S3_CFG_TYPE_ID, &id)?;
+ let mut data: S3ClientConf = config.lookup(S3_CFG_TYPE_ID, &id)?;
if let Some(delete) = delete {
for delete_prop in delete {
match delete_prop {
DeletableProperty::Port => {
- data.port = None;
+ data.config.port = None;
}
DeletableProperty::Region => {
- data.region = None;
+ data.config.region = None;
}
DeletableProperty::Fingerprint => {
- data.fingerprint = None;
+ data.config.fingerprint = None;
}
DeletableProperty::PathStyle => {
- data.path_style = None;
+ data.config.path_style = None;
}
}
}
}
if let Some(endpoint) = update.endpoint {
- data.endpoint = endpoint;
+ data.config.endpoint = endpoint;
}
if let Some(port) = update.port {
- data.port = Some(port);
+ data.config.port = Some(port);
}
if let Some(region) = update.region {
- data.region = Some(region);
+ data.config.region = Some(region);
}
if let Some(access_key) = update.access_key {
- data.access_key = access_key;
+ data.config.access_key = access_key;
}
if let Some(fingerprint) = update.fingerprint {
- data.fingerprint = Some(fingerprint);
+ data.config.fingerprint = Some(fingerprint);
}
if let Some(path_style) = update.path_style {
- data.path_style = Some(path_style);
+ data.config.path_style = Some(path_style);
}
- if let Some(secret_key) = update.secret_key {
+
+ if let Some(secret_key) = secret_key {
data.secret_key = secret_key;
}
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 2/2] datastore: check s3 bucket access before create datastore task
2025-07-22 16:36 [pbs-devel] [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend Christian Ebner
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox 1/1] s3 client: split config api type into 3 config structs Christian Ebner
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox-backup 1/2] config: s3: adapt to new config struct layouts Christian Ebner
@ 2025-07-22 16:36 ` Christian Ebner
2025-07-22 20:25 ` [pbs-devel] applied: [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend Thomas Lamprecht
3 siblings, 0 replies; 5+ messages in thread
From: Christian Ebner @ 2025-07-22 16:36 UTC (permalink / raw)
To: pbs-devel
In order to give immediate feedback to the caller, so it is not
required to re-enter all the datastore configuration if the bucket
cannot be accessed.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
src/api2/config/datastore.rs | 78 ++++++++++++++++++++++++------------
1 file changed, 53 insertions(+), 25 deletions(-)
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index 2702c7db3..f7b852cb7 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -137,31 +137,28 @@ pub(crate) fn do_create_datastore(
match backend_config.ty.unwrap_or_default() {
DatastoreBackendType::Filesystem => (),
DatastoreBackendType::S3 => {
- let s3_client_id = backend_config
- .client
- .as_ref()
- .ok_or_else(|| format_err!("missing required client"))?;
- let bucket = backend_config
- .bucket
- .clone()
- .ok_or_else(|| format_err!("missing required bucket"))?;
- let (config, _config_digest) =
- pbs_config::s3::config().context("failed to get s3 config")?;
- let config: S3ClientConf = config
- .lookup(S3_CFG_TYPE_ID, s3_client_id)
- .with_context(|| format!("no '{s3_client_id}' in config"))?;
- let options = S3ClientOptions::from_config(
- config.config,
- config.secret_key,
- bucket,
- datastore.name.to_owned(),
- );
- let s3_client = S3Client::new(options).context("failed to create s3 client")?;
- // Fine to block since this runs in worker task
- proxmox_async::runtime::block_on(s3_client.head_bucket())
- .context("failed to access bucket")?;
-
if !overwrite_in_use {
+ let s3_client_id = backend_config
+ .client
+ .as_ref()
+ .ok_or_else(|| format_err!("missing required client"))?;
+ let bucket = backend_config
+ .bucket
+ .clone()
+ .ok_or_else(|| format_err!("missing required bucket"))?;
+ let (config, _config_digest) =
+ pbs_config::s3::config().context("failed to get s3 config")?;
+ let config: S3ClientConf = config
+ .lookup(S3_CFG_TYPE_ID, s3_client_id)
+ .with_context(|| format!("no '{s3_client_id}' in config"))?;
+ let options = S3ClientOptions::from_config(
+ config.config,
+ config.secret_key,
+ bucket,
+ datastore.name.to_owned(),
+ );
+ let s3_client = S3Client::new(options).context("failed to create s3 client")?;
+
let object_key = S3ObjectKey::try_from(S3_DATASTORE_IN_USE_MARKER)
.context("failed to generate s3 object key")?;
if let Some(response) =
@@ -180,8 +177,8 @@ pub(crate) fn do_create_datastore(
bail!("Bucket already contains datastore in use");
}
}
+ backend_s3_client = Some(Arc::new(s3_client));
}
- backend_s3_client = Some(Arc::new(s3_client));
}
}
}
@@ -352,6 +349,37 @@ pub fn create_datastore(
};
let store_name = config.name.to_string();
+
+ let backend_config: DatastoreBackendConfig = config.backend.as_deref().unwrap_or("").parse()?;
+ match backend_config.ty.unwrap_or_default() {
+ DatastoreBackendType::Filesystem => (),
+ DatastoreBackendType::S3 => {
+ let s3_client_id = backend_config
+ .client
+ .as_ref()
+ .ok_or_else(|| format_err!("missing required client"))?;
+ let bucket = backend_config
+ .bucket
+ .clone()
+ .ok_or_else(|| format_err!("missing required bucket"))?;
+ let (config, _config_digest) =
+ pbs_config::s3::config().context("failed to get s3 config")?;
+ let config: S3ClientConf = config
+ .lookup(S3_CFG_TYPE_ID, s3_client_id)
+ .with_context(|| format!("no '{s3_client_id}' in config"))?;
+ let options = S3ClientOptions::from_config(
+ config.config,
+ config.secret_key,
+ bucket,
+ store_name.clone(),
+ );
+ let s3_client = S3Client::new(options).context("failed to create s3 client")?;
+ // Fine to block since this runs in worker task
+ proxmox_async::runtime::block_on(s3_client.head_bucket())
+ .context("failed to access bucket")?;
+ }
+ }
+
WorkerTask::new_thread(
"create-datastore",
Some(store_name.clone()),
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* [pbs-devel] applied: [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend
2025-07-22 16:36 [pbs-devel] [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend Christian Ebner
` (2 preceding siblings ...)
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox-backup 2/2] datastore: check s3 bucket access before create datastore task Christian Ebner
@ 2025-07-22 20:25 ` Thomas Lamprecht
3 siblings, 0 replies; 5+ messages in thread
From: Thomas Lamprecht @ 2025-07-22 20:25 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Christian Ebner
Am 22.07.25 um 18:36 schrieb Christian Ebner:
> These are followup patches to be applied on top of the patch series
> found here:
> https://lore.proxmox.com/pbs-devel/20250722101106.526438-1-c.ebner@proxmox.com/T/
>
> Most notably, they fix a regression introduced in version v11, as
> discovered by Lukas. The api now did incorrectly return also the secret
> key for s3 endpoint configurations.
>
> Further, there is a patch which checks bucket access during datastore
> creation already in the api, before spawning the datastore create task.
> By this, if the bucket cannot be accessed, the user is notified right
> away and does not need to re-enter all the datastore configurations.
>
> proxmox:
>
> Christian Ebner (1):
> s3 client: split config api type into 3 config structs
>
> proxmox-s3-client/src/api_types.rs | 62 ++++++++++++++++++++++++------
> proxmox-s3-client/src/client.rs | 3 +-
> 2 files changed, 52 insertions(+), 13 deletions(-)
>
> proxmox-backup:
>
> Christian Ebner (2):
> config: s3: adapt to new config struct layouts
> datastore: check s3 bucket access before create datastore task
>
> pbs-config/src/s3.rs | 7 ++--
> pbs-datastore/src/datastore.rs | 11 +++--
> src/api2/admin/s3.rs | 7 ++--
> src/api2/config/datastore.rs | 76 ++++++++++++++++++++++++----------
> src/api2/config/s3.rs | 65 ++++++++++++++++++++---------
> 5 files changed, 114 insertions(+), 52 deletions(-)
>
applied, thanks!
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2025-07-22 20:24 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-07-22 16:36 [pbs-devel] [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend Christian Ebner
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox 1/1] s3 client: split config api type into 3 config structs Christian Ebner
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox-backup 1/2] config: s3: adapt to new config struct layouts Christian Ebner
2025-07-22 16:36 ` [pbs-devel] [PATCH proxmox-backup 2/2] datastore: check s3 bucket access before create datastore task Christian Ebner
2025-07-22 20:25 ` [pbs-devel] applied: [PATCH proxmox{, -backup} 0/3] followups for PBS s3 backend Thomas Lamprecht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox