[pbs-devel] [PATCH proxmox{, -backup} v2 0/4] s3: extend config by provider-quirks and client options by feature list

[pbs-devel] [PATCH proxmox{, -backup} v2 0/4] s3: extend config by provider-quirks and client options by feature list

[Christian Ebner]

These patches extend the s3 client configuration by the additional
`provider-quirks` enum, allowing to switch to provider specific implementation
details. The provider specific quirks are then mapped to a list of features and
limitations, added to the s3 client options.

As first use-case, the `If-None-Match` header is not set during put object
requests to Backblaze B2 or Infomaniak object stores, as these fail with an
error with status code 501, therefore chunk uploads will fail.

The patches expose the provider quirks also in an dropdown list in the advanced
column of the s3 endpoint edit window.

Changes since v1:
- add Infomaniak as further provider not supporting the If-None-Match header
- introduce dedicate enum api type for both provider quirks and s3 client features
- add logic to map from provider to feature list
- skip If-None-Match header based on feature list, not provider

proxmox:

Christian Ebner (2):
  s3 client: api types: extend client config by optional provider quirks
  s3 client: extend client options by feature list

 proxmox-s3-client/examples/s3_client.rs |  1 +
 proxmox-s3-client/src/api_types.rs      | 31 +++++++++++++++++++++++++
 proxmox-s3-client/src/client.rs         | 26 +++++++++++++++++++--
 3 files changed, 56 insertions(+), 2 deletions(-)


proxmox-backup:

Christian Ebner (2):
  api: s3 config: allow to update or delete endpoint quirks
  ui: s3 endpoint: add provider specific quirk selector

 src/api2/config/s3.rs      |  8 ++++++++
 www/window/S3ClientEdit.js | 18 ++++++++++++++++++
 2 files changed, 26 insertions(+)


Summary over all repositories:
  5 files changed, 82 insertions(+), 2 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox v2 1/2] s3 client: api types: extend client config by optional provider quirks

[Christian Ebner]

Add optional provider specific quirks to the client configuration,
allowing to set a variant of the client to follow some custom
implementation variants.

Add backblaze as first variant, as it does not accept the
`If-None-Match` header in put object requests.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 1:
- use dedicated enum api type ProviderQuirks instead of schema only
- rename from `quirks` to `provider-quirks` to be more specific and
  imply correlation to the s3 object store provider

 proxmox-s3-client/src/api_types.rs | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/proxmox-s3-client/src/api_types.rs b/proxmox-s3-client/src/api_types.rs
index eff15f37..7d1ad24e 100644
--- a/proxmox-s3-client/src/api_types.rs
+++ b/proxmox-s3-client/src/api_types.rs
@@ -80,6 +80,19 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
     .max_length(63)
     .schema();
 
+#[api]
+#[derive(Copy, Clone, Deserialize, Serialize, PartialEq, Eq)]
+#[serde(rename_all = "kebab-case")]
+/// Provider specific feature implementation quirks.
+pub enum ProviderQuirks {
+    /// Implementation quirks for Backblaze.
+    Backblaze,
+    /// Implementation quirks for Infomaniak.
+    Infomaniak,
+}
+serde_plain::derive_display_from_serialize!(ProviderQuirks);
+serde_plain::derive_fromstr_from_deserialize!(ProviderQuirks);
+
 #[api(
     properties: {
         endpoint: {
@@ -109,6 +122,10 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
             type: u64,
             optional: true,
         },
+        "provider-quirks": {
+            type: ProviderQuirks,
+            optional: true,
+        },
     },
 )]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -134,6 +151,9 @@ pub struct S3ClientConfig {
     /// Rate limit for put requests given as #reqest/s.
     #[serde(skip_serializing_if = "Option::is_none")]
     pub put_rate_limit: Option<u64>,
+    /// Provider specific feature implementation quirks.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub provider_quirks: Option<ProviderQuirks>,
 }
 
 impl S3ClientConfig {
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox v2 2/2] s3 client: extend client options by feature list

[Christian Ebner]

Adds a feature list to the s3 client options, which allows to define
a set of provider specific implementation features or limitations to
be used by the client.

As a first limitation, the SkipIfNoneMatchHeader is added for
provider not accepting and implementing this headers functionality.

Defines also an associated helper function on S3ClientOptions to map
a given provider quirk to a feature list.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 1:
- use feature list instead of provider for s3 client options, adding
  flexibility for further extension
- add helper to map from provider quirk to feature list
- set If-None-Match header based on feature instead of provider quirk

 proxmox-s3-client/examples/s3_client.rs |  1 +
 proxmox-s3-client/src/api_types.rs      | 11 +++++++++++
 proxmox-s3-client/src/client.rs         | 26 +++++++++++++++++++++++--
 3 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/proxmox-s3-client/examples/s3_client.rs b/proxmox-s3-client/examples/s3_client.rs
index 1cbb3939..21b0e9ce 100644
--- a/proxmox-s3-client/examples/s3_client.rs
+++ b/proxmox-s3-client/examples/s3_client.rs
@@ -38,6 +38,7 @@ async fn run() -> Result<(), anyhow::Error> {
         // `openssl s_client -connect testbucket.s3.pve-c1.local:7480 < /dev/null | openssl x509 -fingerprint -sha256 -noout`
         fingerprint: Some("<s3-api-fingerprint>".to_string()),
         put_rate_limit: None,
+        features: Vec::new(),
     };
 
     // Creating a client instance and connect to api endpoint
diff --git a/proxmox-s3-client/src/api_types.rs b/proxmox-s3-client/src/api_types.rs
index 7d1ad24e..a87e0804 100644
--- a/proxmox-s3-client/src/api_types.rs
+++ b/proxmox-s3-client/src/api_types.rs
@@ -211,3 +211,14 @@ pub struct S3ClientConfigWithoutSecret {
     #[serde(flatten)]
     pub config: S3ClientConfig,
 }
+
+#[api]
+#[derive(Copy, Clone, Deserialize, Serialize, PartialEq, Eq)]
+#[serde(rename_all = "kebab-case")]
+/// Provider specific implementation feature or limitation.
+pub enum S3ClientFeature {
+    /// If-None-Match http header not implemented and not accepted.
+    SkipIfNoneMatchHeader,
+}
+serde_plain::derive_display_from_serialize!(S3ClientFeature);
+serde_plain::derive_fromstr_from_deserialize!(S3ClientFeature);
diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 3a981bf4..ce57722a 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -22,7 +22,7 @@ use proxmox_http::client::HttpsConnector;
 use proxmox_http::{Body, RateLimit, RateLimiter};
 use proxmox_schema::api_types::CERT_FINGERPRINT_SHA256_SCHEMA;
 
-use crate::api_types::S3ClientConfig;
+use crate::api_types::{ProviderQuirks, S3ClientConfig, S3ClientFeature};
 use crate::aws_sign_v4::AWS_SIGN_V4_DATETIME_FORMAT;
 use crate::aws_sign_v4::{aws_sign_v4_signature, aws_sign_v4_uri_encode};
 use crate::object_key::S3ObjectKey;
@@ -69,6 +69,8 @@ pub struct S3ClientOptions {
     pub fingerprint: Option<String>,
     /// Rate limit for put requests given as #reqest/s.
     pub put_rate_limit: Option<u64>,
+    /// Provider implementation specific features and limitations
+    pub features: Vec<S3ClientFeature>,
 }
 
 impl S3ClientOptions {
@@ -90,6 +92,18 @@ impl S3ClientOptions {
             access_key: config.access_key,
             secret_key,
             put_rate_limit: config.put_rate_limit,
+            features: Self::map_provider_quirks_to_features(config.provider_quirks),
+        }
+    }
+
+    fn map_provider_quirks_to_features(
+        provider_quirks: Option<ProviderQuirks>,
+    ) -> Vec<S3ClientFeature> {
+        match provider_quirks {
+            Some(ProviderQuirks::Backblaze) | Some(ProviderQuirks::Infomaniak) => {
+                vec![S3ClientFeature::SkipIfNoneMatchHeader]
+            }
+            _ => Vec::new(),
         }
     }
 }
@@ -393,7 +407,15 @@ impl S3Client {
             .header(header::CONTENT_TYPE, "binary/octet");
 
         if !replace {
-            request = request.header(header::IF_NONE_MATCH, "*");
+            // Some providers not implement this and fails with error if the header is set,
+            // see https://forum.proxmox.com/threads/168834/post-786278
+            if !self
+                .options
+                .features
+                .contains(&S3ClientFeature::SkipIfNoneMatchHeader)
+            {
+                request = request.header(header::IF_NONE_MATCH, "*");
+            }
         }
 
         let request = request.body(object_data)?;
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v2 1/2] api: s3 config: allow to update or delete endpoint quirks

[Christian Ebner]

Allows to update the configured quirk value, to be used by the s3
endpoint config edit window as well as the cli.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 1:
- adapt config key to new provider-quirks instead of quirks only

 src/api2/config/s3.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/api2/config/s3.rs b/src/api2/config/s3.rs
index 04b801028..56fa2c41e 100644
--- a/src/api2/config/s3.rs
+++ b/src/api2/config/s3.rs
@@ -125,6 +125,8 @@ pub enum DeletableProperty {
     Fingerprint,
     /// Delete the path-style property.
     PathStyle,
+    /// Delete the provider quirks property.
+    ProviderQuirks,
 }
 
 #[api(
@@ -197,6 +199,9 @@ pub fn update_s3_client_config(
                 DeletableProperty::PathStyle => {
                     data.config.path_style = None;
                 }
+                DeletableProperty::ProviderQuirks => {
+                    data.config.provider_quirks = None;
+                }
             }
         }
     }
@@ -219,6 +224,9 @@ pub fn update_s3_client_config(
     if let Some(path_style) = update.path_style {
         data.config.path_style = Some(path_style);
     }
+    if let Some(provider_quirks) = update.provider_quirks {
+        data.config.provider_quirks = Some(provider_quirks);
+    }
 
     if let Some(secret_key) = secret_key {
         data.secret_key = secret_key;
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v2 2/2] ui: s3 endpoint: add provider specific quirk selector

[Christian Ebner]

Add an advanced column with a provider specific quirk selector, which
allows to set the s3 endpoint accordingly and use the provider
specific implementation detail in the s3 client.

Clear the value if it is set to the default one, being no quirks.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 1:
- add Infomaniak as further provider with quirks
- adapt to `provider-quirks` instead of `quirks` only

 www/window/S3ClientEdit.js | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/www/window/S3ClientEdit.js b/www/window/S3ClientEdit.js
index f0a5efba8..dc3d69284 100644
--- a/www/window/S3ClientEdit.js
+++ b/www/window/S3ClientEdit.js
@@ -119,6 +119,24 @@ Ext.define('PBS.window.S3ClientEdit', {
                 },
             },
         ],
+
+        advancedColumn1: [
+            {
+                xtype: 'proxmoxKVComboBox',
+                name: 'provider-quirks',
+                fieldLabel: gettext('Provider specific quirks'),
+                value: '__default__',
+                defaultValue: '__default__',
+                comboItems: [
+                    ['__default__', 'None (default)'],
+                    ['backblaze', 'Backblaze B2'],
+                    ['infomaniak', 'Infomaniak'],
+                ],
+                cbind: {
+                    deleteEmpty: '{!isCreate}',
+                },
+            },
+        ],
     },
 
     getValues: function () {
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox{, -backup} v2 0/4] s3: extend config by provider-quirks and client options by feature list

[Lukas Wagner]

On Tue Jul 29, 2025 at 10:17 AM CEST, Christian Ebner wrote:
> These patches extend the s3 client configuration by the additional
> `provider-quirks` enum, allowing to switch to provider specific implementation
> details. The provider specific quirks are then mapped to a list of features and
> limitations, added to the s3 client options.
>
> As first use-case, the `If-None-Match` header is not set during put object
> requests to Backblaze B2 or Infomaniak object stores, as these fail with an
> error with status code 501, therefore chunk uploads will fail.
>
> The patches expose the provider quirks also in an dropdown list in the advanced
> column of the s3 endpoint edit window.
>

Looks good to me:

Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>

Also quickly compiled and installed, I can at least attest that new GUI
field works as expected, however I did not test the quirks against an
actually quirky provider.

Tested-by: Lukas Wagner <l.wagner@proxmox.com>


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] superseded: [PATCH proxmox{, -backup} v2 0/4] s3: extend config by provider-quirks and client options by feature list

[Christian Ebner]

superseded-by version 3:
https://lore.proxmox.com/pbs-devel/20250804065331.45272-1-c.ebner@proxmox.com/T/


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel