[pbs-devel] [PATCH proxmox{, -backup} v3 0/4] s3: extend config by provider-quirks and client options by feature list

[pbs-devel] [PATCH proxmox{, -backup} v3 0/4] s3: extend config by provider-quirks and client options by feature list

[Christian Ebner]

These patches extend the s3 client configuration by the additional
`provider-quirks` enum, allowing to switch to provider specific implementation
details. The provider specific quirks are then mapped to a list of features and
limitations, added to the s3 client options.

As first use-case, the `If-None-Match` header is not set during put object
requests to Backblaze B2 or Infomaniak object stores, as these fail with an
error with status code 501, therefore chunk uploads will fail.

The patches expose the provider quirks also in an dropdown list in the advanced
column of the s3 endpoint edit window.

Changes since v2:
- Rebased onto current master

Changes since v1:
- add Infomaniak as further provider not supporting the If-None-Match header
- introduce dedicate enum api type for both provider quirks and s3 client features
- add logic to map from provider to feature list
- skip If-None-Match header based on feature list, not provider

proxmox:

Christian Ebner (2):
  s3 client: api types: extend client config by optional provider quirks
  s3 client: extend client options by feature list

 proxmox-s3-client/examples/s3_client.rs |  1 +
 proxmox-s3-client/src/api_types.rs      | 31 +++++++++++++++++++++++++
 proxmox-s3-client/src/client.rs         | 26 +++++++++++++++++++--
 3 files changed, 56 insertions(+), 2 deletions(-)


proxmox-backup:

Christian Ebner (2):
  api: s3 config: allow to update or delete endpoint quirks
  ui: s3 endpoint: add provider specific quirk selector

 src/api2/config/s3.rs      |  8 ++++++++
 www/window/S3ClientEdit.js | 18 ++++++++++++++++++
 2 files changed, 26 insertions(+)


Summary over all repositories:
  5 files changed, 82 insertions(+), 2 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox v3 1/2] s3 client: api types: extend client config by optional provider quirks

[Christian Ebner]

Add optional provider specific quirks to the client configuration,
allowing to set a variant of the client to follow some custom
implementation variants.

Add backblaze as first variant, as it does not accept the
`If-None-Match` header in put object requests.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
---
changes since version 2:
- no changes

 proxmox-s3-client/src/api_types.rs | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/proxmox-s3-client/src/api_types.rs b/proxmox-s3-client/src/api_types.rs
index 265fedb9..3db6f7a4 100644
--- a/proxmox-s3-client/src/api_types.rs
+++ b/proxmox-s3-client/src/api_types.rs
@@ -80,6 +80,19 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
     .max_length(63)
     .schema();
 
+#[api]
+#[derive(Copy, Clone, Deserialize, Serialize, PartialEq, Eq)]
+#[serde(rename_all = "kebab-case")]
+/// Provider specific feature implementation quirks.
+pub enum ProviderQuirks {
+    /// Implementation quirks for Backblaze.
+    Backblaze,
+    /// Implementation quirks for Infomaniak.
+    Infomaniak,
+}
+serde_plain::derive_display_from_serialize!(ProviderQuirks);
+serde_plain::derive_fromstr_from_deserialize!(ProviderQuirks);
+
 #[api(
     properties: {
         endpoint: {
@@ -109,6 +122,10 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
             type: u64,
             optional: true,
         },
+        "provider-quirks": {
+            type: ProviderQuirks,
+            optional: true,
+        },
     },
 )]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -134,6 +151,9 @@ pub struct S3ClientConfig {
     /// Rate limit for put requests given as #reqest/s.
     #[serde(skip_serializing_if = "Option::is_none")]
     pub put_rate_limit: Option<u64>,
+    /// Provider specific feature implementation quirks.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub provider_quirks: Option<ProviderQuirks>,
 }
 
 impl S3ClientConfig {
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox v3 2/2] s3 client: extend client options by feature list

[Christian Ebner]

Adds a feature list to the s3 client options, which allows to define
a set of provider specific implementation features or limitations to
be used by the client.

As a first limitation, the SkipIfNoneMatchHeader is added for
provider not accepting and implementing this headers functionality.

Defines also an associated helper function on S3ClientOptions to map
a given provider quirk to a feature list.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
---
changes since version 2:
- rebased onto current master

 proxmox-s3-client/examples/s3_client.rs |  1 +
 proxmox-s3-client/src/api_types.rs      | 11 +++++++++++
 proxmox-s3-client/src/client.rs         | 26 +++++++++++++++++++++++--
 3 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/proxmox-s3-client/examples/s3_client.rs b/proxmox-s3-client/examples/s3_client.rs
index 4a9625cb..600e2586 100644
--- a/proxmox-s3-client/examples/s3_client.rs
+++ b/proxmox-s3-client/examples/s3_client.rs
@@ -38,6 +38,7 @@ async fn run() -> Result<(), anyhow::Error> {
         // `openssl s_client -connect testbucket.s3.pve-c1.local:7480 < /dev/null | openssl x509 -fingerprint -sha256 -noout`
         fingerprint: Some("<s3-api-fingerprint>".to_string()),
         put_rate_limit: None,
+        features: Vec::new(),
     };
 
     // Creating a client instance and connect to api endpoint
diff --git a/proxmox-s3-client/src/api_types.rs b/proxmox-s3-client/src/api_types.rs
index 3db6f7a4..baac6909 100644
--- a/proxmox-s3-client/src/api_types.rs
+++ b/proxmox-s3-client/src/api_types.rs
@@ -226,3 +226,14 @@ pub struct S3BucketListItem {
     /// S3 bucket name.
     pub name: String,
 }
+
+#[api]
+#[derive(Copy, Clone, Deserialize, Serialize, PartialEq, Eq)]
+#[serde(rename_all = "kebab-case")]
+/// Provider specific implementation feature or limitation.
+pub enum S3ClientFeature {
+    /// If-None-Match http header not implemented and not accepted.
+    SkipIfNoneMatchHeader,
+}
+serde_plain::derive_display_from_serialize!(S3ClientFeature);
+serde_plain::derive_fromstr_from_deserialize!(S3ClientFeature);
diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index a2c62811..1c923d5f 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -22,7 +22,7 @@ use proxmox_http::client::HttpsConnector;
 use proxmox_http::{Body, RateLimit, RateLimiter};
 use proxmox_schema::api_types::CERT_FINGERPRINT_SHA256_SCHEMA;
 
-use crate::api_types::S3ClientConfig;
+use crate::api_types::{ProviderQuirks, S3ClientConfig, S3ClientFeature};
 use crate::aws_sign_v4::AWS_SIGN_V4_DATETIME_FORMAT;
 use crate::aws_sign_v4::{aws_sign_v4_signature, aws_sign_v4_uri_encode};
 use crate::object_key::S3ObjectKey;
@@ -69,6 +69,8 @@ pub struct S3ClientOptions {
     pub fingerprint: Option<String>,
     /// Rate limit for put requests given as #reqest/s.
     pub put_rate_limit: Option<u64>,
+    /// Provider implementation specific features and limitations
+    pub features: Vec<S3ClientFeature>,
 }
 
 impl S3ClientOptions {
@@ -90,6 +92,18 @@ impl S3ClientOptions {
             access_key: config.access_key,
             secret_key,
             put_rate_limit: config.put_rate_limit,
+            features: Self::map_provider_quirks_to_features(config.provider_quirks),
+        }
+    }
+
+    fn map_provider_quirks_to_features(
+        provider_quirks: Option<ProviderQuirks>,
+    ) -> Vec<S3ClientFeature> {
+        match provider_quirks {
+            Some(ProviderQuirks::Backblaze) | Some(ProviderQuirks::Infomaniak) => {
+                vec![S3ClientFeature::SkipIfNoneMatchHeader]
+            }
+            _ => Vec::new(),
         }
     }
 }
@@ -412,7 +426,15 @@ impl S3Client {
             .header(header::CONTENT_TYPE, "binary/octet");
 
         if !replace {
-            request = request.header(header::IF_NONE_MATCH, "*");
+            // Some providers not implement this and fails with error if the header is set,
+            // see https://forum.proxmox.com/threads/168834/post-786278
+            if !self
+                .options
+                .features
+                .contains(&S3ClientFeature::SkipIfNoneMatchHeader)
+            {
+                request = request.header(header::IF_NONE_MATCH, "*");
+            }
         }
 
         let request = request.body(object_data)?;
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v3 1/2] api: s3 config: allow to update or delete endpoint quirks

[Christian Ebner]

Allows to update the configured quirk value, to be used by the s3
endpoint config edit window as well as the cli.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
---
changes since version 2:
- no changes

 src/api2/config/s3.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/api2/config/s3.rs b/src/api2/config/s3.rs
index 047bf1fb1..803a1f792 100644
--- a/src/api2/config/s3.rs
+++ b/src/api2/config/s3.rs
@@ -125,6 +125,8 @@ pub enum DeletableProperty {
     Fingerprint,
     /// Delete the path-style property.
     PathStyle,
+    /// Delete the provider quirks property.
+    ProviderQuirks,
 }
 
 #[api(
@@ -197,6 +199,9 @@ pub fn update_s3_client_config(
                 DeletableProperty::PathStyle => {
                     data.config.path_style = None;
                 }
+                DeletableProperty::ProviderQuirks => {
+                    data.config.provider_quirks = None;
+                }
             }
         }
     }
@@ -219,6 +224,9 @@ pub fn update_s3_client_config(
     if let Some(path_style) = update.path_style {
         data.config.path_style = Some(path_style);
     }
+    if let Some(provider_quirks) = update.provider_quirks {
+        data.config.provider_quirks = Some(provider_quirks);
+    }
 
     if let Some(secret_key) = secret_key {
         data.secret_key = secret_key;
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v3 2/2] ui: s3 endpoint: add provider specific quirk selector

[Christian Ebner]

Add an advanced column with a provider specific quirk selector, which
allows to set the s3 endpoint accordingly and use the provider
specific implementation detail in the s3 client.

Clear the value if it is set to the default one, being no quirks.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
---
changes since version 2:
- no changes

 www/window/S3ClientEdit.js | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/www/window/S3ClientEdit.js b/www/window/S3ClientEdit.js
index f0a5efba8..dc3d69284 100644
--- a/www/window/S3ClientEdit.js
+++ b/www/window/S3ClientEdit.js
@@ -119,6 +119,24 @@ Ext.define('PBS.window.S3ClientEdit', {
                 },
             },
         ],
+
+        advancedColumn1: [
+            {
+                xtype: 'proxmoxKVComboBox',
+                name: 'provider-quirks',
+                fieldLabel: gettext('Provider specific quirks'),
+                value: '__default__',
+                defaultValue: '__default__',
+                comboItems: [
+                    ['__default__', 'None (default)'],
+                    ['backblaze', 'Backblaze B2'],
+                    ['infomaniak', 'Infomaniak'],
+                ],
+                cbind: {
+                    deleteEmpty: '{!isCreate}',
+                },
+            },
+        ],
     },
 
     getValues: function () {
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox{, -backup} v3 0/4] s3: extend config by provider-quirks and client options by feature list

[Thomas Lamprecht]

Am 04.08.25 um 08:54 schrieb Christian Ebner:
> These patches extend the s3 client configuration by the additional
> `provider-quirks` enum, allowing to switch to provider specific implementation
> details. The provider specific quirks are then mapped to a list of features and
> limitations, added to the s3 client options.
> 
> As first use-case, the `If-None-Match` header is not set during put object
> requests to Backblaze B2 or Infomaniak object stores, as these fail with an
> error with status code 501, therefore chunk uploads will fail.

Why not expose the actual quirk (features) directly? Even if we'd like to
have specific providers a user can select, that part could be a frontend
only feature, and tbh. for starters I'd even skip that, maybe documenting
provider to quirks in the PBS wiki might be a good option for now.

I'm just a bit wary of locking our backend in to those provider, as there
are quite a few of S3 provider, and not all might be here with us forever,
and having dozens of providers all mapping just to one or two actual
quirks each seems a bit overkill to me.
Sorry, I should have taken a closer check earlier to provider quicker
feedback.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox{, -backup} v3 0/4] s3: extend config by provider-quirks and client options by feature list

[Christian Ebner]

On 8/4/25 10:17 PM, Thomas Lamprecht wrote:
> Am 04.08.25 um 08:54 schrieb Christian Ebner:
>> These patches extend the s3 client configuration by the additional
>> `provider-quirks` enum, allowing to switch to provider specific implementation
>> details. The provider specific quirks are then mapped to a list of features and
>> limitations, added to the s3 client options.
>>
>> As first use-case, the `If-None-Match` header is not set during put object
>> requests to Backblaze B2 or Infomaniak object stores, as these fail with an
>> error with status code 501, therefore chunk uploads will fail.
> 
> Why not expose the actual quirk (features) directly? Even if we'd like to
> have specific providers a user can select, that part could be a frontend
> only feature, and tbh. for starters I'd even skip that, maybe documenting
> provider to quirks in the PBS wiki might be a good option for now.
> 
> I'm just a bit wary of locking our backend in to those provider, as there
> are quite a few of S3 provider, and not all might be here with us forever,
> and having dozens of providers all mapping just to one or two actual
> quirks each seems a bit overkill to me.
> Sorry, I should have taken a closer check earlier to provider quicker
> feedback.

The intention was to have it easy for the user to select the required 
features, therefore the mapping from provider to features.

But you are absolutely right that this might introduce maintenance 
burden in the long run, as the list of providers is then backed into the 
config.

So I do agree, will move this over to only expose the features in the 
config and the single NoIfNoneMatchHeader in the UI for now (and maybe 
call them quirks, after all?)


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox{, -backup} v3 0/4] s3: extend config by provider-quirks and client options by feature list

[Thomas Lamprecht]

Am 05.08.25 um 07:58 schrieb Christian Ebner:
> On 8/4/25 10:17 PM, Thomas Lamprecht wrote:
>> Am 04.08.25 um 08:54 schrieb Christian Ebner:
>>> These patches extend the s3 client configuration by the additional
>>> `provider-quirks` enum, allowing to switch to provider specific implementation
>>> details. The provider specific quirks are then mapped to a list of features and
>>> limitations, added to the s3 client options.
>>>
>>> As first use-case, the `If-None-Match` header is not set during put object
>>> requests to Backblaze B2 or Infomaniak object stores, as these fail with an
>>> error with status code 501, therefore chunk uploads will fail.
>>
>> Why not expose the actual quirk (features) directly? Even if we'd like to
>> have specific providers a user can select, that part could be a frontend
>> only feature, and tbh. for starters I'd even skip that, maybe documenting
>> provider to quirks in the PBS wiki might be a good option for now.
>>
>> I'm just a bit wary of locking our backend in to those provider, as there
>> are quite a few of S3 provider, and not all might be here with us forever,
>> and having dozens of providers all mapping just to one or two actual
>> quirks each seems a bit overkill to me.
>> Sorry, I should have taken a closer check earlier to provider quicker
>> feedback.
> 
> The intention was to have it easy for the user to select the required features, therefore the mapping from provider to features.

Yes, I get that, and don't get me wrong, in general it _is_ a nice idea for
the UX. But IMO the backend should not have to care about this, especially
not a relatively low-level and generic S3 client crate. And the only benefit
I can see over a frontend implementation would be that we could adapt the
backend automatically to change in behavior of a provider. But such change is
IMO rather unlikely, as that would be a breaking change on their parts, and if
one provides a new API version we'd also start to version the providers in the
backend, and users might need to update PBS to get the updated provider
profile, instead of being able to enable any of the existing quirks. If we
need a new quirk in general one needs to update in any way, so for that
nothing changes – just to give some more rationale.

That's why I think that the user doesn't gets much value over this being a
front-end only profile once selected on endpoint creation – again, would not
do that now, but could see this as future UX improvement, depending on user
feedback.

> So I do agree, will move this over to only expose the features in the config and the single NoIfNoneMatchHeader in the UI for now (and maybe call them quirks, after all?)

It's a short name and definitively not wrong, so "quirks" works for me.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] superseded: [PATCH proxmox{, -backup} v3 0/4] s3: extend config by provider-quirks and client options by feature list

[Christian Ebner]

superseded-by version 4:
https://lore.proxmox.com/pbs-devel/20250805105120.623089-1-c.ebner@proxmox.com/T/


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel