[pbs-devel] [PATCH proxmox{, -backup} v4 0/3] s3: extend endpoint config and client options by provider-quirks

[pbs-devel] [PATCH proxmox{, -backup} v4 0/3] s3: extend endpoint config and client options by provider-quirks

[Christian Ebner]

These patches extend the s3 client configuration by an additional array
of `provider-quirks`, allowing to switch to provider specific implementation
details. The provider specific quirks are passed to the s3 client via it's
options on instantiation.

As first use-case, the `If-None-Match` header is not set during put object
requests to Backblaze B2 or Infomaniak object stores, as these fail with an
error with status code 501, therefore chunk uploads will fail.

The patches expose the provider quirks also in an dropdown list in the advanced
column of the s3 endpoint edit window.

Changes since v3:
- Use and expose quirks directly instead of mapping by provider list

Changes since v2:
- Rebased onto current master

Changes since v1:
- add Infomaniak as further provider not supporting the If-None-Match header
- introduce dedicate enum api type for both provider quirks and s3 client features
- add logic to map from provider to feature list
- skip If-None-Match header based on feature list, not provider

proxmox:

Christian Ebner (1):
  s3 client: extend client config and options by optional provider
    quirks

 proxmox-s3-client/examples/s3_client.rs |  1 +
 proxmox-s3-client/src/api_types.rs      | 21 +++++++++++++++++++++
 proxmox-s3-client/src/client.rs         | 15 +++++++++++++--
 3 files changed, 35 insertions(+), 2 deletions(-)


proxmox-backup:

Christian Ebner (2):
  api: s3 config: allow to update or delete endpoint quirks
  ui: s3 endpoint: add provider specific quirk selector

 src/api2/config/s3.rs      |  8 ++++++++
 www/window/S3ClientEdit.js | 17 +++++++++++++++++
 2 files changed, 25 insertions(+)


Summary over all repositories:
  5 files changed, 60 insertions(+), 2 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox v4 1/1] s3 client: extend client config and options by optional provider quirks

[Christian Ebner]

Add optional provider specific quirks as array to the client
configuration and client options, allowing to set a variant of the
client to follow some custom implementation variants.

As first variant, add the quirk to skip setting the `If-None-Match`
header in put object requests, as some providers do not accept and
implement this headers functionality.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 3:
- Drop provider list to feature list mapping, use and expose provider
  quirks directly.
- Squashed together with next patch, as this is now pretty much self
  contained

 proxmox-s3-client/examples/s3_client.rs |  1 +
 proxmox-s3-client/src/api_types.rs      | 21 +++++++++++++++++++++
 proxmox-s3-client/src/client.rs         | 15 +++++++++++++--
 3 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/proxmox-s3-client/examples/s3_client.rs b/proxmox-s3-client/examples/s3_client.rs
index 4a9625cb..61b88077 100644
--- a/proxmox-s3-client/examples/s3_client.rs
+++ b/proxmox-s3-client/examples/s3_client.rs
@@ -38,6 +38,7 @@ async fn run() -> Result<(), anyhow::Error> {
         // `openssl s_client -connect testbucket.s3.pve-c1.local:7480 < /dev/null | openssl x509 -fingerprint -sha256 -noout`
         fingerprint: Some("<s3-api-fingerprint>".to_string()),
         put_rate_limit: None,
+        provider_quirks: Vec::new(),
     };
 
     // Creating a client instance and connect to api endpoint
diff --git a/proxmox-s3-client/src/api_types.rs b/proxmox-s3-client/src/api_types.rs
index 265fedb9..c68d85ef 100644
--- a/proxmox-s3-client/src/api_types.rs
+++ b/proxmox-s3-client/src/api_types.rs
@@ -80,6 +80,17 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
     .max_length(63)
     .schema();
 
+#[api]
+#[derive(Copy, Clone, Deserialize, Serialize, PartialEq, Eq)]
+#[serde(rename_all = "kebab-case")]
+/// Provider specific feature implementation quirks.
+pub enum ProviderQuirks {
+    /// Prvider does not support the If-None-Match http header
+    SkipIfNoneMatchHeader,
+}
+serde_plain::derive_display_from_serialize!(ProviderQuirks);
+serde_plain::derive_fromstr_from_deserialize!(ProviderQuirks);
+
 #[api(
     properties: {
         endpoint: {
@@ -109,6 +120,13 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
             type: u64,
             optional: true,
         },
+        "provider-quirks": {
+            type: Array,
+            optional: true,
+            items: {
+                type: ProviderQuirks,
+            },
+        },
     },
 )]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -134,6 +152,9 @@ pub struct S3ClientConfig {
     /// Rate limit for put requests given as #reqest/s.
     #[serde(skip_serializing_if = "Option::is_none")]
     pub put_rate_limit: Option<u64>,
+    /// List of provider specific feature implementation quirks.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub provider_quirks: Option<Vec<ProviderQuirks>>,
 }
 
 impl S3ClientConfig {
diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index ae61c590..3bc2a672 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -22,7 +22,7 @@ use proxmox_http::client::HttpsConnector;
 use proxmox_http::{Body, RateLimit, RateLimiter};
 use proxmox_schema::api_types::CERT_FINGERPRINT_SHA256_SCHEMA;
 
-use crate::api_types::S3ClientConfig;
+use crate::api_types::{ProviderQuirks, S3ClientConfig};
 use crate::aws_sign_v4::AWS_SIGN_V4_DATETIME_FORMAT;
 use crate::aws_sign_v4::{aws_sign_v4_signature, aws_sign_v4_uri_encode};
 use crate::object_key::S3ObjectKey;
@@ -69,6 +69,8 @@ pub struct S3ClientOptions {
     pub fingerprint: Option<String>,
     /// Rate limit for put requests given as #reqest/s.
     pub put_rate_limit: Option<u64>,
+    /// Provider implementation specific features and limitations
+    pub provider_quirks: Vec<ProviderQuirks>,
 }
 
 impl S3ClientOptions {
@@ -90,6 +92,7 @@ impl S3ClientOptions {
             access_key: config.access_key,
             secret_key,
             put_rate_limit: config.put_rate_limit,
+            provider_quirks: config.provider_quirks.unwrap_or_default(),
         }
     }
 }
@@ -411,7 +414,15 @@ impl S3Client {
             .header(header::CONTENT_TYPE, "binary/octet");
 
         if !replace {
-            request = request.header(header::IF_NONE_MATCH, "*");
+            // Some providers not implement this and fails with error if the header is set,
+            // see https://forum.proxmox.com/threads/168834/post-786278
+            if !self
+                .options
+                .provider_quirks
+                .contains(&ProviderQuirks::SkipIfNoneMatchHeader)
+            {
+                request = request.header(header::IF_NONE_MATCH, "*");
+            }
         }
 
         let request = request.body(object_data)?;
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v4 1/2] api: s3 config: allow to update or delete endpoint quirks

[Christian Ebner]

Allows to update the configured quirk value, to be used by the s3
endpoint config edit window as well as the cli.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 3:
- no changes

 src/api2/config/s3.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/api2/config/s3.rs b/src/api2/config/s3.rs
index 047bf1fb1..803a1f792 100644
--- a/src/api2/config/s3.rs
+++ b/src/api2/config/s3.rs
@@ -125,6 +125,8 @@ pub enum DeletableProperty {
     Fingerprint,
     /// Delete the path-style property.
     PathStyle,
+    /// Delete the provider quirks property.
+    ProviderQuirks,
 }
 
 #[api(
@@ -197,6 +199,9 @@ pub fn update_s3_client_config(
                 DeletableProperty::PathStyle => {
                     data.config.path_style = None;
                 }
+                DeletableProperty::ProviderQuirks => {
+                    data.config.provider_quirks = None;
+                }
             }
         }
     }
@@ -219,6 +224,9 @@ pub fn update_s3_client_config(
     if let Some(path_style) = update.path_style {
         data.config.path_style = Some(path_style);
     }
+    if let Some(provider_quirks) = update.provider_quirks {
+        data.config.provider_quirks = Some(provider_quirks);
+    }
 
     if let Some(secret_key) = secret_key {
         data.secret_key = secret_key;
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v4 2/2] ui: s3 endpoint: add provider specific quirk selector

[Christian Ebner]

Add an advanced column with a provider specific quirk selector, which
allows to set the s3 endpoint accordingly and use the provider
specific implementation detail in the s3 client.

Clear the value if it is set to the default one, being no quirks.

For now only the one If-None-Match http header quirk is available, but
this can be extended to a multi selection list if required.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 3:
- use provider quirks directly instead of provider list

 www/window/S3ClientEdit.js | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/www/window/S3ClientEdit.js b/www/window/S3ClientEdit.js
index f0a5efba8..5128f6a54 100644
--- a/www/window/S3ClientEdit.js
+++ b/www/window/S3ClientEdit.js
@@ -119,6 +119,23 @@ Ext.define('PBS.window.S3ClientEdit', {
                 },
             },
         ],
+
+        advancedColumn1: [
+            {
+                xtype: 'proxmoxKVComboBox',
+                name: 'provider-quirks',
+                fieldLabel: gettext('Provider specific quirks'),
+                value: '__default__',
+                defaultValue: '__default__',
+                comboItems: [
+                    ['__default__', 'None (default)'],
+                    ['skip-if-none-match-header', 'Skip If-None-Match header'],
+                ],
+                cbind: {
+                    deleteEmpty: '{!isCreate}',
+                },
+            },
+        ],
     },
 
     getValues: function () {
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] applied: [PATCH proxmox v4 1/1] s3 client: extend client config and options by optional provider quirks

[Thomas Lamprecht]

On Tue, 05 Aug 2025 12:51:18 +0200, Christian Ebner wrote:
> Add optional provider specific quirks as array to the client
> configuration and client options, allowing to set a variant of the
> client to follow some custom implementation variants.
> 
> As first variant, add the quirk to skip setting the `If-None-Match`
> header in put object requests, as some providers do not accept and
> implement this headers functionality.
> 
> [...]

Applied, thanks!

[1/1] s3 client: extend client config and options by optional provider quirks
      commit: 1b1e8c61ddd48fe046204094eb95a69ddfcb1a42


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] applied: [PATCH proxmox-backup v4 1/2] api: s3 config: allow to update or delete endpoint quirks

[Thomas Lamprecht]

On Tue, 05 Aug 2025 12:51:19 +0200, Christian Ebner wrote:
> Allows to update the configured quirk value, to be used by the s3
> endpoint config edit window as well as the cli.
> 
> 

Applied, thanks!

[1/2] api: s3 config: allow to update or delete endpoint quirks
      commit: 08a0f8774c2f0ab23e3a7eb12ff5e241b8f62497


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] applied: [PATCH proxmox-backup v4 2/2] ui: s3 endpoint: add provider specific quirk selector

[Thomas Lamprecht]

On Tue, 05 Aug 2025 12:51:20 +0200, Christian Ebner wrote:
> Add an advanced column with a provider specific quirk selector, which
> allows to set the s3 endpoint accordingly and use the provider
> specific implementation detail in the s3 client.
> 
> Clear the value if it is set to the default one, being no quirks.
> 
> For now only the one If-None-Match http header quirk is available, but
> this can be extended to a multi selection list if required.
> 
> [...]

Applied, with minor label rewording in a follow up, thanks!

Btw. if we get more than this I'd probably favor moving this to a list of check
boxes, e.g. nested inside a FieldSet [0] with collapsible disabled and Provider
Quirks as title, then it would IMO stay a bit more discoverable compared to a
multi-selection inside a combobox. But only relevant once we get new quirks.

[0]: https://docs.sencha.com/extjs/7.0.0/classic/Ext.form.FieldSet.htm

[2/2] ui: s3 endpoint: add provider specific quirk selector
      commit: fc2a5d7c03bec1766b9f374b559946c4b47a91e0


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] applied: [PATCH proxmox-backup v4 2/2] ui: s3 endpoint: add provider specific quirk selector

[Christian Ebner]

On 8/5/25 5:40 PM, Thomas Lamprecht wrote:
> On Tue, 05 Aug 2025 12:51:20 +0200, Christian Ebner wrote:
>> Add an advanced column with a provider specific quirk selector, which
>> allows to set the s3 endpoint accordingly and use the provider
>> specific implementation detail in the s3 client.
>>
>> Clear the value if it is set to the default one, being no quirks.
>>
>> For now only the one If-None-Match http header quirk is available, but
>> this can be extended to a multi selection list if required.
>>
>> [...]
> 
> Applied, with minor label rewording in a follow up, thanks!
> 
> Btw. if we get more than this I'd probably favor moving this to a list of check
> boxes, e.g. nested inside a FieldSet [0] with collapsible disabled and Provider
> Quirks as title, then it would IMO stay a bit more discoverable compared to a
> multi-selection inside a combobox. But only relevant once we get new quirks.
> 
> [0]: https://docs.sencha.com/extjs/7.0.0/classic/Ext.form.FieldSet.html

Okay, thanks for the suggestion!


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel