[pbs-devel] [PATCH proxmox{, -backup} 0/4] s3: extend config by quirks, implement backblaze B2 specific quirk

[pbs-devel] [PATCH proxmox{, -backup} 0/4] s3: extend config by quirks, implement backblaze B2 specific quirk

[Christian Ebner]

These patches extend the s3 client configuration and options by the additional
`quirk` enum, allowing to switch to provider specific implementation details in
the s3 client.

As first use-case, the `If-None-Match` header is not set during put object
requests to Backblaze B2 object stores, as this fails with an error with status
code 501, therefore chunk uploads will fail.

The patches expose the quirks also in an dropdown list in the advanced column
of the s3 endpoint edit window.

proxmox:

Christian Ebner (2):
  s3 client: api types: extend client config by optional quirks
  s3 client: skip setting `If-None-Match` header for Backblaze B2

 proxmox-s3-client/examples/s3_client.rs |  1 +
 proxmox-s3-client/src/api_types.rs      | 18 +++++++++++++++++-
 proxmox-s3-client/src/client.rs         | 10 +++++++++-
 3 files changed, 27 insertions(+), 2 deletions(-)


proxmox-backup:

Christian Ebner (2):
  api: s3 config: allow to update or delete endpoint quirks
  ui: s3 endpoint: add provider specific quirk selector

 src/api2/config/s3.rs      |  8 ++++++++
 www/window/S3ClientEdit.js | 17 +++++++++++++++++
 2 files changed, 25 insertions(+)


Summary over all repositories:
  5 files changed, 52 insertions(+), 2 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox 1/2] s3 client: api types: extend client config by optional quirks

[Christian Ebner]

Add optional provider specific quirks to the client configuration,
allowing to set a variant of the client to follow some custom
implementation variants.

Add backblaze as first variant, as it does not accept the
`If-None-Match` header in put object requests.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/examples/s3_client.rs |  1 +
 proxmox-s3-client/src/api_types.rs      | 18 +++++++++++++++++-
 proxmox-s3-client/src/client.rs         |  3 +++
 3 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/proxmox-s3-client/examples/s3_client.rs b/proxmox-s3-client/examples/s3_client.rs
index 1cbb3939..25df9609 100644
--- a/proxmox-s3-client/examples/s3_client.rs
+++ b/proxmox-s3-client/examples/s3_client.rs
@@ -38,6 +38,7 @@ async fn run() -> Result<(), anyhow::Error> {
         // `openssl s_client -connect testbucket.s3.pve-c1.local:7480 < /dev/null | openssl x509 -fingerprint -sha256 -noout`
         fingerprint: Some("<s3-api-fingerprint>".to_string()),
         put_rate_limit: None,
+        quirks: None,
     };
 
     // Creating a client instance and connect to api endpoint
diff --git a/proxmox-s3-client/src/api_types.rs b/proxmox-s3-client/src/api_types.rs
index eff15f37..59fda383 100644
--- a/proxmox-s3-client/src/api_types.rs
+++ b/proxmox-s3-client/src/api_types.rs
@@ -5,7 +5,7 @@ use serde::{Deserialize, Serialize};
 use proxmox_schema::api_types::{
     CERT_FINGERPRINT_SHA256_SCHEMA, DNS_LABEL_STR, IPRE_STR, SAFE_ID_FORMAT,
 };
-use proxmox_schema::{api, const_regex, ApiStringFormat, Schema, StringSchema, Updater};
+use proxmox_schema::{api, const_regex, ApiStringFormat, EnumEntry, Schema, StringSchema, Updater};
 
 #[rustfmt::skip]
 /// Regex to match S3 endpoint full qualified domain names, including template patterns for bucket
@@ -80,6 +80,15 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
     .max_length(63)
     .schema();
 
+/// Provider specific feature implementation quirks
+pub const S3_PROVIDER_QUIRKS_SCHEMA: Schema =
+    StringSchema::new("Provider specific feature implementation quirks.")
+        .format(&ApiStringFormat::Enum(&[EnumEntry::new(
+            "backblaze",
+            "Backblaze",
+        )]))
+        .schema();
+
 #[api(
     properties: {
         endpoint: {
@@ -109,6 +118,10 @@ pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3
             type: u64,
             optional: true,
         },
+        quirks: {
+            schema: S3_PROVIDER_QUIRKS_SCHEMA,
+            optional: true,
+        },
     },
 )]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -134,6 +147,9 @@ pub struct S3ClientConfig {
     /// Rate limit for put requests given as #reqest/s.
     #[serde(skip_serializing_if = "Option::is_none")]
     pub put_rate_limit: Option<u64>,
+    /// Provider specific feature implementation quirks
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub quirks: Option<String>,
 }
 
 impl S3ClientConfig {
diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 3a981bf4..bbe363da 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -69,6 +69,8 @@ pub struct S3ClientOptions {
     pub fingerprint: Option<String>,
     /// Rate limit for put requests given as #reqest/s.
     pub put_rate_limit: Option<u64>,
+    /// Provider specific feature implementation quirks
+    pub quirks: Option<String>,
 }
 
 impl S3ClientOptions {
@@ -90,6 +92,7 @@ impl S3ClientOptions {
             access_key: config.access_key,
             secret_key,
             put_rate_limit: config.put_rate_limit,
+            quirks: config.quirks,
         }
     }
 }
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox 2/2] s3 client: skip setting `If-None-Match` header for Backblaze B2

[Christian Ebner]

Backblaze B2 does not allow to set the `If-None-Match` http header
for put requests, failing with an error instead of implementing or
ignoring it. Skip setting the header (and therefore re-uploading
the object) if the quirk configuration is set to backblaze.

Reported-by: https://forum.proxmox.com/threads/168834/post-786278
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index bbe363da..bd7aff18 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -396,7 +396,12 @@ impl S3Client {
             .header(header::CONTENT_TYPE, "binary/octet");
 
         if !replace {
-            request = request.header(header::IF_NONE_MATCH, "*");
+            // Backblaze B2 does not implement this and fails with error if the header is set,
+            // see https://forum.proxmox.com/threads/168834/post-786278
+            match self.options.quirks.as_deref() {
+                Some("backblaze") => (),
+                _ => request = request.header(header::IF_NONE_MATCH, "*"),
+            }
         }
 
         let request = request.body(object_data)?;
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup 1/2] api: s3 config: allow to update or delete endpoint quirks

[Christian Ebner]

Allows to update the configured quirk value, to be used by the s3
endpoint config edit window as well as the cli.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 src/api2/config/s3.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/api2/config/s3.rs b/src/api2/config/s3.rs
index 04b801028..ec8e40188 100644
--- a/src/api2/config/s3.rs
+++ b/src/api2/config/s3.rs
@@ -125,6 +125,8 @@ pub enum DeletableProperty {
     Fingerprint,
     /// Delete the path-style property.
     PathStyle,
+    /// Delete the quirks property.
+    Quirks,
 }
 
 #[api(
@@ -197,6 +199,9 @@ pub fn update_s3_client_config(
                 DeletableProperty::PathStyle => {
                     data.config.path_style = None;
                 }
+                DeletableProperty::Quirks => {
+                    data.config.quirks = None;
+                }
             }
         }
     }
@@ -219,6 +224,9 @@ pub fn update_s3_client_config(
     if let Some(path_style) = update.path_style {
         data.config.path_style = Some(path_style);
     }
+    if let Some(quirks) = update.quirks {
+        data.config.quirks = Some(quirks);
+    }
 
     if let Some(secret_key) = secret_key {
         data.secret_key = secret_key;
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup 2/2] ui: s3 endpoint: add provider specific quirk selector

[Christian Ebner]

Add an advanced column with a provider specific quirk selector, which
allows to set the s3 endpoint accordingly and use the provider
specific implementation detail in the s3 client.

Clear the value if it is set to the default one, being no quirks.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 www/window/S3ClientEdit.js | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/www/window/S3ClientEdit.js b/www/window/S3ClientEdit.js
index f0a5efba8..7844a29ca 100644
--- a/www/window/S3ClientEdit.js
+++ b/www/window/S3ClientEdit.js
@@ -119,6 +119,23 @@ Ext.define('PBS.window.S3ClientEdit', {
                 },
             },
         ],
+
+        advancedColumn1: [
+            {
+                xtype: 'proxmoxKVComboBox',
+                name: 'quirks',
+                fieldLabel: gettext('Provider specific quirks'),
+                value: '__default__',
+                defaultValue: '__default__',
+                comboItems: [
+                    ['__default__', 'None (default)'],
+                    ['backblaze', 'Backblaze B2'],
+                ],
+                cbind: {
+                    deleteEmpty: '{!isCreate}',
+                },
+            },
+        ],
     },
 
     getValues: function () {
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox{, -backup} 0/4] s3: extend config by quirks, implement backblaze B2 specific quirk

[Christian Ebner]

Please disregard this patches, I will send a v2 which incorporates 
another provider which does not support the If-None-Match header as 
reported in the community forum [0]. Further, I plan to split this a bit 
better, allowing to configure the provider quirks in the s3 endpoint 
config on PBS, but translate this to a feature list on the s3 client 
options. By that this should be more flexible, allowing to more easily 
extend for new quirky provider implementations with the same feature 
limitations.


[0] https://forum.proxmox.com/threads/168954/


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] superseded: [PATCH proxmox{, -backup} 0/4] s3: extend config by quirks, implement backblaze B2 specific quirk

[Christian Ebner]

superseded-by version 2:
https://lore.proxmox.com/pbs-devel/20250729081750.203736-1-c.ebner@proxmox.com/T/


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel