public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [RFC proxmox 2/2] pbs-api-types: extend datastore config by backend config enum
Date: Mon, 19 May 2025 13:46:03 +0200	[thread overview]
Message-ID: <20250519114640.303640-3-c.ebner@proxmox.com> (raw)
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>

Allows to configure a backend config variant for a datastore on
creation. The current default `Filesystem` backend variant is
introduced to be compatible with existing storages. A new S3 backend
variant allows to create datastores backed by an S3 compatible object
store instead.

For S3 backends, the id of the corresponding S3 client configuration
is storered. A valid datastore backend configuration for S3 therefore
contains:
```
    ...
    backend s3=<S3_CONFIG_ID>
    ...
```

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-api-types/src/datastore.rs | 58 +++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 5bd953ac..2b983cb2 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -336,7 +336,11 @@ pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore
             optional: true,
             format: &proxmox_schema::api_types::UUID_FORMAT,
             type: String,
-        }
+        },
+        backend: {
+            schema: DATASTORE_BACKEND_SCHEMA,
+            optional: true,
+        },
     }
 )]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -389,8 +393,59 @@ pub struct DataStoreConfig {
     #[updater(skip)]
     #[serde(skip_serializing_if = "Option::is_none")]
     pub backing_device: Option<String>,
+
+    /// Backend to be used by datastore
+    #[updater(skip)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub backend: Option<String>,
+}
+
+pub const DATASTORE_BACKEND_SCHEMA: Schema = StringSchema::new("Backend config to be used for datastore.")
+    .format(&ApiStringFormat::VerifyFn(verify_datastore_backend))
+    .type_text("<filesystem|s3=S3_CONFIG_ID>")
+    .schema();
+
+fn verify_datastore_backend(input: &str) -> Result<(), Error> {
+    DatastoreBackendConfig::from_str(input).map(|_| ())
+}
+
+#[derive(Clone, Default)]
+/// Available backend configurations for datastores.
+pub enum DatastoreBackendConfig {
+    #[default]
+    Filesystem,
+    S3(String),
 }
 
+impl std::str::FromStr for DatastoreBackendConfig {
+    type Err = Error;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        if s == "filesystem" {
+            return Ok(Self::Filesystem);
+        }
+        match s.split_once('=') {
+            Some(("s3", value)) => {
+                let s3_config_id = value.parse()?;
+                Ok(Self::S3(s3_config_id))
+            }
+            _ => bail!("invalid datastore backend configuration"),
+        }
+    }
+}
+
+impl std::fmt::Display for DatastoreBackendConfig {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::Filesystem => write!(f, "filesystem"),
+            Self::S3(s3_config_id) => write!(f, "s3:{s3_config_id}"),
+        }
+    }
+}
+
+proxmox_serde::forward_serialize_to_display!(DatastoreBackendConfig);
+proxmox_serde::forward_deserialize_to_from_str!(DatastoreBackendConfig);
+
 #[api]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq, Default)]
 #[serde(rename_all = "kebab-case")]
@@ -424,6 +479,7 @@ impl DataStoreConfig {
             tuning: None,
             maintenance_mode: None,
             backing_device: None,
+            backend: None,
         }
     }
 
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


  parent reply	other threads:[~2025-05-19 11:47 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-05-19 11:46 [pbs-devel] [RFC proxmox proxmox-backup 00/39] S3 storage backend for datastores Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox 1/2] pbs-api-types: add types for S3 client configs and secrets Christian Ebner
2025-05-19 11:46 ` Christian Ebner [this message]
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 03/39] fmt: fix minor formatting issues Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 04/39] verify: refactor verify related functions to be methods of worker Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 05/39] s3 client: add crate for AWS S3 compatible object store client Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 06/39] s3 client: implement AWS signature v4 request authentication Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 07/39] s3 client: add dedicated type for s3 object keys Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 08/39] s3 client: add helper for last modified timestamp parsing Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 09/39] s3 client: add helper to parse http date headers Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 10/39] s3 client: implement methods to operate on s3 objects in bucket Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 11/39] config: introduce s3 object store client configuration Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 12/39] api: config: implement endpoints to manipulate and list s3 configs Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 13/39] api: datastore: check S3 backend bucket access on datastore create Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 14/39] datastore: allow to get the backend for a datastore Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 15/39] api: backup: store datastore backend in runtime environment Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 16/39] api: backup: conditionally upload chunks to S3 object store backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 17/39] api: backup: conditionally upload blobs " Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 18/39] api: backup: conditionally upload indices " Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 19/39] api: backup: conditionally upload manifest " Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 20/39] api: reader: fetch chunks based on datastore backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 21/39] datastore: local chunk reader: read chunks based on backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 22/39] verify worker: add datastore backed to verify worker Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 23/39] verify: implement chunk verification for stores with s3 backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 24/39] api: remove snapshot from S3 backend on snapshot delete Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 25/39] datastore: prune groups/snapshots from S3 object store backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 26/39] datastore: implement garbage collection for s3 backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 27/39] ui: add S3 client edit window for configuration create/edit Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 28/39] ui: add S3 client view for configuration Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 29/39] ui: expose the S3 client view in the navigation tree Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 30/39] ui: add s3 bucket selector and allow to set s3 backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 31/39] api/bin: add endpoint and command to test s3 backend for datastore Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 32/39] tools: lru cache: add removed callback for evicted nodes Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 33/39] tools: async lru cache: implement insert, remove and contains methods Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 34/39] datastore: add local datastore cache for network attached storages Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 35/39] api: backup: use local datastore cache on S3 backend chunk upload Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 36/39] api: reader: use local datastore cache on S3 backend chunk fetching Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 37/39] api: backup: add no-cache flag to bypass local datastore cache Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 38/39] datastore: get and set owner for S3 store backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 39/39] datastore: create namespace marker in S3 backend Christian Ebner
2025-05-29 14:33 ` [pbs-devel] superseded: [RFC proxmox proxmox-backup 00/39] S3 storage backend for datastores Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250519114640.303640-3-c.ebner@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal